Supplier default risk assessment method based on hierarchical reinforcement learning

Abstract

The invention provides a supplier default risk assessment method based on hierarchical reinforcement learning. Comprising a global task distribution module, a local task execution module, an experience playback buffer module and a supplier default risk assessment module. For the input order tasks, the global task allocation module is used for judging the priority of each production task and allocating the production tasks to each factory according to the priority order; the local task execution module formulates a production plan from a task to a production machine for each factory; the experience playback buffer module records the state of the system and upper and lower layer scheduling strategy information in the model training process; the supplier default risk assessment module predicts a delivery rate and a default rate of each batch of orders according to the production schedule plans corresponding to the global task allocation module and the local task execution module, and performs simulation assessment according to a plurality of input production examples of suppliers; and evaluating the default risk and delivery credit of the supplier and outputting a prediction result.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a supplier default risk assessment method based on hierarchical reinforcement learning. Background Art

[0002] The supply chain is composed of multiple entities such as suppliers, manufacturers, distributors, and retailers to realize the process from production to sales of products. This process includes the procurement, production, transportation, inventory management, sales and other links of raw materials, as well as the information flow, capital flow and logistics related to these links. The production link is the upstream link of the supply chain ecology. Reasonable production scheduling is the key to ensuring the orderly progress of production activities. It can improve production efficiency, reduce costs, and shorten delivery time, thereby enhancing the competitiveness of suppliers and even the entire supply chain. However, order requirements that do not meet the actual situation of the factory and unreasonable production scheduling plans will lead to inefficient production lines and failure to complete production tasks on time, which will in turn affect the operation of the entire supply chain and seriously reduce the company's performance credit quality assessment. Therefore, if the downstream demand side of the supply chain can obtain information such as the supplier's factory scale, workshop machinery and manpower conditions, order requirements, etc., it can establish and solve the production scheduling model to obtain the supplier's ideal delivery situation, and infer the possibility of supplier delivery default, so as to make a reasonable assessment of the manufacturer's default risk and delivery credit.

[0003] The key to the above idea is to model and solve the workshop production scheduling problem, and researchers have proposed many methods for this. These methods are mainly divided into three categories: traditional optimization methods, methods based on global search, and methods based on deep reinforcement learning. Traditional optimization methods establish a mathematical model for production scheduling and solve it through traditional optimization methods such as dynamic programming, linear programming, and convex optimization. However, the establishment of mathematical models is highly dependent on domain knowledge and is not scalable, making it difficult to apply to dynamically changing order requirements. Global search methods avoid the establishment of mathematical models and the reliance on domain knowledge by traversing and evaluating the solution space. This type of method mainly includes genetic algorithms, swarm intelligence, and their hybrid with optimization methods. However, exploring and traversing the solution space brings high computational costs and low solution efficiency, which cannot meet the needs of rapid response to orders in real business.

[0004] In recent years, deep reinforcement learning has demonstrated its powerful capabilities in complex continuous decision-making tasks such as Go and video games. This method uses deep neural networks to extract and characterize data and mine associations, avoiding the need to explicitly establish mathematical models. At the same time, it interacts with the environment through exploratory interaction with the goal of maximizing rewards, avoiding inefficient traversal of the global solution space, and thus shows full potential in solving dynamic production scheduling problems. However, existing deep reinforcement learning methods only consider completely static production scheduling scenarios, or only consider relatively static scenarios where production task orders are dynamically generated, without considering scenarios where multiple factors such as orders, factories, and the number of machines change in real time. In addition, these methods all obey the unified scheduling of centralized strategies in terms of production task allocation and production process scheduling, while ignoring the distributed characteristics of the actual workshop production environment, which makes the algorithm face a huge observation decision space.

[0005] In summary, the existing production scheduling methods are still difficult to adapt to the dynamically arriving tasks, dynamically changing number of factories and machines in real scenarios, and these deficiencies will lead to deviations or even errors in the modeling of the scenarios, resulting in suboptimal or even wrong production planning, which will lead to a serious overestimation of the manufacturer's default risk. These problems are particularly evident in the evaluation tasks of large-scale supply chain manufacturers. In summary, in order to meet the needs of the supply chain production link, a new method needs to be constructed to solve the dynamic scheduling problem of the workshop.

[0006] The key to assessing a manufacturer's default risk and delivery credit is to model and solve the production scheduling problem, and to accurately predict the company's delivery time based on the company's current optimal production scheduling plan. Generally speaking, the production scheduling problem can be divided into two sub-problems: job allocation and job sequencing. The former requires that continuous generation tasks be assigned to each factory, while the latter requires determining the order in which machines in the factory are occupied to perform job tasks. The production scheduling strategy faces the following difficulties in the above process:

[0007] (1) Dynamic arrival of work tasks: In the supply chain scenario, order demand is not fixed but is generated and released immediately. The production scheduling strategy needs to formulate new production plans in real time for the new order demand that is constantly generated. New production tasks may also have an impact on ongoing workshop production tasks due to resource conflicts and other reasons. This is the primary consideration of the production scheduling strategy.

[0008] (2) Dynamic changes in the number of factories: In the process of allocating production tasks, the changes in the number of factories should also be considered. This phenomenon is mainly based on two reasons: first, under different industrial scales, the number of factories that the algorithm strategy needs to schedule is different; second, under the same supply chain scenario, the expansion, contraction and migration of the industry will also lead to the addition and reduction of factories, which brings great uncertainty to the production scheduling strategy and algorithm. Therefore, the allocation of production tasks should not be modeled only under the allocation mode of a fixed number of factories.

[0009] (3) Dynamic changes in production machines: In addition to the uncertainty brought about by the addition or reduction of factories, the number changes caused by the addition, scrapping and maintenance of production machines should also be considered in the sorting of work tasks in each factory. The factory's production scheduling strategy needs to take the above changes into account and fine-tune the production plan in real time to ensure that the factory delivers the assigned production tasks on time. At the same time, the upper-level production allocation strategy also needs to dynamically measure the production capacity and scale of each factory based on the changes in the number of production machines, and adjust the allocation strategy in real time.

[0010] However, existing deep reinforcement learning methods only consider completely static production scheduling scenarios, or only consider relatively static scenarios where production task orders are dynamically generated, without considering scenarios where multiple factors such as the number of orders, factories and machines change in real time. In addition, these methods all obey the unified scheduling of centralized strategies in production task allocation and production process scheduling, but ignore the distributed characteristics of the actual workshop production environment, which makes the algorithm face a huge observation decision space. Summary of the invention

[0011] The purpose of the embodiments of the present disclosure is to provide a supplier default risk assessment method based on hierarchical reinforcement learning. The main research goal of the present invention is to realize the automatic generation of code summary documents with project-level granularity through the organic combination of hierarchical-based models and program analysis technology. The specific object of the study is English documents, and the programming language targeted is Java. The summary document is mainly reflected in a short natural language summary, which does not involve complex information such as directory structure, instructions for use, etc. Based on this model, an open source code repository summary automatic generation tool is designed and implemented on the Eclipse IDE in the form of an Eclipse plug-in.

[0012] In a general aspect, a supplier default risk assessment method based on hierarchical reinforcement learning is provided, comprising a global task allocation module, a local task execution module, an experience replay buffer module, and a supplier default risk assessment module;

[0013] For the input order tasks, an upper-level strategy is constructed in the global task allocation module to judge the priority of each production task and allocate the production tasks to each factory according to the priority order; a lower-level strategy is constructed in the local task execution module to formulate a production plan for each factory from tasks to production machines;

[0014] The experience playback buffer module records the system status and upper and lower layer scheduling strategy information during the model training process, which is used to optimize and update the upper and lower layer strategies;

[0015] The supplier default risk assessment module predicts the delivery rate and default rate of each batch of orders based on the production scheduling plans corresponding to the global task allocation module and the local task execution module, performs simulation evaluation based on multiple production samples of the input supplier, evaluates the supplier's default risk and delivery credit, and outputs the prediction results.

[0016] The global task allocation module has n order tasks at time t. When the batch arrives, the execution of the order task is delegated to m factories {f1,f2,…f m}, solving the task allocation problem based on three steps:

[0017] Step 1: Task and factory status representation: Define the task status as s j =[jpt,at,ldt,jiw], where jpt represents the total time required for task processing, at represents the task arrival time, ldt represents the latest delivery time of the task, and jiw represents the importance weight of the task; the state of the factory is defined as s f =[apt,noj,awt,nofm], where apt represents the average processing time of tasks in the factory, noj represents the tasks currently assigned but not completed in the factory, awt represents the average waiting time of tasks in the factory, and nofm represents the number of unoccupied machines required for each process in the factory;

[0018] Through the task state representation module E j and plant status characterization module E f , respectively get the task status table and plant status characterization

[0019] Step 2: Assignment of tasks to factories: First, for task j u and factory v , by calculating task j u Status representation To the factory v Status representation The attention weight between them is used to establish the strength of the association between the task and the factory, as shown in the following formula:

[0020]

[0021] in and They are the calculation matrices of query value and key value in the attention layer respectively;

[0022] Secondly, we calculate the weighted sum of the factory state representation and the attention weight of the task to the factory to obtain task j u Weighted average representation of factories in is the calculation matrix of the value in the attention layer;

[0023] At the same time, the attention weights between the weighted average representations of the above tasks for the factory are calculated to establish the degree of attention to other tasks when arranging a factory for a certain task: in and They are the calculation matrices of query value and key value in the attention layer respectively;

[0024] Then, we calculate the weighted sum of the attention weights of each task i and other tasks to get task j. u The global state representation of in is the calculation matrix of the value in the attention layer;

[0025] Finally, the task assignment Q network is used to calculate the Q value of assigning task u to factory v: And choose a task u in the ε-greedy way * Assigned to a factory v * , where F Q_global Q network model for global task allocation module, with task-to-factory weights And the global state representation of the task As input; when the order task batch arrives, step 1 and step 2 are executed repeatedly until all current tasks are assigned to each factory;

[0026] Step 3: Evaluation of task assignment: Let r global (t) is the reward of the global task allocation module, which is calculated by the maximum processing time of the factory and the minimum processing time difference of the factory. The formula is:

[0027] r global (t)=-(max k∈F (apt k (t)nojk (t))-min k∈F (apt k (t)noj k (t)))

[0028] Where F = {f1,f2,…f m} represents a collection of factories, apt k (t) and noj k (t) represent the average processing time and number of tasks in factory k at time t respectively.

[0029] The specific implementation method of the local task execution module is: define the production task j to be passed through np j (≤p) processes must be processed before entering the delivery stage. The maximum number of processes is p, and production and processing must strictly follow Each process needs to be completed on a specified type of machine, that is, p types of processes correspond to p types of machines, and the total number of these p types of machines that can be used for production in a factory k is: At time t, factory k received the production task assigned by the upper level. At the same time, there are several tasks in the factory waiting to be assigned to machines to start different processes and tasks that are occupying the machine to perform different processes And the number of p-type machines that are operating normally and idle at time t is Combine the tasks that arrive at factory k at time t and the tasks that are waiting to be processed into a set of tasks to be processed Where n is the number of tasks waiting to be processed; in order to achieve the goal of shortening the factory's completion time to the maximum extent and completing the delivery of orders on time;

[0030] The module implements the factory task setting based on three steps:

[0031] Step 1: Task and factory status representation: When assigning machines to tasks waiting for different processes, it is necessary to refer to the status of each task to formulate an assignment plan. Unlike the global task assignment module, different task status information needs to be considered in this module. Therefore, the status of the task in this module is defined as s j =[ptpo,jno,amno,wt,ldt,jiw], where ptpo represents the processing time required for each process of the task, jno represents the next process of the task, amno represents the number of machines required for the next process of the task, wt represents the waiting time before the task executes the next process, ldt represents the latest delivery time of the task, and jiw represents the importance weight of the task;

[0032] Task state representation module through local task execution module (Different from the task status representation module in the global task allocation) to obtain the task status table

[0033] Step 2: Assign tasks to machines: First, by calculating the attention weights between tasks, we can obtain the degree of association with other tasks that needs to be considered when assigning machines to each task:

[0034]

[0035] in and They are the calculation matrices of query value and key value in the attention layer respectively;

[0036] By calculating the weighted sum of the attention weights of each task i and other tasks, task j in the local task execution module is obtained. u The global state representation of in is the calculation matrix of the value in the attention layer;

[0037] Finally, the machine allocation Q network calculates the Q value of the machine required to assign task i to the next process: where F Q_local is the Q network model of the local task execution module, It means that if the number of available machines required for the next step of task i is 0, no machine will be assigned to task i and the Q value will be set to 0. After calculating the Q value, task i will be assigned to the next step in an ε-greedy manner. * Assigned to machine m * ; When the order task batch is assigned to factory k at a certain moment, steps 1 and 2 will be executed repeatedly until all current tasks are assigned or there is no machine that can be assigned to the task;

[0038] Step 3: Evaluation of task execution: Calculate the reward for lower-level task execution by the time difference of factory task completion: r local (t)=-(C t -C t-1 )-α late n late , where C t is the time point at which the factory completed the last task at time t. This reward reflects the frequency of the factory's delivery, and thus reflects the factory's production efficiency. In addition, n late represents the number of orders that the factory completed delivery at time t but exceeded the delivery time, α late Indicates the penalty coefficient for order timeout.

[0039] The experience replay buffer module reads data in batches from the replay buffer during the model update phase for model update. The specific implementation steps are as follows:

[0040] First, the global task allocation module is updated: For the global task allocation module, let B be the batch data taken from the global module playback buffer, represents the data of a decision step, where and represents the status of all unassigned tasks and the status of all factories at time t. t represents the allocation action of allocating a task to a factory at time t, r global (t) represents the reward of the global module at time t; the error function of the global task allocation module is as follows:

[0041]

[0042] Where Q global The whole represents the network of the global task allocation module, b represents the size of the data batch, and γ represents the discount factor;

[0043] After that, the local task execution module is updated: Let B be the batch data taken from the local module playback buffer, which stores the decision data sampled when all factories arrange production operations. represents the data of a decision step, where a represents the status of all tasks waiting for the machine required for the next step at time t, t represents the allocation action of allocating a machine to a task at time t, r local (t) represents the reward of the local module at time t; the error function of the local task execution module is as follows: Where Q local The whole represents the network of the local task execution module, b represents the size of the data batch, and γ represents the discount factor.

[0044] The supplier default risk assessment module is based on historical and current enterprise order data, simulates multiple batches of orders and the enterprise's production and manufacturing processes through sample estimation, and calculates the supplier's default rate based on the simulated delivery time results. The specific implementation method is: Assume that J1, J2, ..., J N The order batch for the current supplier in the simulation, where For the i-th batch of order tasks, after the global task allocation module and the local task execution module, the default rate of each batch of orders is simulated:

[0045]

[0046] Where ni is the number of orders in the i-th batch, For orders The simulated completion time, For orders The order delivery period is ; after the supply chain production scheduling model predicts the default rate of multiple batches of orders, the sample estimate of the supplier default risk Θ is obtained Take the sample mean as the sample estimate of supplier default risk Θ, that is:

[0047]

[0048] Set a series of standard default risk lines η1=0.01,η2=

[0049] 0.05,η3=0.1, Suppliers with values ​​lower than η1 can be evaluated as low default risk suppliers, while suppliers between η1 and η2, η2 and η3, and above η3 can be evaluated as third-level default risk suppliers, second-level default risk suppliers, and first-level default risk suppliers, respectively.

[0050] The innovation of the embodiment of the present invention lies in:

[0051] 1. Based on the hierarchical reinforcement learning method, the system decomposes the production scheduling strategy into upper-level strategy and lower-level strategy. The upper-level strategy is responsible for allocating each job task to each factory based on the current orders and factory conditions in real time; the lower-level strategy arranges the production plan based on the assigned job tasks and the status of the production machines, and allocates the machine to each job task to deliver the order on time. By modeling the production scheduling strategy through hierarchical reinforcement learning, a production scheduling architecture that is more in line with the actual scenario is realized with upper-level macro allocation and lower-level specific execution.

[0052] 2. The upper and lower strategy models of this method both introduce attention structures, and the ability of the attention model to process variable-length sequences is used to solve the dynamically changing order tasks and production conditions such as factories and machines. By introducing the characteristics of the attention module to process variable-length sequences, the dynamic information scale problem brought about by dynamically changing orders, factories, and machines in the production scheduling scenario is solved.

[0053] 3. Use the trained supply chain production scheduling model to predict and estimate the delivery time of the enterprise's production order demand samples, thereby obtaining a sample estimate of the supplier enterprise's default risk, thereby evaluating the supplier enterprise's performance and credit indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0055] Figure 1 It is a schematic diagram showing the architecture diagram of a supplier default risk assessment method based on hierarchical reinforcement learning according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0057] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0058] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0059] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0060] In the specification, when an element (such as a layer, a region or a substrate) is described as being “on”, “connected to” or “coupled to” another element, the element may be directly “on”, “connected to” or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to” or “directly coupled to” another element, there may be no other elements present therebetween.

[0061] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0062] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0063] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0064] Figure 1 is a schematic diagram showing a supplier default risk assessment method based on hierarchical reinforcement learning according to an embodiment of the present disclosure.

[0065] like Figure 1 As shown in the figure, the system is divided into a global task allocation module, a local task execution module, an experience replay buffer module, and a supplier default risk assessment module. The global task allocation module belongs to the upper-level strategy, which is responsible for judging the priority of each production task and assigning the production task to each factory according to the priority order. The local task execution module belongs to the lower-level strategy, which is responsible for formulating a production plan from tasks to production machines for each factory. The experience replay buffer module is responsible for recording the system status and upper and lower scheduling strategies (actions) and other information during the model training process, which is used to optimize and update the upper and lower strategies. The supplier default risk assessment module predicts the delivery rate and default rate of each batch of orders based on the production scheduling plan corresponding to the global and local task modules, and accurately assesses the supplier's default risk and delivery credit through simulation evaluation of multiple production samples of the supplier.

[0066] Global task allocation module:

[0067] When the order tasks arrive in batches, the global task allocation module is responsible for delegating the execution of production tasks to each factory. is the new order task at time t, {f1,f2,…f m} is the current m factories, and both the number of orders n and the number of factories m arriving at each moment will change dynamically. This module solves the upper-level task allocation problem through the ability of the attention structure to process variable-length sequences, and implements the training of the upper-level task allocation strategy through the DQN algorithm. Specifically, this module is mainly based on the following three steps:

[0068] (1) Task and factory status representation: In the process of allocating tasks to factories, it is necessary to refer to the status of each task and factory to formulate an allocation plan. Specifically, this method defines the task status as s j =[jpt,at,ldt,jiw], and the state of the factory is defined as s f =[apt,noj,awt,nofm], whose meaning is shown in Table 1. This step is performed by the task state representation module E j and plant status characterization module E f , respectively get the task status table and plant status characterization In this method, the state representation of tasks and factories is realized by multi-layer perceptron (MLP). Taking the task state representation as an example:

[0069]

[0070] Where W1, W2 and b1, b2 represent the weights and offsets of the two linear layers in the MLP structure respectively.

[0071] (2) Assignment of tasks to factories: This step first calculates the task status representation To factory status characterization The attention weights of task j are used to establish u To the factory v The strength of the association is shown in the following formula: and They are the calculation matrices of the query value and key value in the attention layer respectively.

[0072]

[0073] Secondly, as shown in the following formula, this step calculates the weighted sum of the factory state representation and the task-to-factory attention weight to obtain task j u Weighted average representation of factories in It is the calculation matrix of the value in the attention layer.

[0074]

[0075] At the same time, the attention weights between the weighted average representations of the above tasks on the factory are calculated according to the following formula to establish the degree of attention to other tasks when arranging a factory for a certain task, where and They are the calculation matrices of the query value and key value in the attention layer respectively.

[0076]

[0077] Then, as shown in the following formula, this step calculates the weighted sum of the attention weights of each task i and other tasks to obtain task j u The global state representation of in It is the calculation matrix of the value in the attention layer.

[0078]

[0079] Finally, as shown in the following formula, the task assignment Q network calculates the Q value of assigning task u to factory v, and selects a task u in an ε-greedy manner. * Assigned to a factory v * , where F Q_global The Q network model of the global task allocation module is based on the task-to-factory weights. And the global state representation of the task When the order tasks arrive in batches, the above steps 1 and 2 are executed repeatedly until all current tasks are assigned.

[0080]

[0081] (3) Evaluation of task allocation: The goal of the global task allocation module should be to balance the task loads among the factories, and should not involve the specific arrangements of the production plans of each factory. Therefore, this method calculates the reward of the global task allocation module by the maximum processing time of the factory and the minimum processing time difference of the factory as shown in the following formula, where F = {f1, f2, ... f m} represents a collection of factories, apt k (t) and noj k (t) represent the average processing time and number of tasks in factory k at time t respectively.

[0082] r global (t)=-(max k∈F (apt k (t)noj k (t))-min k∈F (apt k (t)noj k (t)))

[0083] Table 1 Definition of task and factory status in the global task allocation module

[0084]

[0085]

[0086] Local task execution module:

[0087] In every factory production task, the product often needs to go through multiple production and processing steps before it can be finally delivered. In this module, we assume that a production task j needs to go through np j (≤p) processes must be processed before entering the delivery stage. The maximum number of processes is p, and production and processing must strictly follow Each process needs to be completed on a specified type of machine, that is, p types of processes correspond to p types of machines, and the total number of these p types of machines that can be used for production in a factory k is Based on the modeling of the above lower-level production environment, consider the following situation: At time t, factory k receives the production task assigned by the upper layer At the same time, there are several tasks in the factory waiting to be assigned to machines to start different processes and tasks that are occupying the machine to perform different processes And the number of p-type machines that are operating normally and idle at time t is

[0088] We merge the tasks that arrive at factory k at time t and the tasks that are waiting to be processed into the set of pending tasks Where n is the number of tasks waiting to be processed. Considering the dynamic change of the number of tasks at different times, this method also uses the attention structure to deal with the lower-level task execution problem, and also uses the DQN algorithm to implement the training of the lower-level task execution strategy. Specifically, this module is mainly based on the following three steps:

[0089] (1) Task and factory status representation: When assigning machines to tasks waiting for different processes, it is necessary to refer to the status of each task to formulate an assignment plan. Unlike the global task assignment module, different task status information needs to be considered in this module. Therefore, the task status in this module is defined as s j =[ptpo,jno,amno,wt,ldt,jiw], whose meaning is shown in Table 2. This step is performed by the task state representation module of the local task execution module (Different from the task status representation module in the global task allocation) to obtain the task status table

[0090] (2) Assignment of tasks to machines: In this step, we first calculate the attention weights between tasks to obtain the degree of association with other tasks that needs to be considered when assigning machines to each task, where and They are the calculation matrices of the query value and key value in the attention layer respectively.

[0091]

[0092] Then, as shown in the following formula, this step calculates the weighted sum of the attention weights of each task i and other tasks to obtain the local task execution module, task j u The global state representation of in It is the calculation matrix of the value in the attention layer.

[0093]

[0094] Finally, as shown in the following formula, the machine allocation Q network calculates the Q value of the machine required to assign task i to perform the next process, where F Q_local is the Q network model of the local task execution module, It means that if the number of available machines required for the next step of task i is 0, no machine will be assigned to task i and the Q value will be set to 0. After calculating the Q value, the ε-greedy method is used to select task i. * Assigned to machine m * When the order task batch is assigned to factory k at a certain moment, the above steps 1 and 2 will be executed in a loop until all current tasks are assigned or there is no machine that can be assigned to the task.

[0095]

[0096] (3) Evaluation of task execution: The goal of the local task execution module is to minimize the completion time of the factory so as to complete the delivery of the order on time. Therefore, this method calculates the reward of the lower-level task execution by the time difference of the factory task completion as shown in the following formula, where C t is the time point at which the factory completed the last task at time t. This reward reflects the frequency of the factory's delivery, and thus reflects the factory's production efficiency. In addition, n late represents the number of orders that the factory completed delivery at time t but exceeded the delivery time, α late Indicates the penalty coefficient for order timeout.

[0097] r local (t)=-(C t -C t-1 )-α late nlate

[0098] Table 2 Definition of task status in local task execution module

[0099]

[0100] Experience replay buffer module:

[0101] In this method, the global task allocation module and the local task execution module store the data of each step in the execution process into the experience replay buffer module. During the model update phase, this module reads data in batches from the replay buffer for model update.

[0102] (1) Update of the global task allocation module: For the global task allocation module, let B be the batch data taken from the global module playback buffer. represents the data of a decision step, where and represents the status of all unassigned tasks and the status of all factories at time t. t represents the allocation action at time t (assigning a task to a factory), r global (t) represents the reward of the global module at time t. The error function of the global task allocation module is shown in the following formula, where Q global The whole represents the network of the global task allocation module, b represents the size of the data batch, and γ represents the discount factor.

[0103]

[0104] (2) Update of local task execution module: Let B be the batch data taken from the local module playback buffer, which stores the decision data sampled when all factories arrange production operations. represents the data of a decision step, where a represents the status of all tasks waiting for the machine required for the next step at time t, t represents the allocation action at time t (allocating a machine for a task), r local (t) represents the reward of the local module at time t. The error function of the local task execution module is shown in the following formula, where Q local The whole represents the network of the local task execution module, b represents the size of the data batch, and γ represents the discount factor.

[0105]

[0106] 5.4 Supplier Default Risk Assessment Module

[0107] The supply chain production scheduling model updated and optimized by the above process can predict the supplier's reasonable production plan under the condition of any batch of order demand, so as to predict the default risk of the order. Therefore, we can simulate multiple batches of orders and the company's production and manufacturing process through sample estimation based on historical and current enterprise order data, and calculate the supplier's default rate based on the simulated delivery time results. Assume that J1, J2, ..., J N The order batch for the current supplier in the simulation, where is the i-th batch of order tasks. After the global task allocation and local task execution of this supply chain production scheduling model, the default rate of each batch of orders is simulated:

[0108]

[0109] Where n i is the number of orders in the i-th batch, For orders The simulated completion time, For orders The supply chain production scheduling model predicts the default rate of multiple batches of orders, and a sample estimate of the supplier default risk Θ can be obtained. We take the sample mean as the sample estimate of supplier default risk Θ, that is:

[0110]

[0111] We set a series of standard default risk lines eta1=0.01, eta2=0.05, eta3=0.1, Suppliers with values ​​lower than η1 can be evaluated as low default risk suppliers, while suppliers between η1 and η2, η2 and η3, and higher than η3 can be evaluated as third-level default risk suppliers, second-level default risk suppliers, and first-level default risk suppliers, respectively, and different bidding policies can be adopted.

[0112] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A supplier default risk assessment method based on hierarchical reinforcement learning, characterized in that: It includes global task allocation module, local task execution module, experience playback buffer module and supplier default risk assessment module; For the input order tasks, an upper-level strategy is constructed in the global task allocation module to judge the priority of each production task and allocate the production tasks to each factory according to the priority order; a lower-level strategy is constructed in the local task execution module to formulate a production plan for each factory from tasks to production machines; The experience playback buffer module records the system status and upper and lower layer scheduling strategy information during the model training process, which is used to optimize and update the upper and lower layer strategies; The supplier default risk assessment module predicts the delivery rate and default rate of each batch of orders based on the production scheduling plans corresponding to the global task allocation module and the local task execution module, performs simulation evaluation based on multiple production samples of the input supplier, evaluates the supplier's default risk and delivery credit, and outputs the prediction results.

2. A supplier default risk assessment method based on hierarchical reinforcement learning as claimed in claim 1, characterized in that: The global task allocation module has n order tasks at time t. When the batch arrives, the execution of the order task is delegated to m factories {f1, f2, ...f m }, solving the task allocation problem based on three steps: Step 1: Task and factory status representation: Define the task status as s j =[jpt,at,ldt,jiw], where jpt represents the total time required for task processing, at represents the task arrival time, ldt represents the latest delivery time of the task, and jiw represents the importance weight of the task; the state of the factory is defined as s f =[apt,noj,awt,nofm], where apt represents the average processing time of tasks in the factory, noj represents the tasks currently assigned but not completed in the factory, awt represents the average waiting time of tasks in the factory, and nofm represents the number of unoccupied machines required for each process in the factory; Through the task state representation module E j and plant status characterization module E f , respectively get the task status table and plant status characterization Step 2: Assignment of tasks to factories: First, for task j u and factory v , by calculating task j u Status representation To the factory v Status representation The attention weight between them is used to establish the strength of the association between the task and the factory, as shown in the following formula: in and They are the calculation matrices of query value and key value in the attention layer respectively; Secondly, by calculating the weighted sum of the factory state representation and the task-to-factory attention weight, we can get the weighted average representation of the task ju on the factory. in is the calculation matrix of the value in the attention layer; At the same time, the attention weights between the weighted average representations of the above tasks for the factory are calculated to establish the degree of attention to other tasks when arranging a factory for a certain task: in and They are the calculation matrices of query value and key value in the attention layer respectively; Then, we calculate the weighted sum of the attention weights of each task i and other tasks to get task j. u The global state representation of in is the calculation matrix of the value in the attention layer; Finally, the task assignment Q network is used to calculate the Q value of assigning task u to factory v: And choose a task u in the ε-greedy way * Assigned to a factory v * , where F Q_global Q network model for global task allocation module, with task-to-factory weights And the global state representation of the task As input; when the order task batch arrives, step 1 and step 2 are executed repeatedly until all current tasks are assigned to each factory; Step 3: Evaluation of task assignment: Let r global (t) is the reward of the global task allocation module, which is calculated by the maximum processing time of the factory and the minimum processing time difference of the factory. The formula is: r global (t)=-(max k∈F (apt. k (t)eat k (t))-min k∈F (apt. k (t)eat k (t))) Where F = {f1, f2, ... f m } represents a collection of factories, apt k (t) and noj k (t) represent the average processing time and number of tasks in factory k at time t respectively.

3. A supplier default risk assessment method based on hierarchical reinforcement learning as claimed in claim 2, characterized in that: The specific implementation method of the local task execution module is: define the production task j to be passed through np j (≤p) processes must be processed before entering the delivery stage. The maximum number of processes is p, and production and processing must strictly follow Each process needs to be completed on a specified type of machine, that is, p types of processes correspond to p types of machines, and the total number of these p types of machines that can be used for production in a factory k is: At time t, factory k received the production task assigned by the upper level. At the same time, there are several tasks in the factory waiting to be assigned to machines to start different processes and tasks that are occupying the machine to perform different processes And the number of p-type machines that are operating normally and idle at time t is Combine the tasks that arrive at factory k at time t and the tasks that are waiting to be processed into a set of tasks to be processed Where n is the number of tasks waiting to be processed; in order to achieve the goal of shortening the factory's completion time to the maximum extent and completing the delivery of orders on time; The module implements the factory task setting based on three steps: Step 1: Task and factory status representation: When assigning machines to tasks waiting for different processes, it is necessary to refer to the status of each task to formulate an assignment plan. Unlike the global task assignment module, different task status information needs to be considered in this module. Therefore, the status of the task in this module is defined as s j =[ptpo,jno,amno,wt,ldt,jiw], where ptpo represents the processing time required for each process of the task, jno represents the next process of the task, amno represents the number of machines required for the next process of the task, wt represents the waiting time before the task executes the next process, ldt represents the latest delivery time of the task, and jiw represents the importance weight of the task; Task state representation module through local task execution module (Different from the task status representation module in the global task allocation) to obtain the task status table Step 2: Assign tasks to machines: First, by calculating the attention weights between tasks, we can obtain the degree of association with other tasks that needs to be considered when assigning machines to each task: in and They are the calculation matrices of query value and key value in the attention layer respectively; By calculating the weighted sum of the attention weights of each task i and other tasks, task j in the local task execution module is obtained. u The global state representation of in is the calculation matrix of the value in the attention layer; Finally, the machine allocation Q network calculates the Q value of the machine required to assign task i to the next process: where F Q_local is the Q network model of the local task execution module, It indicates that if the number of available machines required for the next step of task i is 0, no machine will be assigned to task i, and the Q value will be set to 0; after calculating the Q value, task i* will be assigned to machine m* in an ε-greedy manner; when the order task is assigned to factory k at a certain moment, steps 1 and 2 will be executed repeatedly until all current tasks are assigned or there is no machine that can be assigned to the task; Step 3: Evaluation of task execution: Calculate the reward for lower-level task execution by the time difference of factory task completion: r local (t)=-(C t -C t-1 )-α late n late , where C t is the time point at which the factory completed the last task at time t. This reward reflects the frequency of the factory's delivery, and thus reflects the factory's production efficiency. In addition, n late represents the number of orders that the factory completed delivery at time t but exceeded the delivery time, α late Indicates the penalty coefficient for order timeout.

4. A supplier default risk assessment method based on hierarchical reinforcement learning as claimed in claim 3, characterized in that: The experience replay buffer module reads data in batches from the replay buffer during the model update phase for model update. The specific implementation steps are as follows: First, the global task allocation module is updated: For the global task allocation module, let B be the batch data taken from the global module playback buffer, represents the data of a decision step, where and represents the status of all unassigned tasks and the status of all factories at time t. t represents the allocation action of allocating a task to a factory at time t, r global (t) represents the reward of the global module at time t; the error function of the global task allocation module is as follows: Where Qglobal represents the network of the global task allocation module, b represents the size of the data batch, and γ represents the discount factor; After that, the local task execution module is updated: Let B be the batch data taken from the local module playback buffer, which stores the decision data sampled when all factories arrange production operations. represents the data of a decision step, where a represents the status of all tasks waiting for the machine required for the next step at time t, t represents the allocation action of allocating a machine to a task at time t, r local (t) represents the reward of the local module at time t; the error function of the local task execution module is as follows: Where Qlocal represents the network of the local task execution module as a whole, b represents the size of the data batch, and γ represents the discount factor.

5. A supplier default risk assessment method based on hierarchical reinforcement learning as claimed in claim 4, characterized in that: The supplier default risk assessment module is based on historical and current enterprise order data, simulates multiple batches of orders and the enterprise's production and manufacturing processes through sample estimation, and calculates the supplier's default rate based on the simulated delivery time results. The specific implementation method is: Assume that J1, J2, ..., J N The order batch for the current supplier in the simulation, where For the i-th batch of order tasks, after the global task allocation module and the local task execution module, the default rate of each batch of orders is simulated: where n i is the number of orders in the i-th batch, For orders The simulated completion time, For orders The order delivery period is ; after the supply chain production scheduling model predicts the default rate of multiple batches of orders, the sample estimate of the supplier default risk Θ is obtained Take the sample mean as the sample estimate of supplier default risk Θ, that is: Set a series of standard default risk lines eta1=0.01, eta2=0.05, eta3=0.1, Suppliers with values ​​lower than η1 can be evaluated as low default risk suppliers, while suppliers between η1 and η2, η2 and η3, and above η3 can be evaluated as third-level default risk suppliers, second-level default risk suppliers, and first-level default risk suppliers, respectively.

Images