Spare part scheduling model training method, spare part scheduling method and device

Abstract

The invention relates to the technical field of communication network operation and maintenance, and provides a spare part scheduling model training method and a spare part scheduling method and device.The spare part scheduling method comprises the steps that current spare part scheduling information is collected, and state information is constructed based on the current spare part scheduling information, the spare part scheduling information comprises inventory information, demand prediction information, equipment fault information and transportation resource information; and inputting the constructed state information into a trained spare part scheduling model to obtain an expected value of a long-term accumulated reward which can be obtained by executing each action, and taking the action corresponding to the maximum expected value as a spare part scheduling strategy. Based on the constructed spare part scheduling environment model and data, a BP neural network and reinforcement learning network structure is trained to obtain a spare part scheduling model, the spare part scheduling model is used for spare part scheduling, and efficient, low-cost and intelligent management of communication network spare part supply is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of communication network operation and maintenance, and in particular, relates to a spare parts scheduling model training method, a spare parts scheduling method and a device. Background Art

[0002] As the infrastructure of modern society, the efficient and stable operation of communication networks is crucial to daily life and key industries. Whether it is social entertainment, online office, financial transactions, medical services, or intelligent transportation, the smooth operation of communication networks directly affects the convenience and safety of information transmission, business development, and social life. Taking the financial industry as an example, millisecond-level response speed is crucial for stock trading and electronic payments, and network failures may lead to huge economic losses. In the medical field, network interruptions may delay remote diagnosis and emergency treatment, endangering patients' lives.

[0003] In communication network maintenance, spare parts inventory and scheduling management are key to ensuring network reliability. However, traditional spare parts scheduling models face many challenges and are difficult to adapt to the complex needs of modern communication networks. Traditional scheduling models are usually based on static rules or simple algorithms and lack flexibility. When business demand fluctuates, these models are difficult to adjust quickly. For example, during the e-commerce shopping festival, the surge in network traffic caused a sharp increase in spare parts demand in a short period of time, and the traditional scheduling method could not respond in time, which easily caused network congestion or even paralysis, seriously affecting user experience and business operations.

[0004] As the scale and technical complexity of communication networks increase, scheduling scenarios are becoming increasingly complex, including a large number of spare parts types, multiple warehouse nodes, and complex logistics routes. Traditional algorithms have large computational workloads and slow response times when dealing with these problems, making it difficult to meet actual needs. After a network failure occurs, the time required to generate a scheduling plan often exceeds the acceptable repair window, resulting in long-term network instability, losses, and inconvenience.

[0005] In addition, the applicability of traditional scheduling models is limited, and they are usually optimized for a single type of spare parts demand. Modern communication networks integrate a variety of technologies and equipment, and spare parts demands are diverse and complex, including electronic components, transmission equipment, and switching equipment. In actual operation and maintenance, spare parts demands often involve multiple types, and a single scheduling model cannot effectively respond to them and is difficult to meet the requirements of all-round maintenance.

[0006] Therefore, there is an urgent need for a new spare parts scheduling method to cope with the complexity and dynamics of the communication network operating environment and meet the requirements of efficient, accurate, low-cost and intelligent management of spare parts scheduling. Summary of the invention

[0007] To solve the above problems, the present invention provides a spare parts scheduling model training method, a spare parts scheduling method and a device. Based on the constructed spare parts scheduling environment model and data, the BP neural network and reinforcement learning network structure are trained to obtain a spare parts scheduling model, thereby realizing efficient, low-cost and intelligent management of communication network spare parts supply.

[0008] In a first aspect, a spare parts scheduling model training method is provided, the training method is based on a BP neural network and a reinforcement learning network structure, and includes:

[0009] Construct a multi-objective optimization function based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment;

[0010] Conduct scheduling environment modeling and data preparation: define the state space, quantify, integrate and transform inventory information, demand forecast information, equipment failure information and transportation resource information to obtain state information; determine the action space, number the actions in spare parts supply management, and obtain an action space set, where the action includes dispatching specific spare parts from a specific warehouse to a specified failure location through a specific transportation method; construct a reward function: construct a reward function based on the constructed multi-objective optimization function; data collection and preprocessing, collect logistics time, spare parts risk and scheduling cost data in spare parts supply management, and preprocess the data as input to the multi-objective optimization function;

[0011] Construct an experience replay memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience;

[0012] Randomly extract multiple experiences from the experience replay memory pool as training samples;

[0013] Entering the training phase, the main BP neural network and the target BP neural network are trained using training samples. The main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data.

[0014] Entering the learning phase, the loss function is calculated using the current value calculated by the main BP neural network and the target value calculated by the target BP neural network, and the parameters of the main BP neural network are updated using the gradient of the loss function;

[0015] Repeat the above training phase and learning phase;

[0016] Whenever the above training phase and learning phase complete a certain number of iterations, the parameters of the main BP neural network are synchronized to the target BP neural network, and the next training phase and learning phase are continued until the loss function converges below the preset threshold, or the change amplitude of the loss function is less than the set small value;

[0017] The trained target BP neural network is used as the trained spare parts scheduling model.

[0018] Furthermore, the constructed multi-objective optimization function is V = ω t ×T+ω c ×C+ω r ×R; where V is the multi-objective optimization value, T is the spare parts supply time, C is the spare parts scheduling cost, R is the spare parts risk assessment, ω t ,ω c ,ω r They are the weight of spare parts supply time, the weight of spare parts scheduling cost, and the weight of spare parts risk assessment.

[0019] Furthermore, the reward function is constructed based on the constructed multi-objective optimization function:

[0020]

[0021] Among them, r is the reward value, V base is the benchmark optimization value set based on historical experience and expert evaluation, V t is the multi-objective optimization value corresponding to this action.

[0022] Furthermore, inventory information includes: types of spare parts, number of warehouses and number of spare parts; demand forecast information includes: number of regions and number of spare parts required; fault information includes: type of faulty equipment, fault location and fault severity; transportation resource information includes: available number of transportation tools, current location and transportation capacity parameters.

[0023] Furthermore, inventory information, demand forecast information, equipment failure information, and transportation resource information are quantified, including:

[0024] Assume that there are N types of spare parts distributed in M ​​warehouses, and define the inventory matrix I, whose element I ij represents the inventory quantity of the i-th spare part in the j-th warehouse;

[0025] Assume that there are P regions involved and N types of spare parts need to be considered. Define the demand matrix D, whose elements D ij represents the demand quantity of the jth spare part in the i-th region;

[0026] Assuming that there are P regions involved, the demand matrix represents the number of spare parts required, and the failure matrix indicates the severity of the spare parts. Define the failure matrix F, whose elements F ij represents the failure degree of the jth spare part in the i-th area;

[0027] Assume that there are K types of transportation tools, distributed in M ​​warehouses, and define the inventory matrix T, whose elements T ij Represents the available quantity of the i-th means of transport in the j-th warehouse.

[0028] Furthermore, the actions in spare parts supply management are numbered, including:

[0029] Each action resulting from the deployment of different warehouse-spare parts-fault location combinations is numbered.

[0030] In a second aspect, a spare parts scheduling method is provided, the scheduling method comprising:

[0031] Collect current spare parts scheduling information and build status information based on the current spare parts scheduling information; wherein the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information;

[0032] The constructed state information is input into the trained spare parts scheduling model to obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and the action corresponding to the maximum expected value is used as the spare parts scheduling strategy, wherein the trained spare parts scheduling model is obtained by the above-mentioned training method.

[0033] Furthermore, the scheduling method further includes:

[0034] The status information of the build and spare parts scheduling strategy are saved as experience.

[0035] In a third aspect, a spare parts scheduling model training device is provided, which is based on a BP neural network and a reinforcement learning network structure, and includes: a cost function construction module, a modeling and data preparation module, a memory pool construction module, a training module and a model building module; wherein:

[0036] Cost function building module, used to build multi-objective optimization functions based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment;

[0037] Modeling and data preparation module, used for scheduling environment modeling and data preparation: define state space, quantify, integrate and transform inventory information, demand forecast information, equipment failure information and transportation resource information to obtain state information; determine action space, number actions in spare parts supply management to obtain action space set, where actions include dispatching specific spare parts from specific warehouses to designated failure locations through specific transportation methods; construct reward function: construct reward function based on the constructed multi-objective optimization function; data collection and preprocessing, collect logistics time, spare parts risk and scheduling cost data in spare parts supply management, and preprocess the data as input of the multi-objective optimization function;

[0038] The memory pool construction module is used to construct the experience playback memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience;

[0039] The training module is used to randomly extract multiple experiences from the experience playback memory pool as training samples; enter the training phase, use the training samples to train the main BP neural network and the target BP neural network, and the main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data; enter the learning phase, use the current value calculated by the main BP neural network and the target value calculated by the target BP neural network to calculate the loss function, and use the gradient of the loss function to update the parameters of the main BP neural network; repeat the above training phase and learning phase; whenever the above training phase and learning phase complete a certain number of iterations, synchronize the parameters of the main BP neural network to the target BP neural network, and continue to the next training phase and learning phase until the loss function converges to below the preset threshold, or the change amplitude of the loss function is less than the set small value;

[0040] The model building module is used to use the trained target BP neural network as a trained spare parts scheduling model.

[0041] In a fourth aspect, a spare parts scheduling device is provided, the scheduling device comprising: a collection module and a scheduling strategy determination module; wherein:

[0042] A collection module is used to collect current spare parts scheduling information and build status information based on the current spare parts scheduling information; wherein the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information;

[0043] The scheduling strategy determination module is used to input the constructed state information into the trained spare parts scheduling model, obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and use the action corresponding to the maximum expected value as the spare parts scheduling strategy, wherein the trained spare parts scheduling model is obtained using the above-mentioned training device.

[0044] In a fifth aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0045] Memory, used to store computer programs;

[0046] The processor is used to implement the steps of the above method when executing the program stored in the memory.

[0047] In a sixth aspect, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the steps of the above method are performed when the computer program is executed by a processor.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] Modeling and data preparation for spare parts scheduling environment, training spare parts scheduling model based on BP neural network and reinforcement learning network structure, using spare parts scheduling model, scheduling can improve scheduling efficiency, reduce scheduling cost, achieve priority sorting, adapt to multi-type spare parts demand and provide optimized supply solutions, as follows:

[0050] Improve scheduling efficiency: Build a flexible spare parts scheduling model and optimize it to quickly respond to imbalanced demand, effectively improve the speed and accuracy of spare parts scheduling, reduce fault handling time, and improve the availability of communication networks.

[0051] Reduce scheduling costs: A multi-objective optimization function is constructed by comprehensively considering spare parts supply time, spare parts scheduling costs and spare parts risk assessment, which enables refined management in daily scheduling and reasonable selection of scheduling strategies in emergency fault handling. It can significantly reduce labor costs, transportation costs and inventory costs in the spare parts scheduling process and improve the economic benefits of the enterprise.

[0052] Prioritization: Introducing factors such as spare parts risk and fault severity into model design can reasonably allocate resources based on the severity and urgency of the fault, ensure that high-severity faults are handled in a timely manner, and improve the efficiency and quality of fault recovery.

[0053] Adapt to the needs of multiple types of spare parts: It can handle the scheduling problems of various types of spare parts and is suitable for complex and diverse communication network maintenance scenarios, which improves the versatility and practicality of the algorithm.

[0054] Providing optimized supply solutions: Through multi-objective optimization functions and continuous optimization of scheduling strategies through reinforcement learning, it is possible to find a balance between multiple objectives such as cost, time and risk, provide a systematic and optimized supply solution for communication network spare parts scheduling, and improve the overall operation and maintenance management level.

[0055] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 A schematic diagram of the overall process of network communication spare parts scheduling provided according to an embodiment of the present disclosure is shown;

[0058] Figure 2 The schematic diagram of the classic BP neural network structure according to the embodiment of the present disclosure is shown;

[0059] Figure 3 A schematic diagram of a perceptron at each layer according to an embodiment of the present disclosure is shown;

[0060] Figure 4 A schematic diagram of the overall process of network communication spare parts scheduling according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0062] The embodiment of the present disclosure provides a spare parts scheduling model training method. The network structure of the spare parts scheduling model is based on a BP neural network and a reinforcement learning network structure. Figure 1 As shown, the training method comprises:

[0063] Construct a multi-objective optimization function based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment;

[0064] Conduct scheduling environment modeling and data preparation: define the state space, quantify, integrate and transform inventory information, demand forecast information, equipment failure information and transportation resource information to obtain state information; determine the action space, number the actions in spare parts supply management, and obtain an action space set, where the action includes dispatching specific spare parts from a specific warehouse to a specified failure location through a specific transportation method; construct a reward function: construct a reward function based on the constructed multi-objective optimization function; data collection and preprocessing, collect logistics time, spare parts risk and scheduling cost data in spare parts supply management, and preprocess the data as input to the multi-objective optimization function;

[0065] Construct an experience replay memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience;

[0066] Randomly extract multiple experiences from the experience replay memory pool as training samples;

[0067] Entering the training phase, the main BP neural network and the target BP neural network are trained using training samples. The main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data.

[0068] Entering the learning phase, the loss function is calculated using the current value calculated by the main BP neural network and the target value calculated by the target BP neural network, and the parameters of the main BP neural network are updated using the gradient of the loss function;

[0069] Repeat the above training phase and learning phase;

[0070] Whenever the above training phase and learning phase complete a certain number of iterations, the parameters of the main BP neural network are synchronized to the target BP neural network, and the next training phase and learning phase are continued until the loss function converges below the preset threshold, or the change amplitude of the loss function is less than the set small value;

[0071] The trained target BP neural network is used as the trained spare parts scheduling model.

[0072] The following is a detailed description of the construction of multi-objective optimization functions, environmental modeling and data preparation, and networks based on BP neural networks and reinforcement learning involved in the above training methods:

[0073] 1. Construct a multi-objective optimization function

[0074] The definition of spare parts supply time is that the spare parts supply time is the total time from the issuance of spare parts scheduling instructions to the arrival of spare parts at the designated location and their availability for use. This includes the time to extract spare parts from the warehouse, the time in transit, and the possible time for warehousing inspection and preparation for use. By collecting historical spare parts scheduling data, the average time spent on different types of spare parts in each link is calculated, and a time estimation model is constructed. For example, for transportation time, it can be estimated based on the average speed of different modes of transportation (such as road, rail, and aviation), transportation distance, and historical transportation delay data. Let T 1 is the extraction time from the warehouse, T 2 is the transportation time, T 3 is the time for warehousing inspection and preparation, then the spare parts supply time is T = T 1 +T 2 +T 3 .

[0075] The definition of spare parts scheduling cost is the various expenses directly related to spare parts scheduling. It includes labor costs, inventory costs and possible out-of-stock costs. Among them, labor costs include wages and related benefits of warehouse managers and transportation personnel) and transportation costs; here, transportation costs are calculated according to the mode of transportation and distance, such as road transportation is charged at the freight per kilometer multiplied by the transportation mileage, and air transportation is charged according to flight fees and cargo weight; inventory costs refer to the costs of storing spare parts in the warehouse, taking into account warehouse rent, inventory management system maintenance costs, and interest on funds occupied by spare parts, etc., and are usually calculated by multiplying the inventory cost of each spare part per unit time by the number of spare parts and storage time; out-of-stock costs refer to the estimated economic losses caused by communication network failures caused by out-of-stock spare parts, which can be estimated based on the duration of network failures, affected business volume and unit business value. Let labor cost be C 1 , the transportation cost is C 2 , the inventory cost is C 3 , the out-of-stock cost is C 4 , then the spare parts scheduling cost C = C 1 +C 2 +C 3 +C 4 .

[0076] Spare parts risk is a risk indicator based on the characteristics of spare parts themselves and the degree of impact on the operation of communication networks. It is considered comprehensively from three dimensions: spare parts failure rate, importance and scarcity. Among them, the spare parts failure rate is obtained based on historical failure data statistics. For example, the failure rate is obtained by dividing the number of failures of a spare part in the past year by the total operating time; the importance is determined by scoring factors such as the criticality of the communication network node where the spare part is located and the importance of related businesses. For example, the importance score of spare parts for core network equipment is higher than that of spare parts for edge equipment; the scarcity is based on the supply of spare parts in the market. For example, if there are few suppliers and the production cycle is long, the scarcity is high. Set weights ω for failure rate, importance and scarcity respectively. f ,ω i ,ω s If the equipment failure rate is F, the importance is I, and the scarcity is S, then the spare parts risk R = ω f ×F+ω i ×I+ω s ×S, where ω f ,ω i ,ω s The sum of the three weights is 1.

[0077] Finally, the multi-objective optimization function V = ω is constructed t ×T+ω c ×C+ω r ×R, where ω t ,ω c ,ω r are the weights of spare parts supply time, cost and risk respectively. The constructed multi-objective optimization function can be used to evaluate the quality of a spare parts scheduling plan.

[0078] 2. Environmental modeling and data preparation

[0079] Define the state space: Integrate multiple information to construct state information. In terms of inventory information, the inventory quantity of each spare part in each warehouse and the identification information of the warehouse are recorded in detail and stored in the form of a matrix. If there are N spare parts and M warehouses, the dimension of the inventory matrix I is N×M. The demand forecast information is based on the in-depth analysis of historical data, equipment operation status monitoring and network traffic trend research. It predicts the demand quantity of various spare parts in different regions in a specific period of time in the future to form a demand matrix. Assuming that P regions and N spare parts are involved, the dimension of the demand matrix D is P×N. Equipment fault information is collected immediately when a fault occurs, including the type of faulty equipment (according to the pre-set classification code), the fault location (accurate to the base station number or the location of the computer room) and the severity of the fault (quantified by numerical values, such as 1-5 levels from light to heavy), and combined into a fault matrix F. Transportation resource information monitors the available number of transportation tools (such as vehicles, drones, etc.) and the location of their warehouses in real time to construct a transportation resource matrix T. Finally, the inventory matrix, demand vector, fault vector and transportation resource vector are integrated and converted to obtain a unified state vector S as the input of the BP neural network.

[0080] (a) Inventory matrix:

[0081] Suppose there are N types of spare parts distributed in M ​​warehouses. Define an inventory matrix I, whose element I ij Represents the inventory quantity of the i-th spare part in the j-th warehouse. That is:

[0082]

[0083] The dimension of the inventory matrix is ​​N×M.

[0084] (b) Demand Matrix:

[0085] Assume that there are P regions involved and the demand for N types of spare parts needs to be considered. Define the demand matrix D, whose elements D ij represents the demand quantity of the jth spare part in the ith region. The dimension of the demand vector is P×N, which can be expressed as:

[0086]

[0087] (c) Fault matrix:

[0088] Assuming that there are P regions involved, the demand matrix represents the number of spare parts required, and the failure matrix represents the severity of the spare parts. Define the failure matrix F, whose elements F ij It represents the fault level of the jth spare part in the i-th area (taking the most serious fault level of this type of spare part). The dimension of the fault vector is P×N, which can be expressed as:

[0089]

[0090] (d) Transportation resource matrix:

[0091] Suppose there are K types of transportation tools, distributed in M ​​warehouses. Define the inventory matrix T, whose elements T ij It represents the available quantity of the i-th transportation tool in the j-th warehouse. That is:

[0092]

[0093] Finally, the inventory matrix, demand vector, fault vector and transportation resource vector are integrated and transformed to obtain a unified state vector S, namely:

[0094] S=(IDFT)

[0095] Determine the action space: Identify all possible actions in spare parts supply management and number them. Actions include dispatching specific spare parts from a specific warehouse to a specified fault location via a specific transportation method, such as action number A. 1 -A W Corresponding to different warehouse-transportation mode-spare parts-fault location combinations, the action space set is recorded as A, which can be expressed as:

[0096] A=(A 1 A 2 ....A W )

[0097] Suppose there are m warehouses, q transportation modes, n spare parts, and p possible locations of failure in the system. Then the number of possible combinations of allocating a specific spare part from a specific warehouse to a specified failure location is m×q×n×p. For these combinations, use action number A 1 -A W To express it, where W = m × q × n × p. For example, A 1 Indicates that the first type of spare parts is dispatched from the first warehouse to the first fault location via the first mode of transportation; A 2 It means that the first spare part is deployed from the first warehouse to the second fault location via the second mode of transportation; and so on, until all possible deployment combinations of warehouse-transportation mode-spare part-fault location are covered.

[0098] Construct reward function: Reward is the feedback signal from the environment to the action after the agent performs a certain action, in order to verify the effectiveness of the action. Previously, a multi-objective optimization function V = ω has been constructed to evaluate a spare parts scheduling action. t ×T+ω c ×C+ω r ×R, where T is the spare parts supply time, C is the spare parts scheduling cost, R is the spare parts risk assessment, ω t ,ω c ,ωr are the weights of spare parts supply time, cost, and risk respectively. The smaller the V value, the better the corresponding scheduling action. When an incident occurs suddenly and emergency scheduling is required, the time weight ω should be increased. t Similarly, when cost needs to be controlled, the cost weight ω needs to be increased. c When you need to control risks, you need to improve r In order to judge the quality of an action, a benchmark optimization value V needs to be set based on historical experience and expert evaluation. base , so the reward function is constructed based on this r is the reward value, when the cost of this action is V t When the cost is lower than the benchmark optimization value, the corresponding reward is positive, and the lower the cost, the higher the reward. When the cost is higher than the benchmark optimization value, the corresponding reward is negative, and the higher the cost, the lower the corresponding reward. Therefore, the constructed scheduling model can continuously adjust its own parameters through feedback until a satisfactory effect is achieved.

[0099] Data collection and preprocessing: Collect relevant data such as logistics time, spare parts risk, and scheduling cost from the communication network spare parts management system, clean and organize the data, and remove outliers and erroneous data. Then normalize and standardize the collected data to make it meet the input requirements of the BP neural network, such as converting logistics time into relative time ratios and mapping spare parts risks and scheduling costs to specific numerical ranges.

[0100] 3. Network based on BP neural network and reinforcement learning

[0101] like Figure 2 The figure shows the structure of a classic BP neural network. The number of neurons in the input layer n is determined by the dimension of the state vector S. input , if the state vector length is L, then the input layer sets L neurons to receive the complete system state information. The perceptron framework is as follows Figure 3 shown.

[0102] The hidden layer adopts a multi-layer structure to enhance the network's expressiveness. The number of neurons in the first hidden layer is based on the empirical formula (where n eff The estimated effective information dimension of the output layer is preliminarily determined, and the activation function uses the ReLU function, that is, f ReLU (x) = max(0, x) to accelerate the convergence of network training and enhance the nonlinear mapping ability. The number of neurons in the second hidden layer is appropriately reduced based on the first layer, such as The activation function uses the Sigmoid function Used to further transform and map the data. More hidden layers can be added according to actual needs.

[0103] The number of neurons in the output layer is equal to the size of the action space A, that is, n output =W. The output of each neuron represents the estimated Q value of the corresponding action in the current state, that is, the expected value of the long-term cumulative reward that can be obtained by executing the action. The output layer does not use additional activation functions to maintain the continuity and interpretability of the Q value, which is convenient for the application of subsequent reinforcement learning algorithms.

[0104] (a) Experience replay mechanism:

[0105] Construct an experience playback memory pool D to store the experience data during the interaction between the agent (the decision module based on BP neural network in this system) and the environment. The experience data is in the form of four tuples (s t , a t , r t ,s t+1 ) format, where s t represents the system state at time t, a t is the action taken in this state, r t is the immediate reward after executing the action, s t+1 It is the next system state after the action is executed.

[0106] During the operation of the system, each time the agent interacts with the environment to generate new experience, it is stored in the experience playback memory pool D. When the memory pool reaches a certain capacity C, during the training phase, a batch of experience data is randomly sampled from the memory pool, such as the number of samples is B. This random sampling method breaks the temporal correlation between the experience data, improves the diversity of training data, and enhances the generalization ability of the network.

[0107] (b) Target network and main network architecture:

[0108] In addition to the BP neural network for real-time Q-value estimation constructed above (called the main network, with parameter θ), a target network with the same structure is created (with parameter θ - ). The role of the target network is to provide a relatively stable target Q value calculation benchmark to avoid the main network from being unstable due to the continuous update of Q value estimation during the training process.

[0109] The parameters θ of the main network are copied to the target network periodically (e.g., after every T training steps), so that the target network can track the learning progress of the main network, but maintain a certain lag, thereby ensuring the relative stability of the target Q value.

[0110] The following is a description of the specific details of the training process:

[0111] 1. Initialization

[0112] Initialize the parameters of the main network and the target network by randomly initializing the weights;

[0113] Clear the experience replay memory pool.

[0114] 2. Filling the Experience Replay Memory Pool

[0115] State initialization: given an initial state s of the state space t ;

[0116] Time step loop: The time step is continuously executed in each round until the task is completed or the maximum time step is reached (for example, set to 100 steps);

[0117] The sign of task completion: the elements in the demand matrix and the elements in the fault matrix in the state matrix are all 0.

[0118] Action selection: This invention adopts the exploration-balance (ε-greedy) method to select action a t , set ε, that is, there is a 1-ε probability of selecting the action with the largest Q value, and the probability of ε is to randomly select other actions.

[0119] Environmental interaction: After executing action a, the environment feedbacks the new state s t+1 and reward r t ;

[0120] Storage experience: Experience tuple (s t , a t , r t ,s t+1 ) is stored in the experience replay memory pool D.

[0121] 3. Sampling and training (after certain conditions are met): When the number of experiences stored in the experience replay memory pool is greater than the number of small batch samples, sampling training begins. Several samples are randomly sampled from the experience replay memory pool.

[0122] (a) Calculate the sample (s) using the main network t , a t , r t ,s t+1 ) in state s t Take action a t The obtained Q value estimate Q(s t , a t ,θ), and use the target network to calculate in the new state s t+1 The maximum Q value estimation under

[0123] (b) Update the formula based on the Q-learning algorithm of reinforcement learning Where γ is the discount factor, ranging from 0 to 1, which is used to measure the impact of future rewards on current decisions, and θ - are the parameters of the target network;

[0124] (c) Using the mean square error (MSE) loss function: (Here B is the number of batch samples), calculate the loss, and then use the gradient descent algorithm with a learning rate of α to update the parameters θ of the main network;

[0125] (d) Based on the calculated loss function, the gradient of the loss function with respect to the main network parameter θ is calculated through the back propagation algorithm.

[0126] In the back propagation process, according to the chain rule of BP neural network, the error gradient of each layer of neurons is calculated in turn. For the jth neuron in the lth layer, its error gradient is The calculation of is as follows:

[0127] Output layer (L = L output ): in is the derivative of the output layer activation function (since there is no additional activation function in the output layer, the derivative here is 1), is the input of the jth neuron in the output layer.

[0128] Hidden layer (L <L output ): Where k is the neuron index of the l+1th layer, is the weight from the jth neuron in the lth layer to the kth neuron in the l+1th layer, f l′ is the derivative of the activation function of the lth layer.

[0129] (e) Based on the calculated error gradient, the gradient descent algorithm is used to update the weight ω and bias b parameters of the main network: Where α is the learning rate, is the output of the i-th neuron in the l-1th layer.

[0130] (f) Update the target network: After each training cycle, copy the parameters of the current main network to the target network.

[0131] Through the above training, a trained main network model is obtained. The performance of the main network model needs to be evaluated, which specifically includes the following steps:

[0132] (a) Use the excellent historical manual scheduling solutions evaluated by experts as the test data set to evaluate the performance of the trained BP neural network model (target network). The test data set should cover various types of spare parts supply scenarios, including different inventory status, demand patterns, failure conditions, and transportation resource configurations.

[0133] (b) Input the system states in the test data set into the trained network in sequence to obtain the spare parts supply action recommended by the network and its corresponding Q value estimate. Compare it with the pre-marked optimal action in the test data set (which can be determined by historical experience or expert knowledge) and calculate the action matching accuracy index Where T test is the number of test data set samples, is an indicator function, which is 1 if the predicted action is the same as the optimal action and 0 otherwise.

[0134] When the network model passes the performance evaluation and meets the actual application requirements, it can be deployed to the real communication network spare parts supply management system. Figure 4 The overall process of scheduling training and scheduling method is shown in FIG. The spare parts scheduling method at this time includes the following steps:

[0135] Collect current spare parts scheduling information, where the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information;

[0136] Build status information based on current spare parts scheduling information;

[0137] The status information constructed here is Figure 4 The new data in the input.

[0138] The constructed state information is input into the trained spare parts scheduling model based on BP neural network and reinforcement learning. The spare parts scheduling model based on BP neural network and reinforcement learning outputs the Q value of each action according to the input state information, and selects the action with the largest Q value as the spare parts supply action. The selected action is the communication spare parts scheduling strategy, that is, Figure 4 Spare parts scheduling scheme in.

[0139] Communication spare parts scheduling strategies include determining which spare parts to allocate from which warehouse, which transportation tools to choose, and whether to start horizontal transfer. After the action is completed, the new state is fed back to the model, and the model gives the next action, and so on, until the task is completed.

[0140] (c) All experiences generated by this interaction (s t , a t , r t ,s t+1) is stored in the experience replay memory pool for subsequent continuous training and optimization of the network. Data is sampled from the experience replay memory pool regularly (e.g., every certain period of time G) to fine-tune the BP neural network to adapt to the dynamic changes in the communication network environment, fluctuations in spare parts demand, and changes in transportation resources, ensuring that the BP neural network can always provide efficient and intelligent spare parts supply decision support.

[0141] The following is an example to illustrate the above scheme:

[0142] Assumption: In a certain scenario, there are 2 warehouses (Warehouse A, Warehouse B), 2 types of spare parts (Spare Part A, Spare Part B), 2 maintenance areas (Area A, Area B), and 2 modes of transportation (Mode A, Mode B);

[0143] Warehouse A has one spare part A and two spare parts B. Warehouse B has two spare parts A and one spare part B, so the inventory matrix is

[0144] Maintenance area A detects a fault and needs one spare part A with a fault level of 2 and one spare part B with a fault level of 1. Maintenance area B has no fault, so the demand matrix is The fault matrix is

[0145] Warehouse A has only mode A and no mode B for transportation, and warehouse B has only mode B and no mode A for transportation. Then the transportation matrix is

[0146] This constitutes the overall state matrix

[0147] Correspondingly, the action space is:

[0148] Action 1: Warehouse A delivers spare part A to region A via transport method A;

[0149] Action 2: Warehouse A delivers spare part A to region B via transport method A;

[0150] Action 3: Warehouse A delivers spare part B to region A via transport method A:; .......

[0152] Action 16 (2*2*2*2): Warehouse B delivers spare part B to region B via transport method B.

[0153] According to the state space, the action matrix is ​​(1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1), where 1 means that the action is executed conditionally and 0 means that the action exists in theory but cannot be performed. For example, some actions cannot be performed because warehouse A does not have transportation method B.

[0154] Since the main network has been initialized and the input and output formats have been set, we now fill the experience replay memory pool and flatten the overall state matrix to (1, 2, 2, 1, 1, 1, 0, 0, 2, 1, 0, 0, 1, 0, 0, 1) as the input of the main network. The Q value corresponding to the output action matrix of the main network is assumed to be (0.9, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.3, 0.5, 0.4, 0.5, 0.2). Then, action a is selected according to the exploration-balance (ε-greedy) method. t , setting ε to 0.1 means that there is a 90% probability of selecting the action with the largest Q value, that is, action 1 with a Q value of 0.9, and a 10% probability of randomly selecting other actions. This gives us the current state (s t , a t ), then define the reward function based on the multi-objective optimization function, for action a t , get V as 1000, set the benchmark optimization value V base If the reward is 2000, the reward r is 0.5, and after executing this action, the state matrix becomes (0, 2, 2, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1), which is s t+1 , thus forming a sample (s t , a t , r t ,s t+1 ), can be stored in the experience replay memory pool.

[0155] Then this new state continues to be passed into the main network, and new actions, rewards and new states after executing this action continue to be obtained. When a certain number of samples are accumulated, certain samples can be randomly selected from the experience replay memory pool as training samples for training. After a period of training, the main network parameters are passed to the target network.

[0156] In this way, the target network can be obtained, which can theoretically provide the optimal action for each step given the initial state until the task is completed.

[0157] The previous historical manual scheduling scheme under the same initial state can be compared with the scheme provided by the target network. If it is not as good as the historical manual scheduling scheme, continue training.

[0158] The spare parts scheduling method of the disclosed embodiment can handle daily scheduling and sudden failures. For daily scheduling needs, since the scheduling model is trained based on a multi-objective optimization function, the spare parts supply time and cost can be optimized. Under the premise of meeting the normal maintenance needs of the communication network, the spare parts inventory and scheduling plan are reasonably arranged to reduce inventory costs and transportation costs, and improve the turnover rate and utilization rate of spare parts. For sudden failures, the time weight can be increased so that the scheduling model can adopt lateral transfer and emergency distribution strategies to ensure the rapid scheduling of multiple types of spare parts. By establishing a lateral transfer mechanism with surrounding sites or warehouses, the required spare parts can be quickly deployed and the spare parts supply time can be shortened.

[0159] Based on the above training method, the embodiment of the present disclosure also provides a device corresponding to the above training method, the training device is based on the BP neural network and the reinforcement learning network structure, including: a cost function construction module, a modeling and data preparation module, a memory pool construction module, a training module and a model building module; wherein:

[0160] Cost function building module, used to build multi-objective optimization functions based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment;

[0161] Modeling and data preparation module, used for scheduling environment modeling and data preparation: define state space, quantify, integrate and transform inventory information, demand forecast information, equipment failure information and transportation resource information to obtain state information; determine action space, number actions in spare parts supply management to obtain action space set, where actions include dispatching specific spare parts from specific warehouses to designated failure locations through specific transportation methods; construct reward function: construct reward function based on the constructed multi-objective optimization function; data collection and preprocessing, collect logistics time, spare parts risk and scheduling cost data in spare parts supply management, and preprocess the data as input of the multi-objective optimization function;

[0162] The memory pool construction module is used to construct the experience playback memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience;

[0163] The training module is used to randomly extract multiple experiences from the experience playback memory pool as training samples; enter the training phase, use the training samples to train the main BP neural network and the target BP neural network, and the main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data; enter the learning phase, use the current value calculated by the main BP neural network and the target value calculated by the target BP neural network to calculate the loss function, and use the gradient of the loss function to update the parameters of the main BP neural network; repeat the above training phase and learning phase; whenever the above training phase and learning phase complete a certain number of iterations, synchronize the parameters of the main BP neural network to the target BP neural network, and continue to the next training phase and learning phase until the loss function converges to below the preset threshold, or the change amplitude of the loss function is less than the set small value;

[0164] The model building module is used to use the trained target BP neural network as a trained spare parts scheduling model.

[0165] Based on the above scheduling method, the embodiment of the present disclosure also provides a device corresponding to the above scheduling method, and the scheduling device includes: a collection module and a scheduling strategy determination module;

[0166] A collection module is used to collect current spare parts scheduling information and build status information based on the current spare parts scheduling information, wherein the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information;

[0167] The scheduling strategy determination module is used to input the constructed state information into the trained spare parts scheduling model, obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and use the action corresponding to the maximum expected value as the spare parts scheduling strategy, wherein the trained spare parts scheduling model is obtained using the above-mentioned training device.

[0168] Based on the same inventive concept as the above disclosed content, the present disclosure also provides an electronic device accordingly. The electronic device of the present disclosure embodiment includes at least one electrically connected processor and only one memory, the memory is electrically connected to the processor, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method as described above.

[0169] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. An indirect connection method can be applied to the embodiments of the present disclosure as long as the purpose of the present disclosure is achieved.

[0170] Based on the same inventive concept, the present disclosure also provides a computer storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0171] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A spare parts scheduling model training method, characterized in that: The training method is based on BP neural network and reinforcement learning network structure, including: Construct a multi-objective optimization function based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment; Conduct scheduling environment modeling and data preparation: define the state space, quantify, integrate and transform inventory information, demand forecast information, equipment failure information and transportation resource information to obtain state information; determine the action space, number the actions in spare parts supply management, and obtain an action space set, where the action includes dispatching specific spare parts from a specific warehouse to a specified failure location through a specific transportation method; construct a reward function: construct a reward function based on the constructed multi-objective optimization function; data collection and preprocessing, collect logistics time, spare parts risk and scheduling cost data in spare parts supply management, and preprocess the data as input to the multi-objective optimization function; Construct an experience replay memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience; Randomly extract multiple experiences from the experience replay memory pool as training samples; Entering the training phase, the main BP neural network and the target BP neural network are trained using training samples. The main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data. Entering the learning phase, the loss function is calculated using the current value calculated by the main BP neural network and the target value calculated by the target BP neural network, and the parameters of the main BP neural network are updated using the gradient of the loss function; Repeat the above training phase and learning phase; Whenever the above training phase and learning phase complete a certain number of iterations, the parameters of the main BP neural network are synchronized to the target BP neural network, and the next training phase and learning phase are continued until the loss function converges below the preset threshold, or the change amplitude of the loss function is less than the set small value; The trained target BP neural network is used as the trained spare parts scheduling model.

2. The training method according to claim 1, characterized in that: The constructed multi-objective optimization function is V = ω t ×T+ω c ×C+ω r ×R; where V is the multi-objective optimization value, T is the spare parts supply time, C is the spare parts scheduling cost, R is the spare parts risk assessment, ω t ,ω c ,ω r They are the weight of spare parts supply time, the weight of spare parts scheduling cost, and the weight of spare parts risk assessment.

3. The training method according to claim 1 or 2, characterized in that: The reward function constructed based on the constructed multi-objective optimization function is: Among them, r is the reward value, Y base is the benchmark optimization value set based on historical experience and expert evaluation, V t is the multi-objective optimization value corresponding to this action.

4. The training method according to claim 1 or 2, characterized in that: Inventory information includes: spare parts types, warehouse quantity and spare parts quantity; demand forecast information includes: number of regions and spare parts demand quantity; fault information includes: faulty equipment type, fault location and fault severity; transportation resource information includes: available quantity of transportation tools, current location and transportation capacity parameters.

5. The training method according to claim 4, characterized in that: It is characterized in that Quantify inventory information, demand forecast information, equipment failure information, and transportation resource information, including: Assume that there are N types of spare parts distributed in M ​​warehouses, and define the inventory matrix I, whose element I ij represents the inventory quantity of the i-th spare part in the j-th warehouse; Assume that there are P regions involved and N types of spare parts need to be considered. Define the demand matrix D, whose elements D ij represents the demand quantity of the jth spare part in the i-th region; Assuming that P areas are involved, the demand matrix represents the number of spare parts required, and the failure matrix indicates the severity of the spare parts. Define the failure matrix F, whose elements F ij represents the failure degree of the jth spare part in the i-th area; Assume that there are K types of transportation tools, distributed in M ​​warehouses, and define the inventory matrix T, whose elements T ij Represents the available quantity of the i-th means of transport in the j-th warehouse.

6. The training method according to claim 1 or 2, characterized in that: Number the actions in spare parts supply management, including: Each action resulting from the deployment of different warehouse-spare parts-fault location combinations is numbered.

7. A spare parts scheduling method, characterized in that: The scheduling method comprises: Collect current spare parts scheduling information and build status information based on the current spare parts scheduling information; wherein the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information; The constructed state information is input into the trained spare parts scheduling model to obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and the action corresponding to the maximum expected value is used as the spare parts scheduling strategy, wherein the trained spare parts scheduling model is obtained using the training method described in any one of claims 1-6.

8. The scheduling method according to claim 7, characterized in that: The scheduling method further includes: The status information of the build and spare parts scheduling strategy are saved as experience.

9. A spare parts scheduling model training device, characterized in that: The training device is based on the BP neural network and reinforcement learning network structure, including: a cost function construction module, a modeling and data preparation module, a memory pool construction module, a training module and a model building module; wherein: Cost function building module, used to build multi-objective optimization functions based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment; Modeling and data preparation module, used for scheduling environment modeling and data preparation: define state space, quantify, integrate and transform inventory information, demand forecast information, equipment failure information and transportation resource information to obtain state information; determine action space, number actions in spare parts supply management to obtain action space set, where actions include dispatching specific spare parts from specific warehouses to designated failure locations through specific transportation methods; construct reward function: construct reward function based on the constructed multi-objective optimization function; data collection and preprocessing, collect logistics time, spare parts risk and scheduling cost data in spare parts supply management, and preprocess the data as input of the multi-objective optimization function; The memory pool construction module is used to construct the experience playback memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience; The training module is used to randomly extract multiple experiences from the experience playback memory pool as training samples; enter the training phase, use the training samples to train the main BP neural network and the target BP neural network, and the main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data; enter the learning phase, use the current value calculated by the main BP neural network and the target value calculated by the target BP neural network to calculate the loss function, and use the gradient of the loss function to update the parameters of the main BP neural network; repeat the above training phase and learning phase; whenever the above training phase and learning phase complete a certain number of iterations, synchronize the parameters of the main BP neural network to the target BP neural network, and continue to the next training phase and learning phase until the loss function converges to below the preset threshold, or the change amplitude of the loss function is less than the set small value; The model building module is used to use the trained target BP neural network as a trained spare parts scheduling model.

10. A spare parts scheduling device, characterized in that: The scheduling device includes: a collection module and a scheduling strategy determination module; wherein: A collection module is used to collect current spare parts scheduling information and build status information based on the current spare parts scheduling information; wherein the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information; A scheduling strategy determination module is used to input the constructed status information into a trained spare parts scheduling model to obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and use the action corresponding to the maximum expected value as the spare parts scheduling strategy, wherein the trained spare parts scheduling model is obtained using the training device described in claim 9.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing the steps of any method described in claims 1-7 when executing a program stored in a memory.

12. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any method of claims 1-7 are implemented.

Images