A spare parts scheduling model training method, spare parts scheduling method and device

Abstract

The present disclosure relates to the field of communication network operation and maintenance technology, and provides a spare parts scheduling model training method, spare parts scheduling method, and apparatus. The spare parts scheduling method includes collecting current spare parts scheduling information, constructing status information based on the current spare parts scheduling information, wherein the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information, and transportation resource information; inputting the constructed status information into a trained spare parts scheduling model to obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and using the action corresponding to the maximum expected value as the spare parts scheduling strategy. Based on the constructed spare parts scheduling environment model and data, a BP neural network and reinforcement learning network structure are trained to obtain a spare parts scheduling model, and the spare parts scheduling model is used to perform spare parts scheduling, thereby achieving efficient, low-cost, and intelligent management of the spare parts supply of communication networks.

Description

Technical Field

[0001] The present disclosure belongs to the field of communication network operation and maintenance technology, and in particular relates to a spare parts scheduling model training method, a spare parts scheduling method and a spare parts scheduling device. Background Art

[0002] As the infrastructure of modern society, the efficient and stable operation of communication networks is crucial to daily life and key industries. Whether it's social entertainment, online work, financial transactions, medical services, or intelligent transportation, the smooth operation of communication networks directly impacts information transmission, business operations, and the convenience and security of social life. For example, in the financial industry, millisecond-level response speeds are crucial for stock trading and electronic payments, and network failures can result in significant economic losses. In the medical field, network outages can delay remote diagnosis and emergency treatment, endangering patients' lives.

[0003] In communications network maintenance, spare parts inventory and scheduling management are critical to ensuring network reliability. However, traditional spare parts scheduling models face numerous challenges and struggle to adapt to the complex demands of modern communications networks. Traditional scheduling models are often based on static rules or simple algorithms, lacking flexibility. These models struggle to adapt quickly to fluctuations in service demand. For example, during e-commerce shopping festivals, surges in network traffic lead to a sharp increase in spare parts demand within a short period of time. Traditional scheduling methods are unable to respond in a timely manner, easily causing network congestion or even paralysis, severely impacting user experience and service operations.

[0004] As communication networks scale and become more complex, scheduling scenarios are becoming increasingly complex, necessitating factors such as a large number of spare parts, multiple warehouse nodes, and complex logistics routes. Traditional algorithms, tackling these challenges, suffer from high computational complexity and slow response times, making them inadequate for practical needs. After a network failure, the time required to generate a scheduling solution often exceeds the acceptable repair window, leading to prolonged network instability, losses, and inconvenience.

[0005] Furthermore, traditional scheduling models have limited applicability, typically optimizing for a single type of spare parts requirement. Modern communications networks integrate diverse technologies and equipment, resulting in diverse and complex spare parts requirements, including electronic components, transmission equipment, and switching equipment. In actual operations and maintenance, spare parts requirements often encompass multiple types, making a single scheduling model ineffective and unable to meet comprehensive maintenance requirements.

[0006] Therefore, there is an urgent need for a new spare parts scheduling method to cope with the complexity and dynamics of the communication network operating environment and meet the requirements of efficient, accurate, low-cost and intelligent management of spare parts scheduling. Summary of the Invention

[0007] To solve the above problems, the present disclosure provides a spare parts scheduling model training method, spare parts scheduling method and device. Based on the constructed spare parts scheduling environment model and data, the BP neural network and reinforcement learning network structure are trained to obtain a spare parts scheduling model, thereby realizing efficient, low-cost and intelligent management of communication network spare parts supply.

[0008] In a first aspect, a spare parts scheduling model training method is provided. The training method is based on a BP neural network and a reinforcement learning network structure, and includes:

[0009] Construct a multi-objective optimization function based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment;

[0010] Scheduling environment modeling and data preparation: Define the state space, quantify, integrate, and transform inventory information, demand forecast information, equipment failure information, and transportation resource information to obtain state information; determine the action space, number the actions in spare parts supply management to obtain an action space set, where an action includes dispatching a specific spare part from a specific warehouse to a specified failure location via a specific transportation method; construct a reward function: construct a reward function based on the constructed multi-objective optimization function; data collection and preprocessing: collect logistics time, spare parts risk, and scheduling cost data in spare parts supply management, and preprocess the data as input to the multi-objective optimization function;

[0011] Construct an experience replay memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience;

[0012] Randomly extract multiple experiences from the experience replay memory pool as training samples;

[0013] Entering the training phase, the main BP neural network and the target BP neural network are trained using training samples. The main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data.

[0014] Entering the learning phase, the loss function is calculated using the current value calculated by the main BP neural network and the target value calculated by the target BP neural network, and the parameters of the main BP neural network are updated using the gradient of the loss function;

[0015] Repeat the above training and learning phases;

[0016] Whenever the above training and learning phases complete a certain number of iterations, the parameters of the main BP neural network are synchronized to the target BP neural network, and the next training and learning phases are continued until the loss function converges below the preset threshold, or the change in the loss function is less than the set minimum value;

[0017] The trained target BP neural network is used as the trained spare parts scheduling model.

[0018] Furthermore, the constructed multi-objective optimization function is V = ω t ×T+ω c ×C+ω r ×R; where V is the multi-objective optimization value, T is the spare parts supply time, C is the spare parts scheduling cost, R is the spare parts risk assessment, ω t 、ω c 、ω r They are the weight of spare parts supply time, the weight of spare parts scheduling cost, and the weight of spare parts risk assessment.

[0019] Furthermore, the reward function is constructed based on the constructed multi-objective optimization function:

[0020]

[0021] Among them, r is the reward value, V base is the benchmark optimization value set based on historical experience and expert evaluation, V t is the multi-objective optimization value corresponding to this action.

[0022] Furthermore, inventory information includes: spare parts types, warehouse quantity and spare parts quantity; demand forecast information includes: number of regions and spare parts demand quantity; fault information includes: fault equipment type, fault location and fault severity; transportation resource information includes: available number of transportation tools, current location and transportation capacity parameters.

[0023] Furthermore, inventory information, demand forecast information, equipment failure information, and transportation resource information are quantified, including:

[0024] Assume that there are N types of spare parts distributed in M ​​warehouses, and define the inventory matrix I, whose element I ij represents the inventory quantity of the i-th spare part in the j-th warehouse;

[0025] Assume that there are P regions involved and the demand for N types of spare parts needs to be considered. Define the demand matrix D, whose elements D ij represents the demand quantity of the jth type of spare parts in the i-th region;

[0026] Assuming that there are P regions involved, the demand matrix represents the number of spare parts required, and the failure matrix represents the severity of the spare parts. Define the failure matrix F, whose elements F ij represents the failure degree of the jth spare part in the i-th area;

[0027] Assume that there are K types of transportation tools, distributed in M ​​warehouses, and define the inventory matrix T, whose elements T ij represents the available quantity of the i-th means of transport in the j-th warehouse.

[0028] Furthermore, the actions in spare parts supply management are numbered, including:

[0029] Number each action resulting from the deployment of different warehouse-spare parts-fault location combinations.

[0030] In a second aspect, a spare parts scheduling method is provided, the scheduling method comprising:

[0031] Collect current spare parts scheduling information and build status information based on the current spare parts scheduling information; the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information;

[0032] The constructed state information is input into the trained spare parts scheduling model to obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and the action corresponding to the maximum expected value is used as the spare parts scheduling strategy, where the trained spare parts scheduling model is obtained by the above-mentioned training method.

[0033] Furthermore, the scheduling method further includes:

[0034] The status information of the build and spare parts scheduling strategy are saved as experience.

[0035] In a third aspect, a spare parts scheduling model training device is provided. The training device is based on a BP neural network and a reinforcement learning network structure, and includes: a cost function construction module, a modeling and data preparation module, a memory pool construction module, a training module, and a model building module; wherein:

[0036] Cost function building module, used to build multi-objective optimization functions based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment;

[0037] The modeling and data preparation module is used to model and prepare the scheduling environment: define the state space, quantify, integrate, and transform inventory information, demand forecast information, equipment failure information, and transportation resource information to obtain state information; determine the action space, number the actions in spare parts supply management to obtain an action space set, where an action includes dispatching a specific spare part from a specific warehouse to a specified failure location via a specific transportation method; construct a reward function based on the constructed multi-objective optimization function; and collect and preprocess data on logistics time, spare parts risk, and scheduling costs in spare parts supply management, and preprocess the data as input to the multi-objective optimization function.

[0038] The memory pool construction module is used to build an experience replay memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience;

[0039] The training module is used to randomly extract multiple experiences from the experience replay memory pool as training samples; enter the training phase, use the training samples to train the main BP neural network and the target BP neural network, and the main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data; enter the learning phase, use the current value calculated by the main BP neural network and the target value calculated by the target BP neural network to calculate the loss function, and use the gradient of the loss function to update the parameters of the main BP neural network; repeat the above training phase and learning phase; whenever the above training phase and learning phase complete a certain number of iterations, synchronize the parameters of the main BP neural network to the target BP neural network, and continue to the next training phase and learning phase until the loss function converges below the preset threshold, or the change amplitude of the loss function is less than the set small value;

[0040] The model building module is used to use the trained target BP neural network as a trained spare parts scheduling model.

[0041] In a fourth aspect, a spare parts scheduling device is provided, the scheduling device comprising: a collection module and a scheduling strategy determination module; wherein:

[0042] A collection module is used to collect current spare parts scheduling information and build status information based on the current spare parts scheduling information; wherein the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information;

[0043] The scheduling strategy determination module is used to input the constructed state information into the trained spare parts scheduling model, obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and use the action corresponding to the maximum expected value as the spare parts scheduling strategy, wherein the trained spare parts scheduling model is obtained using the above-mentioned training device.

[0044] In a fifth aspect, an electronic device is provided, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0045] Memory for storing computer programs;

[0046] The processor is configured to implement the steps of the above method when executing the program stored in the memory.

[0047] In a sixth aspect, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the steps of the above method are performed when the computer program is executed by a processor.

[0048] Compared with the prior art, the present disclosure has the following advantages:

[0049] Modeling and data preparation are performed on the spare parts scheduling environment. A spare parts scheduling model is trained based on the BP neural network and reinforcement learning network structure. Using the spare parts scheduling model, scheduling can improve scheduling efficiency, reduce scheduling costs, implement priority sorting, adapt to the needs of multiple types of spare parts, and provide optimized supply solutions. The details are as follows:

[0050] Improve scheduling efficiency: Build a flexible spare parts scheduling model and optimize it to quickly respond to demand imbalances, effectively improve the speed and accuracy of spare parts scheduling, reduce fault handling time, and improve communication network availability.

[0051] Reduce scheduling costs: A multi-objective optimization function is constructed by comprehensively considering spare parts supply time, spare parts scheduling costs, and spare parts risk assessment. This allows for refined management in daily scheduling and the rational selection of scheduling strategies in emergency fault handling. This can significantly reduce labor costs, transportation costs, and inventory costs in the spare parts scheduling process, thereby improving the company's economic benefits.

[0052] Prioritization: Introducing factors such as spare part risk and fault severity into model design allows for the rational allocation of resources based on fault severity and urgency, ensuring that high-severity faults are addressed promptly and improving the efficiency and quality of fault recovery.

[0053] Adapts to the needs of multiple types of spare parts: It can handle the scheduling problems of various types of spare parts and is suitable for complex and diverse communication network maintenance scenarios, improving the versatility and practicality of the algorithm.

[0054] Providing optimized supply solutions: Through multi-objective optimization functions and continuous optimization of scheduling strategies through reinforcement learning, it is possible to find a balance between multiple objectives such as cost, time, and risk, providing a systematic and optimized supply solution for communication network spare parts scheduling, and improving the overall operation and maintenance management level.

[0055] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 A schematic diagram of the overall process of network communication spare parts scheduling provided in accordance with an embodiment of the present disclosure is shown;

[0058] Figure 2 The diagram shows a schematic diagram of a classic BP neural network structure according to an embodiment of the present disclosure;

[0059] Figure 3 shows a schematic diagram of perceptrons at each layer according to an embodiment of the present disclosure;

[0060] Figure 4 A schematic diagram of the overall process of network communication spare parts scheduling according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0062] The embodiment of the present disclosure provides a spare parts scheduling model training method. The network structure of the spare parts scheduling model is based on BP neural network and reinforcement learning network structure. Figure 1 As shown, the training method includes:

[0063] Construct a multi-objective optimization function based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment;

[0064] Scheduling environment modeling and data preparation: Define the state space, quantify, integrate, and transform inventory information, demand forecast information, equipment failure information, and transportation resource information to obtain state information; determine the action space, number the actions in spare parts supply management to obtain an action space set, where an action includes dispatching a specific spare part from a specific warehouse to a specified failure location via a specific transportation method; construct a reward function: construct a reward function based on the constructed multi-objective optimization function; data collection and preprocessing: collect logistics time, spare parts risk, and scheduling cost data in spare parts supply management, and preprocess the data as input to the multi-objective optimization function;

[0065] Construct an experience replay memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience;

[0066] Randomly extract multiple experiences from the experience replay memory pool as training samples;

[0067] Entering the training phase, the main BP neural network and the target BP neural network are trained using training samples. The main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data.

[0068] Entering the learning phase, the loss function is calculated using the current value calculated by the main BP neural network and the target value calculated by the target BP neural network, and the parameters of the main BP neural network are updated using the gradient of the loss function;

[0069] Repeat the above training and learning phases;

[0070] Whenever the above training and learning phases complete a certain number of iterations, the parameters of the main BP neural network are synchronized to the target BP neural network, and the next training and learning phases are continued until the loss function converges below the preset threshold, or the change in the loss function is less than the set minimum value;

[0071] The trained target BP neural network is used as the trained spare parts scheduling model.

[0072] The following is a detailed description of the construction of multi-objective optimization functions, environment modeling and data preparation, and networks based on BP neural networks and reinforcement learning involved in the above training method:

[0073] 1. Construct a multi-objective optimization function

[0074] The definition of spare parts supply time is that the spare parts supply time is the total time from the issuance of spare parts scheduling instructions to the arrival of spare parts at the designated location and their availability for use. This includes the time to extract spare parts from the warehouse, the time in transit, and the possible time for warehousing inspection and preparation for use. By collecting historical spare parts scheduling data, the average time spent on different types of spare parts in each link is calculated, and a time estimation model is constructed. For example, for transportation time, it can be estimated based on the average speed of different modes of transportation (such as road, rail, and air), transportation distance, and historical transportation delay data. Let T1 be the time to extract from the warehouse, T2 be the transportation time, and T3 be the time for warehousing inspection and preparation, then the spare parts supply time is T=T1+T2+T3.

[0075] Spare parts dispatch costs are defined as all expenses directly related to spare parts dispatch. These include labor costs, inventory costs, and potential stockout costs. Labor costs include wages and benefits for warehouse and transportation personnel, and transportation costs. Transportation costs are calculated based on the mode of transport and distance. For example, road transport is calculated by multiplying the freight rate per kilometer by the distance traveled, while air transport is calculated by flight fares and cargo weight. Inventory costs refer to the cost of storing spare parts in the warehouse, taking into account warehouse rent, inventory management system maintenance costs, and interest on spare parts capital. They are typically calculated as the inventory cost per spare part per unit time multiplied by the number of spare parts and the storage time. Stockout costs are the estimated economic losses incurred when a spare part shortage causes a communication network failure. They can be estimated based on the duration of the network failure, the volume of affected business, and the value of each unit of business. Let labor costs be C1, transportation costs be C2, inventory costs be C3, and stockout costs be C4. Then, the spare parts dispatch cost C = C1 + C2 + C3 + C4.

[0076] Spare parts risk is a risk indicator based on the characteristics of spare parts themselves and the degree of impact on the operation of communication networks. It is comprehensively considered from three dimensions: spare parts failure rate, importance, and scarcity. Among them, the spare parts failure rate is obtained based on historical failure data statistics. For example, the failure rate is obtained by dividing the number of failures of a spare part in the past year by the total operating time; the importance is determined by scoring factors such as the criticality of the communication network node where the spare part is located and the importance of related businesses. For example, the importance score of spare parts for core network equipment is higher than that of spare parts for edge equipment; the scarcity is based on the supply of the spare parts in the market. For example, if there are few suppliers and the production cycle is long, the scarcity is high. Set weights ω for failure rate, importance, and scarcity respectively. f 、ω i 、ω s The equipment failure rate is F, the importance is I, and the scarcity is S, then the spare parts risk R = ω f ×F+ω i ×I+ω s ×S, where ω f 、ω i 、ωs The sum of the three weights is 1.

[0077] Finally, construct the multi-objective optimization function V = ω t ×T+ω c ×C+ω r ×R, where ω t 、ω c 、ω r are the weights of spare parts supply time, cost and risk respectively. The constructed multi-objective optimization function can be used to evaluate the quality of a spare parts scheduling plan.

[0078] 2. Environment modeling and data preparation

[0079] Defining the state space: Integrating multiple pieces of information to construct state information. Inventory information details the inventory quantity of each spare part in each warehouse, along with the warehouse identification information, and is stored in a matrix format. For N types of spare parts and M warehouses, the inventory matrix I has dimensions N × M. Demand forecasting information, based on in-depth analysis of historical data, equipment health monitoring, and network traffic trend research, predicts the demand for each type of spare part in different regions during a specific future period, forming a demand matrix. Assuming P regions and N types of spare parts are involved, the demand matrix D has dimensions P × N. Equipment fault information is collected immediately upon failure, including the type of faulty equipment (according to a pre-defined classification code), fault location (accurate to the base station number or equipment room location), and fault severity (quantified numerically, such as 1-5 for mild to severe). This information is combined into the fault matrix F. Transportation resource information monitors the available number of transportation vehicles (such as vehicles, drones, etc.) and their warehouse locations in real time, constructing the transportation resource matrix T. Finally, the inventory matrix, demand vector, fault vector, and transportation resource vector are integrated and transformed to produce a unified state vector S, which serves as the input to the BP neural network.

[0080] (a) Inventory matrix:

[0081] Suppose there are N types of spare parts distributed in M ​​warehouses. Define the inventory matrix I, whose element I ij Represents the inventory quantity of the i-th spare part in the j-th warehouse. That is:

[0082]

[0083] The inventory matrix has dimensions N×M.

[0084] (b) Demand Matrix:

[0085] Assume that there are P regions involved and the demand for N types of spare parts needs to be considered. Define the demand matrix D, whose elements D ijrepresents the demand quantity of the jth spare part in the i-th region. The dimension of the demand vector is P×N, which can be expressed as:

[0086]

[0087] (c) Fault matrix:

[0088] Assuming that there are P regions involved, the demand matrix represents the number of spare parts required, and the failure matrix represents the severity of the spare parts. Define the failure matrix F, whose elements F ij represents the failure level of the jth type of spare part in the i-th region (the most serious failure level of this type of spare part is taken). The dimension of the fault vector is P×N and can be expressed as:

[0089]

[0090] (d) Transportation resource matrix:

[0091] Suppose there are K types of transportation tools, distributed in M ​​warehouses. Define the inventory matrix T, whose elements T ij represents the available quantity of the i-th means of transport in the j-th warehouse. That is:

[0092]

[0093] Finally, the inventory matrix, demand vector, fault vector, and transportation resource vector are integrated and transformed to obtain a unified state vector S, namely:

[0094] S=(IDFT)

[0095] Determine the action space: Identify all possible actions in spare parts supply management and number them. Actions include dispatching a specific spare part from a specific warehouse to a specific fault location via a specific transportation method, such as action number A1-A W Corresponding to different warehouse-transportation mode-spare parts-fault location combinations, the action space set is recorded as A and can be expressed as:

[0096] A=(A1A2....A W )

[0097] Suppose there are m warehouses, q transportation modes, n types of spare parts, and p possible locations of failure in the system. Then the number of possible combinations of deploying a specific spare part from a specific warehouse to a specific failure location is m×q×n×p. For these combinations, use actions numbered A1-A WW = m × q × n × p. For example, A1 represents the dispatch of the first type of spare part from the first warehouse via the first mode of transportation to the first fault location; A2 represents the dispatch of the first type of spare part from the first warehouse via the second mode of transportation to the second fault location; and so on, until all possible warehouse-transportation-spare part-fault location dispatch combinations are covered.

[0098] Constructing a reward function: Reward is the feedback signal from the environment to the agent after it performs an action, which is used to test the effectiveness of the action. Previously, a multi-objective optimization function V = ω has been constructed to evaluate a spare parts scheduling action. t ×T+ω c ×C+ω r ×R, where T is the spare parts supply time, C is the spare parts scheduling cost, R is the spare parts risk assessment, ω t 、ω c 、ω r are the weights of spare parts supply time, cost, and risk respectively. The smaller the V value, the better the corresponding scheduling action. When an incident occurs suddenly and emergency scheduling is required, the time weight ω should be increased. t Similarly, when cost control is needed, the cost weight ω needs to be increased. c When risk needs to be controlled, it is necessary to improve r In order to judge the quality of an action, a benchmark optimization value V needs to be set based on historical experience and expert evaluation. base , so the reward function is constructed based on this r is the reward value, when the cost of this action is V t When the cost is lower than the baseline optimization value, the corresponding reward is positive, and the lower the cost, the higher the reward. When the cost is higher than the baseline optimization value, the corresponding reward is negative, and the higher the cost, the lower the reward. Therefore, the constructed scheduling model can continuously adjust its own parameters through feedback until a satisfactory result is achieved.

[0099] Data collection and preprocessing: Data related to logistics time, spare parts risk, and scheduling costs is collected from the communication network spare parts management system. This data is cleaned and organized to remove outliers and erroneous data. The collected data is then normalized and standardized to meet the input requirements of the BP neural network. For example, logistics time is converted to relative time ratios, and spare parts risk and scheduling costs are mapped to specific numerical ranges.

[0100] 3. Network based on BP neural network and reinforcement learning

[0101] like Figure 2 The figure shows the structure of a classic BP neural network. The number of neurons in the input layer n is determined by the dimension of the state vector S.input , if the state vector length is L, then the input layer sets L neurons to receive the complete system state information. The perceptron framework is as follows Figure 3 shown.

[0102] The hidden layer adopts a multi-layer structure to enhance the network's expressive power. The number of neurons in the first hidden layer is based on the empirical formula (where n eff The estimated effective information dimension of the output layer is preliminarily determined, and the activation function is the ReLU function, that is, f ReLU (x) = max(0, x) to accelerate the convergence of network training and enhance the nonlinear mapping ability. The number of neurons in the second hidden layer is appropriately reduced based on the first layer, such as The activation function uses the Sigmoid function Used to further transform and map the data. More hidden layers can be added according to actual needs.

[0103] The number of neurons in the output layer is equal to the size of the action space A, that is, n output = W. The output of each neuron represents the estimated Q-value of the corresponding action in the current state, that is, the expected long-term cumulative reward from executing that action. The output layer does not use an additional activation function to maintain the continuity and interpretability of the Q-values, facilitating the application of subsequent reinforcement learning algorithms.

[0104] (a) Experience replay mechanism:

[0105] Construct an experience replay memory pool D to store the experience data during the interaction between the agent (decision-making module based on BP neural network in this system) and the environment. The experience data is in the form of a four-tuple (s t , a t , r t , s t+1 ) format, where s t represents the system state at time t, a t is the action taken in this state, r t is the immediate reward after executing the action, s t+1 It is the next system state after the action is executed.

[0106] During system operation, each time the agent interacts with the environment and generates new experience, it is stored in the experience replay memory pool D. When the memory pool reaches a certain capacity C, during the training phase, a batch of experience data is randomly sampled from the memory pool, such as the number of samples B. This random sampling method breaks the temporal correlation between the experience data, improves the diversity of training data, and enhances the network's generalization ability.

[0107] (b) Target network and main network architecture:

[0108] In addition to the BP neural network for real-time Q-value estimation constructed above (called the main network, with parameters θ), a target network with the same structure is created (with parameters θ - The role of the target network is to provide a relatively stable target Q-value calculation benchmark to avoid instability in the main network training due to the continuous update of the Q-value estimation during the training process.

[0109] The parameters θ of the main network are copied to the target network periodically (e.g., every T training steps) so that the target network can track the learning progress of the main network while maintaining a certain lag, thereby ensuring the relative stability of the target Q value.

[0110] The following is a description of the specific details of the training process:

[0111] 1. Initialization

[0112] Initialize the parameters of the main network and the target network by randomly initializing the weights;

[0113] Clears the experience replay memory pool.

[0114] 2. Filling the Experience Replay Memory Pool

[0115] State initialization: given an initial state s in the state space t ;

[0116] Time step loop: The time step is executed continuously in each round until the task is completed or the maximum time step is reached (for example, set to 100 steps);

[0117] The sign of task completion: the elements in the demand matrix and the elements in the fault matrix in the status matrix are all 0.

[0118] Action selection: This invention adopts the exploration-balance (ε-greedy) method to select action a t , set ε, that is, there is a 1-ε probability of selecting the action with the largest Q value, and the probability of ε randomly selects other actions.

[0119] Environmental interaction: After executing action a, the environment feedbacks the new state s t+1 and reward r t ;

[0120] Storage experience: Experience tuple (s t , a t , r t , s t+1 ) is stored in the experience replay memory pool D.

[0121] 3. Sampling and training (after certain conditions are met): When the number of experiences stored in the experience replay memory pool exceeds the number of mini-batch samples, sampling training begins. Several samples are randomly sampled from the experience replay memory pool.

[0122] (a) Using the main network to calculate the sample (s t , a t , r t , s t+1 ) in state s t Next take action a t The obtained Q value estimate Q(s t , a t ,θ), and use the target network to calculate the new state s t+1 The maximum Q value estimation under

[0123] (b) Update the formula based on the Q-learning algorithm of reinforcement learning Where γ is the discount factor, ranging from 0 to 1, which is used to measure the impact of future rewards on current decisions, and θ - are the parameters of the target network;

[0124] (c) Using the mean square error (MSE) loss function: (Here B is the number of batch samples), calculate the loss, and then use the gradient descent algorithm with a learning rate of α to update the parameters θ of the main network;

[0125] (d) Based on the calculated loss function, the gradient of the loss function with respect to the main network parameter θ is calculated through the back propagation algorithm.

[0126] In the back propagation process, according to the chain rule of BP neural network, the error gradient of each layer of neurons is calculated in turn. For the jth neuron in the lth layer, its error gradient is The calculation is as follows:

[0127] Output layer (L = L output ): in is the derivative of the output layer activation function (since there is no additional activation function in the output layer, the derivative here is 1), is the input of the jth neuron in the output layer.

[0128] Hidden layer (L <L output ): Where k is the neuron index of the l+1th layer, is the weight from the jth neuron in the lth layer to the kth neuron in the l+1th layer, f l′ is the derivative of the activation function of the lth layer.

[0129] (e) Based on the calculated error gradient, the gradient descent algorithm is used to update the weight ω and bias b parameters of the main network: Where α is the learning rate, is the output of the i-th neuron in the l-1th layer.

[0130] (f) Update the target network: After each training cycle, the parameters of the current main network are copied to the target network.

[0131] After the above training, the trained main network model is obtained. The performance of the main network model needs to be evaluated, which specifically includes the following steps:

[0132] (a) Evaluate the performance of the trained BP neural network model (target network) using a test dataset of excellent historical manual scheduling solutions evaluated by experts. The test dataset should cover a variety of spare parts supply scenarios, including different inventory status, demand patterns, failure conditions, and transportation resource allocations.

[0133] (b) Input the system states in the test dataset into the trained network in sequence to obtain the spare parts supply action recommended by the network and its corresponding Q value estimate. Compare it with the optimal action pre-labeled in the test dataset (which can be determined through historical experience or expert knowledge) and calculate the action matching accuracy index Where T test is the number of test dataset samples, is an indicator function that is 1 if the predicted action is the same as the optimal action and 0 otherwise.

[0134] Once the network model passes performance evaluation and meets actual application requirements, it can be deployed to a real communication network spare parts supply management system. Figure 4 The overall process of scheduling training and scheduling method is shown in FIG. The spare parts scheduling method at this time includes the following steps:

[0135] Collect current spare parts scheduling information, including inventory information, demand forecast information, equipment failure information, and transportation resource information;

[0136] Build status information based on current spare parts scheduling information;

[0137] The state information constructed here is Figure 4 The new data in the input.

[0138] The constructed state information is input into the trained spare parts scheduling model based on BP neural network and reinforcement learning. The spare parts scheduling model based on BP neural network and reinforcement learning outputs the Q value of each action according to the input state information, and selects the action with the largest Q value as the spare parts supply action. The selected action is the communication spare parts scheduling strategy, that is, Figure 4 Spare parts scheduling scheme in.

[0139] Communication spare parts scheduling strategies include determining which spare parts to allocate from which warehouse, which transportation method to choose, and whether to initiate lateral transfer. After the action is completed, the new state is fed back to the model, and the model determines the next action, and so on, until the task is completed.

[0140] (c) All experiences generated by this interaction (s t , a t , r t , s t+1 ) is stored in the experience replay memory pool for subsequent continuous network training and optimization. Data is periodically sampled from the experience replay memory pool (e.g., every G interval) to fine-tune the BP neural network. This ensures that the BP neural network can always provide efficient and intelligent support for spare parts supply decisions, adapting to dynamic changes in the communication network environment, fluctuations in spare parts demand, and changes in transportation resources.

[0141] The following is an example to illustrate the above scheme:

[0142] Assume: In a scenario, there are two warehouses (Warehouse A, Warehouse B), two types of spare parts (Spare Part A, Spare Part B), two maintenance areas (Area A, Area B), and two modes of transportation (Mode A, Mode B).

[0143] Warehouse A has one spare part A and two spare parts B. Warehouse B has two spare parts A and one spare part B, so the inventory matrix is

[0144] Maintenance area A detects a fault and requires one spare part A with a fault level of 2 and one spare part B with a fault level of 1. Maintenance area B has no fault, so the demand matrix is: The fault matrix is

[0145] Warehouse A has only mode A for transportation and no mode B, while warehouse B has only mode B for transportation and no mode A. Then the transportation matrix is:

[0146] This constitutes the overall state matrix

[0147] Correspondingly, the action space is:

[0148] Action 1: Warehouse A delivers spare part A to region A via transport method A;

[0149] Action 2: Warehouse A delivers spare part A to region B via transport method A;

[0150] Action 3: Warehouse A delivers spare part B to region A via transport method A: ......

[0152] Action 16 (2*2*2*2): Warehouse B delivers spare part B to region B using transport method B.

[0153] According to the state space, the action matrix is ​​(1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1), where 1 indicates that the action is executed conditionally, and 0 indicates that the action exists in theory but cannot be performed. For example, some actions cannot be performed because warehouse A does not have transportation method B.

[0154] Since the main network has been initialized and the input and output formats have been set, we will now fill the experience replay memory pool and flatten the overall state matrix to (1, 2, 2, 1, 1, 1, 0, 0, 2, 1, 0, 0, 1, 0, 0, 1) as the input of the main network. The Q value corresponding to the main network output action matrix is ​​assumed to be (0.9, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.3, 0.5, 0.4, 0.5, 0.2). Then, action a is selected according to the exploration-balance (ε-greedy) method. t , setting ε to 0.1 means that there is a 90% probability of selecting the action with the largest Q value, that is, action 1 with a Q value of 0.9, and a 10% probability of randomly selecting other actions. This gives the current state (s t , a t ), then define the reward function based on the multi-objective optimization function, for action a t , get V as 1000, set the benchmark optimization value V base If the reward is 2000, the reward r is 0.5, and after executing the action, the state matrix becomes (0, 2, 2, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1), which is s t+1 , thus forming a sample (s t , a t , r t , s t+1 ), can be stored in the experience replay memory pool.

[0155] This new state is then passed to the main network, which continues to obtain new actions, rewards, and the new state after executing this action. When a certain number of samples are accumulated, a certain number of samples can be randomly selected from the experience replay memory pool as training samples for training. After a period of training, the main network parameters are passed to the target network.

[0156] In this way, the target network can be obtained, which can theoretically provide the optimal action for each step given the initial state until the task is completed.

[0157] The previous historical manual scheduling scheme can be compared with the scheme provided by the target network under the same initial state. If it is not as good as the historical manual scheduling scheme, continue training.

[0158] The spare parts scheduling method of the disclosed embodiment can handle daily scheduling and sudden failures. For daily scheduling needs, since the scheduling model is trained based on a multi-objective optimization function, the spare parts supply time and cost can be optimized. Under the premise of meeting the normal maintenance needs of the communication network, the spare parts inventory and scheduling plan are reasonably arranged to reduce inventory costs and transportation costs, and improve the turnover rate and utilization rate of spare parts. For sudden failures, the time weight can be increased so that the scheduling model can adopt lateral transfer and emergency distribution strategies to ensure the rapid scheduling of multiple types of spare parts. By establishing a lateral transfer mechanism with surrounding sites or warehouses, the required spare parts can be quickly allocated and the spare parts supply time can be shortened.

[0159] Based on the above training method, the embodiment of the present disclosure further provides a device corresponding to the above training method. The training device is based on the BP neural network and reinforcement learning network structure, including: a cost function construction module, a modeling and data preparation module, a memory pool construction module, a training module and a model establishment module; wherein:

[0160] Cost function building module, used to build multi-objective optimization functions based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment;

[0161] The modeling and data preparation module is used to model and prepare the scheduling environment: define the state space, quantify, integrate, and transform inventory information, demand forecast information, equipment failure information, and transportation resource information to obtain state information; determine the action space, number the actions in spare parts supply management to obtain an action space set, where an action includes dispatching a specific spare part from a specific warehouse to a specified failure location via a specific transportation method; construct a reward function based on the constructed multi-objective optimization function; and collect and preprocess data on logistics time, spare parts risk, and scheduling costs in spare parts supply management, and preprocess the data as input to the multi-objective optimization function.

[0162] The memory pool construction module is used to build an experience replay memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience;

[0163] The training module is used to randomly extract multiple experiences from the experience replay memory pool as training samples; enter the training phase, use the training samples to train the main BP neural network and the target BP neural network, and the main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data; enter the learning phase, use the current value calculated by the main BP neural network and the target value calculated by the target BP neural network to calculate the loss function, and use the gradient of the loss function to update the parameters of the main BP neural network; repeat the above training phase and learning phase; whenever the above training phase and learning phase complete a certain number of iterations, synchronize the parameters of the main BP neural network to the target BP neural network, and continue to the next training phase and learning phase until the loss function converges below the preset threshold, or the change amplitude of the loss function is less than the set small value;

[0164] The model building module is used to use the trained target BP neural network as a trained spare parts scheduling model.

[0165] Based on the above scheduling method, an embodiment of the present disclosure further provides a device corresponding to the above scheduling method, wherein the scheduling device includes: a collection module and a scheduling strategy determination module;

[0166] A collection module is used to collect current spare parts scheduling information and build status information based on the current spare parts scheduling information, wherein the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information;

[0167] The scheduling strategy determination module is used to input the constructed state information into the trained spare parts scheduling model, obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and use the action corresponding to the maximum expected value as the spare parts scheduling strategy, wherein the trained spare parts scheduling model is obtained using the above-mentioned training device.

[0168] Based on the same inventive concept as the above disclosure, the present disclosure also provides an electronic device. The electronic device of the present disclosure embodiment includes at least one electrically connected processor and at least one memory, the memory being electrically connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0169] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. An indirect connection method can be applied to the embodiments of the present disclosure as long as the purpose of the present disclosure is achieved.

[0170] Based on the same inventive concept, the present disclosure further provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0171] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A spare parts scheduling model training method, characterized in that: The training method is based on BP neural network and reinforcement learning network structure, including: Construct a multi-objective optimization function based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment; Scheduling environment modeling and data preparation: Define the state space, quantify, integrate, and transform inventory information, demand forecast information, equipment failure information, and transportation resource information to obtain state information; determine the action space, number the actions in spare parts supply management to obtain an action space set, where an action includes dispatching a specific spare part from a specific warehouse to a specified failure location via a specific transportation method; construct a reward function: construct a reward function based on the constructed multi-objective optimization function; data collection and preprocessing: collect logistics time, spare parts risk, and scheduling cost data in spare parts supply management, and preprocess the data as input to the multi-objective optimization function; Construct an experience replay memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience; Randomly extract multiple experiences from the experience replay memory pool as training samples; Entering the training phase, the main BP neural network and the target BP neural network are trained using training samples. The main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data. Entering the learning phase, the loss function is calculated using the current value calculated by the main BP neural network and the target value calculated by the target BP neural network, and the parameters of the main BP neural network are updated using the gradient of the loss function; Repeat the above training and learning phases; Whenever the above training and learning phases complete a certain number of iterations, the parameters of the main BP neural network are synchronized to the target BP neural network, and the next training and learning phases are continued until the loss function converges below the preset threshold, or the change in the loss function is less than the set minimum value; The trained target BP neural network is used as the trained spare parts scheduling model.

2. The training method according to claim 1, characterized in that The constructed multi-objective optimization function is V = ω t ×T+ω c ×C+ω r ×R; where V is the multi-objective optimization value, T is the spare parts supply time, C is the spare parts scheduling cost, R is the spare parts risk assessment, ω t 、ω c 、ω r They are the weight of spare parts supply time, the weight of spare parts scheduling cost, and the weight of spare parts risk assessment.

3. The training method according to claim 1 or 2, characterized in that: The reward function constructed based on the constructed multi-objective optimization function is: Among them, r is the reward value, Y base is the benchmark optimization value set based on historical experience and expert evaluation, V t is the multi-objective optimization value corresponding to this action.

4. The training method according to claim 1 or 2, characterized in that: Inventory information includes: spare parts types, warehouse quantity and spare parts quantity; demand forecast information includes: number of regions and spare parts demand quantity; fault information includes: fault equipment type, fault location and fault severity; transportation resource information includes: available number of transportation tools, current location and transportation capacity parameters.

5. The training method according to claim 4, characterized in that It is characterized by: Quantify inventory information, demand forecast information, equipment failure information, and transportation resource information, including: Assume that there are N types of spare parts distributed in M ​​warehouses, and define the inventory matrix I, whose element I ij represents the inventory quantity of the i-th spare part in the j-th warehouse; Assume that there are P regions involved and the demand for N types of spare parts needs to be considered. Define the demand matrix D, whose elements D ij represents the demand quantity of the jth type of spare parts in the i-th region; Assuming that there are P regions involved, the demand matrix represents the number of spare parts required, and the failure matrix represents the severity of the spare parts. Define the failure matrix F, whose elements F ij represents the failure degree of the jth spare part in the i-th area; Assume that there are K types of transportation tools, distributed in M ​​warehouses, and define the inventory matrix T, whose elements T ij represents the available quantity of the i-th means of transport in the j-th warehouse.

6. The training method according to claim 1 or 2, characterized in that: Number the actions in spare parts supply management, including: Number each action resulting from the deployment of different warehouse-spare parts-fault location combinations.

7. A spare parts scheduling method, characterized in that: The scheduling method includes: Collect current spare parts scheduling information and build status information based on the current spare parts scheduling information; the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information; The constructed state information is input into the trained spare parts scheduling model to obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and the action corresponding to the maximum expected value is used as the spare parts scheduling strategy, wherein the trained spare parts scheduling model is obtained using the training method described in any one of claims 1-6.

8. The scheduling method according to claim 7, characterized in that: The scheduling method further includes: The status information of the build and spare parts scheduling strategy are saved as experience.

9. A spare parts scheduling model training device, characterized in that: The training device is based on the BP neural network and reinforcement learning network structure, and includes: a cost function construction module, a modeling and data preparation module, a memory pool construction module, a training module and a model establishment module; wherein: Cost function building module, used to build multi-objective optimization functions based on spare parts supply time, spare parts scheduling cost and spare parts risk assessment; The modeling and data preparation module is used to model and prepare the scheduling environment: define the state space, quantify, integrate, and transform inventory information, demand forecast information, equipment failure information, and transportation resource information to obtain state information; determine the action space, number the actions in spare parts supply management to obtain an action space set, where an action includes dispatching a specific spare part from a specific warehouse to a specified failure location via a specific transportation method; construct a reward function based on the constructed multi-objective optimization function; and collect and preprocess data on logistics time, spare parts risk, and scheduling costs in spare parts supply management, and preprocess the data as input to the multi-objective optimization function. The memory pool construction module is used to build an experience replay memory pool to store the interaction data between the main BP neural network and the scheduling environment. When the main BP neural network interacts with the scheduling environment based on the input state information, the state, action, reward of each time step and the state of the next time step are saved as an experience; The training module is used to randomly extract multiple experiences from the experience replay memory pool as training samples; enter the training phase, use the training samples to train the main BP neural network and the target BP neural network, and the main BP neural network is responsible for interacting with the scheduling environment to obtain interaction data; enter the learning phase, use the current value calculated by the main BP neural network and the target value calculated by the target BP neural network to calculate the loss function, and use the gradient of the loss function to update the parameters of the main BP neural network; repeat the above training phase and learning phase; whenever the above training phase and learning phase complete a certain number of iterations, synchronize the parameters of the main BP neural network to the target BP neural network, and continue to the next training phase and learning phase until the loss function converges below the preset threshold, or the change amplitude of the loss function is less than the set small value; The model building module is used to use the trained target BP neural network as a trained spare parts scheduling model.

10. A spare parts scheduling device, characterized in that: The scheduling device includes: a collection module and a scheduling strategy determination module; wherein: A collection module is used to collect current spare parts scheduling information and build status information based on the current spare parts scheduling information; wherein the spare parts scheduling information includes: inventory information, demand forecast information, equipment failure information and transportation resource information; A scheduling strategy determination module is used to input the constructed state information into a trained spare parts scheduling model, obtain the expected value of the long-term cumulative reward that can be obtained by executing each action, and use the action corresponding to the maximum expected value as the spare parts scheduling strategy, wherein the trained spare parts scheduling model is obtained using the training device described in claim 9.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 7 when executing a program stored in a memory.

12. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images