A medicine supply chain scheduling method and system based on reinforcement learning

Abstract

The present application relates to the technical field of supply chain management, in particular to a medicine supply chain scheduling method and system based on reinforcement learning, comprising the following steps: obtaining the temperature difference inside and outside the carriage and the heat conduction coefficient to calculate the heat penetration rate and the thermal safety time limit, planning the path and correcting the speed based on the congestion index to obtain the predicted passing time, comparing the time limit with the deadline to generate the action mask vector, inputting the environmental parameters into the neural network to output the score, and using the action mask to correct and generating the scheduling instruction through the Softmax function. In the present application, the thermal safety time limit is accurately derived, the thermal safety time limit and the delivery deadline are used as double constraints to construct the action mask to eliminate invalid nodes, multi-dimensional parameters are input into the neural network and the probability distribution is generated under the mask constraint, and it is ensured that the vehicle scheduling instruction follows the temperature control safety threshold, effectively avoiding the risk of drug heat failure and improving the accuracy of cold chain logistics distribution.

Description

Technical Field

[0001] This invention relates to the field of supply chain management technology, and in particular to a pharmaceutical supply chain scheduling method and system based on reinforcement learning. Background Technology

[0002] Supply chain management technology refers to the technical field of optimizing the entire product lifecycle from design and production to delivery by coordinating and controlling various links. It encompasses the systematic integration and planning of information flow, logistics, and capital flow. Traditional pharmaceutical supply chain scheduling methods rely on manual experience combined with basic information management software. Operators use enterprise resource planning (ERP) systems to query inventory reports and historical order data for each node, manually generate replenishment lists based on pre-set fixed inventory thresholds, use spreadsheet tools to calculate the total drug demand of various medical institutions or pharmacies, and arrange logistics transportation tasks according to fixed delivery route maps and vehicle schedules. Finally, they issue inbound / outbound and vehicle departure instructions via telephone or email.

[0003] Traditional pharmaceutical supply chain scheduling mainly relies on manual review of reports and fixed route planning. This static data-based operation mode cannot respond in real time to dynamic changes in external environmental temperature and road congestion. Relying solely on experience and pre-set schedules for transportation can easily lead to drug deterioration due to prolonged exposure to heat during transit. Furthermore, fixed threshold statistical methods are insufficient to accurately assess the timeliness requirements and cold chain risks at different delivery nodes. Manual scheduling instructions are delayed and lack comprehensive consideration of multi-dimensional constraints, resulting in low delivery efficiency and causing drug safety hazards and delivery delays. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a pharmaceutical supply chain scheduling method and system based on reinforcement learning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a pharmaceutical supply chain scheduling method based on reinforcement learning, comprising the following steps:

[0006] S1: Obtain the temperature value inside the carriage, the temperature value outside the carriage, and the thermal conductivity value. Subtract the temperature value inside the carriage from the temperature value outside the carriage and multiply by the thermal conductivity value to obtain the heat penetration rate value. Subtract the temperature value inside the carriage from the highest critical temperature value and divide by the heat penetration rate value to obtain the thermal safety time limit value.

[0007] S2: Identify candidate delivery nodes based on location coordinates and plan the shortest driving route. Obtain the mileage value, congestion index value and driving speed value of the shortest driving route. Divide the mileage value by the driving speed value after correction by the congestion index value to obtain the estimated travel time value.

[0008] S3: Obtain the delivery deadline value of the candidate delivery node, add the system time value to the estimated travel time value to obtain the estimated arrival time value, compare the estimated travel time value with the thermal safety time limit value, and compare the estimated arrival time value with the delivery deadline value to mark the node reachability status, and generate an action mask vector.

[0009] S4: Input the temperature value inside the carriage, the temperature value of the external environment, and the position coordinates into the deep neural network to output the original score value. Replace the original score value corresponding to the zero value in the action mask vector with a negative infinity value. Obtain the action selection probability distribution through the Softmax function. Generate vehicle dispatch instructions based on the action selection probability distribution.

[0010] As a further aspect of the present invention, the calculation process for the heat penetration rate value and the thermal safety time limit value includes:

[0011] S11: Real-time monitoring and collection of the internal temperature of the medical refrigerated vehicle, the external ambient temperature of the vehicle's environment, and the thermal conductivity of the refrigerated container material.

[0012] S12: Calculate the temperature difference between the external ambient temperature and the interior temperature of the carriage, and multiply the temperature difference by the thermal conductivity value to generate the heat penetration rate value;

[0013] S13: Obtain the preset maximum critical temperature value of the drug, calculate the difference between the maximum critical temperature value and the temperature value inside the carriage, and divide the difference by the heat penetration rate value to generate the thermal safety time limit value.

[0014] As a further aspect of the present invention, the process of identifying candidate delivery nodes, planning routes, and calculating travel time includes:

[0015] S21: Using the vehicle's current location coordinates as the center, retrieve the locations of orders with incomplete delivery tasks within a preset service radius, and generate the candidate delivery nodes;

[0016] S22: Construct a topology network from the current location coordinates to multiple candidate delivery nodes, plan the shortest driving path using a path search algorithm, and extract the mileage value of the path;

[0017] S23: Obtain the real-time congestion index value and the designed driving speed value of the route segment, use the congestion index value to attenuate and correct the driving speed value to obtain the actual speed, and divide the mileage value by the actual speed to generate the estimated travel time value.

[0018] As a further aspect of the present invention, the process of generating the action mask vector includes:

[0019] S31: Extract the delivery deadline values ​​of multiple candidate delivery nodes from the order information, and obtain the current system time value;

[0020] S32: Add the system time value to the calculated estimated travel time value to generate the estimated arrival time value;

[0021] S33: Perform a dual constraint check to determine whether the estimated travel time is less than the thermal safety time limit and whether the estimated arrival time is less than the delivery deadline. If both conditions are met, mark the node reachability status as valid; otherwise, mark it as invalid.

[0022] S34: Construct the action mask vector based on the reachable states of multiple nodes, where valid states correspond to the value 1 and invalid states correspond to the value 0.

[0023] As a further aspect of the present invention, the process of generating the original score value and outputting the vehicle dispatch instruction includes:

[0024] S41: Normalize the temperature value inside the carriage, the temperature value of the external environment, and the position coordinates, construct a multi-dimensional state feature vector, input it into the deep neural network, and output the original score value after processing by the network layer.

[0025] S42: Use the action mask vector to perform a masking operation on the original score value, and forcibly replace the original score value corresponding to the zero value position in the action mask vector with the negative infinity value;

[0026] S43: Perform Softmax normalization on the replaced score value sequence to generate the action selection probability distribution, and select the target node according to the distribution to generate the vehicle dispatch instruction.

[0027] As a further aspect of the present invention, the process of obtaining the thermal conductivity value includes:

[0028] The basic thermal conductivity and material aging factor of the refrigerated box are obtained, and the absolute temperature difference between the external ambient temperature and the internal temperature of the compartment is calculated in real time.

[0029] Based on the influence of temperature difference on the thermal conductivity of materials, the thermal conductivity value is calculated using a linear correction model, i.e.:

[0030] ;

[0031] in, This represents the stated thermal conductivity value. Represents the basic thermal conductivity. Represents the temperature difference correction factor. This represents the absolute temperature difference between the external ambient temperature and the interior temperature of the vehicle compartment.

[0032] As a further aspect of the present invention, the calculation process of the corrected travel speed value in the calculation of the estimated travel time value includes:

[0033] By introducing a traffic impedance function and using the congestion index value to perform a nonlinear attenuation calculation on the designed driving speed value, a corrected actual driving speed is generated, i.e.:

[0034] ;

[0035] in, This represents the corrected actual driving speed. The stated driving speed value represents the road design. This represents the congestion index value. Represents the road condition sensitivity coefficient. This represents a non-linear exponential factor.

[0036] As a further aspect of the present invention, the marking logic for the reachable state of a node includes:

[0037] Establish logical AND operation rules to determine a node as reachable only when both the thermal safety constraint and the time window delivery constraint are true.

[0038] If the estimated travel time is greater than or equal to the thermal safety time limit, it indicates that the drug will fail thermally before arrival, and the node reachability status will be set to invalid.

[0039] If the estimated arrival time is greater than the delivery deadline, it indicates that even if the drug is safe, the task is overdue, and the node reachability status is considered invalid.

[0040] As a further aspect of the present invention, the construction process of the deep neural network includes:

[0041] Construct a network architecture that includes an input layer, a feature extraction layer, and a decision output layer;

[0042] A multi-head attention mechanism sublayer is set in the feature extraction layer to capture the spatial dependencies in the position coordinate sequence, and a fully connected layer is set to fuse temperature features and spatial features.

[0043] In the decision output layer, a number of neurons are set up that are the same as the number of candidate delivery nodes, and the original score value is output through a linear activation function.

[0044] A reinforcement learning-based pharmaceutical supply chain scheduling system, wherein the reinforcement learning-based pharmaceutical supply chain scheduling system is used to implement the above-mentioned reinforcement learning-based pharmaceutical supply chain scheduling method, the system comprising:

[0045] The thermal risk analysis module is used to obtain the internal temperature of the carriage, the external ambient temperature, and the thermal conductivity, calculate the heat penetration rate, and generate the thermal safety time limit value by combining the highest critical temperature value.

[0046] The logistics route calculation module is used to identify candidate delivery nodes by location coordinates and plan the shortest driving route, and calculate the estimated travel time by combining mileage, congestion index and driving speed values.

[0047] The constraint verification and filtering module is used to calculate the estimated arrival time based on the delivery deadline value and the system time value, and to generate an action mask vector by comparing the estimated travel time value with the thermal safety time limit value and the estimated arrival time value with the delivery deadline value.

[0048] The intelligent scheduling decision module is used to build and run a deep neural network to output the original score value, perform logical processing by combining the action mask vector with the negative infinity value, and generate the action selection probability distribution through the Softmax function to output the vehicle scheduling instruction.

[0049] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0050] In this invention, the heat penetration rate is accurately calculated by collecting the temperature difference and thermal conductivity inside and outside the vehicle compartment. The exact thermal safety time limit is derived by combining the highest critical temperature. The driving speed is corrected by the congestion index to predict the travel time. The thermal safety time limit and the delivery deadline are used as dual hard constraints to construct an action mask. Invalid nodes that cannot meet the cold chain safety requirements are eliminated. Multidimensional environmental parameters are input into a deep neural network and a probability distribution is generated under the mask constraint to ensure that the scheduling instructions strictly follow the temperature control safety threshold, effectively avoid the risk of drug thermal failure and improve the accuracy of cold chain logistics distribution. Attached Figure Description

[0051] Figure 1 This is the main flowchart of the pharmaceutical supply chain scheduling method of the present invention;

[0052] Figure 2 This is a flowchart of the thermal safety time limit calculation for this invention;

[0053] Figure 3 This is a flowchart illustrating the estimated travel time calculation for this invention.

[0054] Figure 4 This is a flowchart of the action mask vector generation process of the present invention;

[0055] Figure 5 This is a flowchart of the vehicle dispatch instruction generation process of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the software-based technical solution is described in detail below with reference to system architecture diagrams and embodiments. It should be understood that the specific embodiments described herein are only for explaining the technical solutions of this invention and do not constitute a limitation on the scope of protection.

[0057] In the description of this invention, the system architecture relationships or data processing flows indicated by terms such as "layer," "module," "interface," "data flow," "client," and "server" are all defined based on the architecture diagram or flowchart corresponding to the embodiments. This way of describing is only used to clearly illustrate the logical relationships between the elements in the technical solution, and not to limit the physical deployment form. The term "multiple" includes two or more technical units, including but not limited to multiple data nodes, processing threads, service instances, or functional components and other scalable elements. The specific number is determined according to the actual business scenario and needs to be specifically specified.

[0058] Please see Figure 1 and Figure 2 This invention provides a technical solution: a pharmaceutical supply chain scheduling method based on reinforcement learning, comprising the following steps:

[0059] S1: Obtain the temperature value inside the carriage, the temperature value outside the carriage, and the thermal conductivity value. Subtract the temperature value inside the carriage from the temperature value outside the carriage and multiply by the thermal conductivity value to obtain the heat penetration rate value. Subtract the temperature value inside the carriage from the maximum critical temperature value and divide by the heat penetration rate value to obtain the thermal safety time limit value.

[0060] The calculation process for the numerical values ​​of heat penetration rate and thermal safety time limit includes:

[0061] S11: Real-time monitoring and collection of the internal temperature of the refrigerated truck compartment, the external ambient temperature of the vehicle, and the thermal conductivity of the refrigerated compartment material.

[0062] S12: Calculate the temperature difference between the external ambient temperature and the interior temperature of the carriage, multiply the temperature difference by the thermal conductivity value to generate the heat penetration rate value.

[0063] S13: Obtain the preset maximum critical temperature value of the drug, calculate the difference between the maximum critical temperature value and the temperature value inside the carriage, and divide the difference by the heat penetration rate value to generate the thermal safety time limit value.

[0064] The process of obtaining the thermal conductivity value includes:

[0065] Obtain the basic thermal conductivity and material aging factor of the refrigerated box, and calculate the absolute temperature difference between the external ambient temperature and the internal temperature of the compartment in real time.

[0066] Based on the influence of temperature difference on the thermal conductivity of materials, the thermal conductivity value is calculated using a linear correction model, i.e.:

[0067] ;

[0068] in, Represents the thermal conductivity value. Represents the basic thermal conductivity. Represents the temperature difference correction factor. This represents the absolute temperature difference between the external ambient temperature and the interior temperature of the vehicle.

[0069] The system first activates the onboard terminal's data acquisition module, using high-precision PT100 temperature sensors positioned at the four corners (front, back, left, right) and center of the refrigerated truck compartment to read the interior temperature in real-time at 10-second intervals. The system then transmits the temperature readings from the five monitoring points to the onboard central processing unit, which calculates the current interior temperature using an arithmetic mean. At the same time, the system calls upon the ambient temperature sensor installed outside the vehicle to collect the ambient temperature value of the environment in which the vehicle is located. The system retrieves the thermal conductivity values ​​of the refrigerator enclosure material from the material property database in the local storage unit. These values ​​include the basic thermal conductivity. and the pre-determined temperature difference correction factor The processor then calculates the heat penetration rate. The system uses the external ambient temperature value... Temperature values ​​inside the carriage Perform the subtraction operation to obtain the absolute temperature difference between the two. Subsequently, based on the effect of temperature difference on the material's thermal conductivity, the system uses a linear correction model to dynamically correct the thermal conductivity. In this calculation step, the following formula is introduced: ;

[0070] in, This represents the corrected thermal conductivity value, in units of... ; This represents the basic thermal conductivity under standard laboratory conditions. This represents the temperature difference correction factor, used to characterize the degree to which the thermal conductivity of a material changes linearly with increasing temperature difference; This represents the absolute temperature difference between the external ambient temperature and the interior temperature of the vehicle.

[0071] The thermal conductivity value mentioned above refers to a physical quantity that measures the ability of refrigerated truck body material to transfer heat per unit time, per unit area, and per unit temperature difference. Its value directly determines the rate at which external heat penetrates into the interior of the truck body.

[0072] The system performs calculations based on collected actual data: setting the basic thermal conductivity of the refrigerator compartment as read from the database. It is 0.018 Temperature difference correction coefficients were measured under different temperature difference conditions in the thermal protection materials laboratory. Set to 0.0045. The temperature value inside the carriage during the current data collection period. The external ambient temperature is 5.0 degrees Celsius. The temperature is 32.0 degrees Celsius. First, calculate the absolute temperature difference. The temperature is 27.0 degrees Celsius. Substituting these parameters into the linear correction model formula, the calculation is performed. ,Right now The results show that under conditions where the internal and external temperature difference reaches 27 degrees Celsius, the actual heat transfer performance of the refrigerated truck compartment decreases by approximately 12.15% compared to the baseline state. Subsequently, the system uses the corrected thermal conductivity value to calculate the heat penetration rate. The system performs a multiplication operation, converting the absolute temperature difference... Compared with the corrected thermal conductivity value Multiplication, that is Degrees Celsius per hour. This value represents the rate at which the interior temperature of the carriage rises per hour under the current insulation performance and temperature difference. Finally, the system calculates the thermal safety time limit. The system extracts the preset maximum critical temperature value for the currently transported medicines from the order metadata. The temperature is set to 8.0 degrees Celsius. The system calculates the difference between the highest critical temperature and the current interior temperature of the carriage, indicating a remaining safe temperature rise margin of 3.0 degrees Celsius. The system divides this difference by the calculated heat penetration rate to obtain the thermal safety time limit. ,Right now The system temporarily stores the thermal safety time limit value of 5.50 hours in a register, which will serve as the core constraint boundary for determining node reachability in subsequent step S3. The relevant environmental and material parameters are shown in Table 1.

[0073] Table 1. Environmental and Material Parameters for Cold Chain Logistics

[0074] ;

[0075] As shown in Table 1, the system records all the physical parameters and real-time monitoring data required for the calculation, providing data support for the accurate extrapolation of the thermal safety time limit; the final thermal safety time limit value is 5.50 hours.

[0076] Please see Figure 1 and Figure 3 S2: Identify candidate delivery nodes based on location coordinates and plan the shortest driving route. Obtain the mileage value, congestion index value and driving speed value of the shortest driving route. Divide the mileage value by the driving speed value after the congestion index value to obtain the estimated travel time value.

[0077] The process of identifying candidate delivery nodes, planning routes, and calculating travel time includes:

[0078] S21: Using the vehicle's current location coordinates as the center, search for the locations of orders with incomplete delivery tasks within a preset service radius and generate candidate delivery nodes;

[0079] S22: Construct a topological network from the current location coordinates to multiple candidate delivery nodes, use a path search algorithm to plan the shortest driving path, and extract the mileage value of the path;

[0080] S23: Obtain the real-time congestion index value and the designed driving speed value of the route segment, use the congestion index value to attenuate and correct the driving speed value to obtain the actual speed, and divide the mileage value by the actual speed to generate the estimated travel time value.

[0081] The calculation process for the corrected travel speed in the estimated travel time calculation includes:

[0082] By introducing a traffic impedance function and using the congestion index value to perform a nonlinear attenuation calculation on the designed driving speed of the road, a corrected actual driving speed is generated, i.e.:

[0083] ;

[0084] in, This represents the corrected actual driving speed. The numerical value representing the driving speed of the road design. This represents the congestion index value. Represents the road condition sensitivity coefficient. This represents a non-linear exponential factor.

[0085] The system first obtains the vehicle's current location coordinates through the onboard positioning module. Using these coordinates as the center, it scans the order database within a preset 50-kilometer service radius. The system filters out all orders marked as pending delivery or in delivery, extracts the destination location information of these orders, and generates a candidate delivery node set containing multiple latitude and longitude coordinates. For each candidate delivery node in the set, the system calls the path planning engine to construct a road topology network from the current vehicle location coordinates to that candidate delivery node. The system uses a path search algorithm to search for the connected sequence with the minimum weight in the topology network to determine the shortest travel path, and accumulates the length of all segments on this path to obtain the mileage value. Simultaneously, the system obtains real-time traffic flow data for each segment of the shortest travel route from the city traffic management center via a data interface, including the designed driving speed values ​​for the roads. and real-time congestion index values The processor then calculates the estimated travel time. Due to the complexity of actual road conditions, directly using the design speed would result in a significant deviation in time estimation; therefore, the system executes speed correction logic. In this calculation stage, the traffic impedance function formula is introduced: ;

[0086] in, Represents the corrected actual driving speed, in units of ; A numerical value representing the designed driving speed of the road; This represents the congestion index value, with a range of 0 to 10. This represents the road condition sensitivity coefficient, used to adjust the sensitivity of speed to changes in the congestion index; Represents a nonlinear exponential factor, used to simulate the collapse effect of traffic flow near the critical density; is the base of the natural logarithm.

[0087] The aforementioned traffic impedance function is a mathematical model used to describe how travel time or speed on a road segment changes with increasing traffic load. It can reflect the nonlinear hindering effect of traffic flow density on vehicle operating conditions.

[0088] The system performs calculations based on collected actual road network data: the system plans the shortest driving path to candidate delivery node A and obtains the mileage value of this path. The distance is 45.0 kilometers. This corresponds to the designed driving speed for the road conditions along the route. The speed limit is set to 60.0 km / h. Real-time congestion index values ​​are being obtained. The index is 6.2, indicating moderate congestion. Based on the analysis of the city's road traffic flow characteristics, a road condition sensitivity coefficient is set. The non-linear exponential factor is 0.08. The value is 1.4. First, the nonlinear term of the congestion index is calculated. ,Right now Calculate the decay exponent, i.e. Calculate the reduction factor, i.e. Substitute the above parameters into the formula to calculate the corrected actual driving speed. ,Right now The result indicates that under a congestion index of 6.2, the actual travel capacity of vehicles is reduced to approximately 35.77% of the design speed. Subsequently, the system calculates the estimated travel time using the mileage value and the actual travel speed. The system performs a division operation, converting the mileage value... Divide by the corrected actual driving speed ,Right now The system repeats the above calculation process for all candidate delivery nodes, associates the calculated estimated travel time with the corresponding node ID, and transmits it to the next decision-making step. Relevant road segment data and calculation results are shown in Table 2.

[0089] Table 2. Calculation of Traffic Parameters and Travel Time for Road Sections

[0090] ;

[0091] See Table 2, which lists the path parameters from the current location to node A and the speed results calculated by the formula; the final estimated travel time is 2.10 hours.

[0092] Please see Figure 1 and Figure 4 S3: Obtain the delivery deadline value of the candidate delivery node, add the system time value to the estimated travel time value to obtain the estimated arrival time value, compare the estimated travel time value with the thermal safety time limit value, and the estimated arrival time value with the delivery deadline value to mark the node reachability status, and generate an action mask vector.

[0093] The process of generating the action mask vector includes:

[0094] S31: Extract the delivery deadline values ​​of multiple candidate delivery nodes from the order information and obtain the current system time value;

[0095] S32: Add the system time value to the calculated estimated travel time value to generate the estimated arrival time value;

[0096] S33: Perform dual constraint verification to determine whether the estimated travel time is less than the thermal safety time limit and whether the estimated arrival time is less than the delivery deadline. If both conditions are met, mark the node reachability as valid; otherwise, mark it as invalid.

[0097] S34: Construct an action mask vector based on the reachable states of multiple nodes, where valid states correspond to a value of 1 and invalid states correspond to a value of 0.

[0098] The logic for marking the reachability state of a node includes:

[0099] Establish logical AND operation rules to determine a node as reachable only when both the thermal safety constraint and the time window delivery constraint are true.

[0100] If the estimated travel time is greater than or equal to the thermal safety time limit, it indicates that the drug will fail thermally before arrival, and the node reachability status will be set to invalid.

[0101] If the estimated arrival time is greater than the delivery deadline, it indicates that even if the drug is safe, the mission is overdue, and the node's reachability status will be considered invalid.

[0102] The system first retrieves the delivery deadlines for all candidate delivery nodes from the order management system's database. The system's internal clock synchronizes with a network time protocol server to obtain the current system time, accurate to the minute. Assuming the current system time is 10:00, the delivery deadline for candidate delivery node A is 12:30. The processor then generates the estimated arrival time. The system calls the estimated travel time calculated in step S2, which is 2.10 hours for node A, converts it to a 2 hours 06 minutes time format, and adds it to the current system time of 10:00 to obtain the estimated arrival time. The time is 12:06. Subsequently, the system performs a crucial double constraint check to generate an action mask vector. This process involves two independent comparison logic judgment operations, ultimately synthesizing the judgment result through a logical AND operation. The system then calls the thermal safety time limit value calculated in step S1. That is, 5.50 hours is used as the first constraint threshold, and the estimated travel time value calculated in step S2 is called. That is, 2.10 hours is used as the first comparison item. Simultaneously, the system retrieves the delivery deadline value from the order information. That is, 12:30 is used as the second constraint threshold, and the estimated arrival time value calculated in this step is called. That is, 12:06 is the second item to be compared.

[0103] The aforementioned action mask vector is a binary vector used in reinforcement learning or decision-making systems to indicate which actions in the action space are allowed to be executed in the current state. A value of 1 represents a valid action, and a value of 0 represents an invalid action.

[0104] For node A, the following steps are performed: First, a thermal safety constraint check is executed, i.e., a check is made. Substitute the value as If the judgment result is true, it indicates that the vehicle can reach the node before the drug experiences thermal failure. Next, the time window delivery constraint judgment is performed, i.e., the judgment... Substitute the value as The judgment result is true, indicating that the vehicle can be delivered before the customer's requested deadline. The system performs a logical AND operation; since both conditions are true, the logical synthesis result is true. Based on the judgment result, the system marks the reachability state of node A as valid and fills in the value 1 at the index position of node A in the action mask vector. A case study is performed for another hypothetical node B to demonstrate the marking logic for an invalid state: Assume that the estimated travel time for node B is 6.0 hours, the estimated arrival time is 16:00, and the delivery deadline is 17:00. Condition one is judged as follows: The result was false, meaning the medicine would become ineffective due to excessive temperature during transport. Although condition two... It may be true, but the logical AND operation result is false. The system marks the reachability state of node B as invalid and fills the corresponding index position of node B in the action mask vector with the value 0. The system traverses all candidate delivery nodes and finally generates a binary action mask vector with a length equal to the number of candidate nodes.

[0105] Please see Figure 1 and Figure 5 S4: Input the temperature value inside the carriage, the temperature value of the external environment, and the location coordinates into the deep neural network to output the original score value. Replace the original score value corresponding to the zero value in the action mask vector with the negative infinity value. Obtain the action selection probability distribution through the Softmax function. Generate vehicle dispatch instructions based on the action selection probability distribution.

[0106] The process of generating the original score values ​​and outputting vehicle dispatch instructions includes:

[0107] S41: Normalize the temperature values ​​inside the carriage, the temperature values ​​of the external environment, and the location coordinates, construct a multi-dimensional state feature vector, input it into a deep neural network, and output the original score value after processing by the network layer.

[0108] S42: Use the motion mask vector to mask the original score value, and forcibly replace the original score value corresponding to the zero value position in the motion mask vector with a negative infinity value.

[0109] S43: Perform Softmax normalization on the replaced score value sequence to generate an action selection probability distribution, select the target node according to the distribution, and generate vehicle dispatch instructions;

[0110] The process of building a deep neural network includes:

[0111] Construct a network architecture that includes an input layer, a feature extraction layer, and a decision output layer;

[0112] A multi-head attention mechanism sublayer is set in the feature extraction layer to capture the spatial dependencies in the position coordinate sequence, and a fully connected layer is set to fuse temperature features and spatial features.

[0113] In the decision output layer, set up a number of neurons that are the same as the number of candidate delivery nodes, and output the original score value through a linear activation function.

[0114] The system first uses the interior temperature of the vehicle (5.0 degrees Celsius), the external ambient temperature (32.0 degrees Celsius), and the vehicle's current coordinates obtained in step S1 as raw state data. The system standardizes this data to construct a multi-dimensional state feature vector. The processor inputs this feature vector into a pre-trained deep neural network. The input layer of the neural network receives the feature vector, which is then processed by a feature extraction layer containing a multi-head attention mechanism to extract the temperature change trend and its spatial location dependence. The signal is then passed to a fully connected layer for feature fusion, finally reaching the decision output layer. The decision output layer contains neurons equal to the number of candidate delivery nodes. Each neuron outputs a scalar value, i.e., the original score value, through a linear activation function. Assume the neural network outputs a sequence of original score values ​​for three candidate nodes A, B, and C. for Node B, despite having the highest score, was marked as an invalid node in step S3. The system then performs a masking operation. The system calls the action mask vector generated in step S3. Its value is The system iterates through the action mask vector, identifying the index positions where the value is 0, namely nodes B and C. The system then forcibly replaces the values ​​at the corresponding invalid positions in the original score value sequence with negative infinity. The score value sequence after this replacement becomes... Its value is Subsequently, the system performs Softmax normalization on the replaced score sequence to generate an action selection probability distribution. In this calculation step, the Softmax function formula is introduced: ;

[0115] in, Representing the The probability of an action being selected; This represents the score value corresponding to the i-th action (after masking). is the base of the natural logarithm; This represents the sum of index scores corresponding to all possible actions.

[0116] The Softmax function mentioned above is a mathematical function that maps any real number vector to a probability distribution vector. Each element in its output vector is between 0 and 1, and the sum of all elements is 1.

[0117] The system performs calculations based on the replaced score value sequence: for node A, calculate... For nodes B and C, since their scores are negative infinity, Approaching 0. Calculate the sum of the denominators, i.e. Calculate the selection probability of each node: the probability of node A. The probability of node B The probability of node C The generated action selection probability distribution is as follows: Finally, the system generates vehicle dispatch instructions based on this probability distribution. Since the probability of node A being selected is 100%, the system locks node A as the target delivery node, generates dispatch instructions containing the location coordinates of node A and the recommended driving route, and sends a navigation request to the driver through the in-vehicle display screen; the final vehicle dispatch instruction is to go to node A.

[0118] A reinforcement learning-based pharmaceutical supply chain scheduling system is provided. This system is used to execute the aforementioned reinforcement learning-based pharmaceutical supply chain scheduling method. The system includes:

[0119] The thermal risk analysis module is used to obtain the internal temperature of the carriage, the external ambient temperature, and the thermal conductivity, calculate the heat penetration rate, and generate the thermal safety time limit value by combining the highest critical temperature value.

[0120] The logistics route calculation module is used to identify candidate delivery nodes by location coordinates and plan the shortest driving route, and calculate the estimated travel time by combining mileage, congestion index and driving speed values.

[0121] The constraint verification and filtering module is used to calculate the estimated arrival time based on the delivery deadline and the system time, and to generate an action mask vector by comparing the estimated travel time with the thermal safety time limit and the estimated arrival time with the delivery deadline.

[0122] The intelligent scheduling decision module is used to build and run a deep neural network to output the original score value, combine the action mask vector with the negative infinity value for logical processing, and generate the action selection probability distribution through the Softmax function to output the vehicle scheduling instruction.

[0123] The above embodiments illustrate preferred embodiments of the present invention. Any equivalent adjustments to the technical solution based on software engineering methods are within the scope of protection, including but not limited to: implementing algorithm logic using different programming languages, refactoring functional modules into services, adjusting data interaction protocols, and optimizing resource scheduling strategies. Any implementation scheme derived from reasonable modifications to the data processing flow, service call chain, or system architecture layer without departing from the core technology of the present invention should be considered within the protection scope defined by the technical solution of the present invention.

Claims

1. A pharmaceutical supply chain scheduling method based on reinforcement learning, characterized in that, Includes the following steps: S1: Obtain the temperature value inside the carriage, the temperature value outside the carriage, and the thermal conductivity value. Subtract the temperature value inside the carriage from the temperature value outside the carriage and multiply by the thermal conductivity value to obtain the heat penetration rate value. Subtract the temperature value inside the carriage from the maximum critical temperature value and divide by the heat penetration rate value to obtain the thermal safety time limit value. S2: Identify candidate delivery nodes based on location coordinates and plan the shortest driving route. Obtain the mileage value, congestion index value and driving speed value of the shortest driving route. Divide the mileage value by the driving speed value after correction by the congestion index value to obtain the estimated travel time value. S3: Obtain the delivery deadline value of the candidate delivery node, add the system time value to the estimated travel time value to obtain the estimated arrival time value, compare the estimated travel time value with the thermal safety time limit value, and compare the estimated arrival time value with the delivery deadline value to mark the node reachability status, and generate an action mask vector. S4: Input the temperature value inside the carriage, the temperature value of the external environment and the position coordinates into the deep neural network to output the original score value, replace the original score value corresponding to the zero value in the action mask vector with a negative infinity value, obtain the action selection probability distribution through the Softmax function, and generate vehicle dispatch instructions based on the action selection probability distribution. The process of identifying candidate delivery nodes, planning routes, and calculating travel time includes: S21: Using the vehicle's current location coordinates as the center, retrieve the locations of orders with incomplete delivery tasks within a preset service radius, and generate the candidate delivery nodes; S22: Construct a topology network from the current location coordinates to multiple candidate delivery nodes, plan the shortest driving path using a path search algorithm, and extract the mileage value of the path; S23: Obtain the real-time congestion index value and the designed driving speed value of the route segment, use the congestion index value to attenuate and correct the driving speed value to obtain the actual speed, and divide the mileage value by the actual speed to generate the estimated travel time value. The calculation process for the corrected travel speed value in the calculation of the estimated travel time includes: By introducing a traffic impedance function and using the congestion index value to perform a nonlinear attenuation calculation on the designed driving speed value, a corrected actual driving speed is generated, i.e.: ； in, This represents the corrected actual driving speed. The stated driving speed value represents the road design. This represents the congestion index value. Represents the road condition sensitivity coefficient. This represents a non-linear exponential factor.

2. The pharmaceutical supply chain scheduling method based on reinforcement learning according to claim 1, characterized in that, The calculation process for the heat penetration rate and the thermal safety time limit includes: S11: Real-time monitoring and collection of the internal temperature of the medical refrigerated vehicle, the external ambient temperature of the vehicle's environment, and the thermal conductivity of the refrigerated container material. S12: Calculate the temperature difference between the external ambient temperature and the interior temperature of the carriage, and multiply the temperature difference by the thermal conductivity value to generate the heat penetration rate value; S13: Obtain the preset maximum critical temperature value of the drug, calculate the difference between the maximum critical temperature value and the temperature value inside the carriage, and divide the difference by the heat penetration rate value to generate the thermal safety time limit value.

3. The pharmaceutical supply chain scheduling method based on reinforcement learning according to claim 1, characterized in that, The process of generating the action mask vector includes: S31: Extract the delivery deadline values ​​of multiple candidate delivery nodes from the order information, and obtain the current system time value; S32: Add the system time value to the calculated estimated travel time value to generate the estimated arrival time value; S33: Perform a dual constraint check to determine whether the estimated travel time is less than the thermal safety time limit and whether the estimated arrival time is less than the delivery deadline. If both conditions are met, mark the node reachability status as valid; otherwise, mark it as invalid. S34: Construct the action mask vector based on the reachable states of multiple nodes, where valid states correspond to the value 1 and invalid states correspond to the value 0.

4. The pharmaceutical supply chain scheduling method based on reinforcement learning according to claim 1, characterized in that, The process of generating the original score value and outputting the vehicle dispatch instruction includes: S41: Normalize the temperature value inside the carriage, the temperature value of the external environment, and the position coordinates, construct a multi-dimensional state feature vector, input it into the deep neural network, and output the original score value after processing by the network layer. S42: Use the action mask vector to perform a masking operation on the original score value, and forcibly replace the original score value corresponding to the zero value position in the action mask vector with the negative infinity value; S43: Perform Softmax normalization on the replaced score value sequence to generate the action selection probability distribution, and select the target node according to the distribution to generate the vehicle dispatch instruction.

5. The pharmaceutical supply chain scheduling method based on reinforcement learning according to claim 2, characterized in that, The process of obtaining the thermal conductivity value includes: The basic thermal conductivity and material aging factor of the refrigerated box are obtained, and the absolute temperature difference between the external ambient temperature and the internal temperature of the compartment is calculated in real time. Based on the influence of temperature difference on the thermal conductivity of materials, the thermal conductivity value is calculated using a linear correction model, i.e.: ； in, This represents the stated thermal conductivity value. Represents the basic thermal conductivity. Represents the temperature difference correction factor. This represents the absolute temperature difference between the external ambient temperature and the interior temperature of the vehicle compartment.

6. The pharmaceutical supply chain scheduling method based on reinforcement learning according to claim 3, characterized in that, The marking logic for the reachability state of a node includes: Establish logical AND operation rules to determine a node as reachable only when both the thermal safety constraint and the time window delivery constraint are true. If the estimated travel time is greater than or equal to the thermal safety time limit, it indicates that the drug will fail thermally before arrival, and the node reachability status will be set to invalid. If the estimated arrival time is greater than the delivery deadline, it indicates that even if the drug is safe, the task is overdue, and the node reachability status is considered invalid.

7. The pharmaceutical supply chain scheduling method based on reinforcement learning according to claim 4, characterized in that, The construction process of the deep neural network includes: Construct a network architecture that includes an input layer, a feature extraction layer, and a decision output layer; A multi-head attention mechanism sublayer is set in the feature extraction layer to capture the spatial dependencies in the position coordinate sequence, and a fully connected layer is set to fuse temperature features and spatial features. In the decision output layer, a number of neurons are set up that are the same as the number of candidate delivery nodes, and the original score value is output through a linear activation function.

8. A pharmaceutical supply chain scheduling system based on reinforcement learning, characterized in that, The system is used to implement the reinforcement learning-based pharmaceutical supply chain scheduling method according to any one of claims 1-7, and the system comprises: The thermal risk analysis module is used to obtain the internal temperature of the carriage, the external ambient temperature, and the thermal conductivity, calculate the heat penetration rate, and generate the thermal safety time limit value by combining the highest critical temperature value. The logistics route calculation module is used to identify candidate delivery nodes by location coordinates and plan the shortest driving route, and calculate the estimated travel time by combining mileage, congestion index and driving speed values. The constraint verification and filtering module is used to calculate the estimated arrival time based on the delivery deadline value and the system time value, and to generate an action mask vector by comparing the estimated travel time value with the thermal safety time limit value and the estimated arrival time value with the delivery deadline value. The intelligent scheduling decision module is used to build and run a deep neural network to output the original score value, perform logical processing by combining the action mask vector with the negative infinity value, and generate the action selection probability distribution through the Softmax function to output the vehicle scheduling instruction.

Images