Family medical care path planning and scheduling method based on deep reinforcement learning

Abstract

The invention discloses a family medical care path planning and scheduling method based on deep reinforcement learning, and the method comprises the steps: obtaining HHCRSP instance data, modeling the HHCRSP instance data into a mixed integer linear programming model, carrying out the problem decomposition and sorting, and obtaining a plurality of VRP sub-problems with the types of services needed by patients as the grouping basis and the service dependency as the solving sequence; modeling the solving process of each VRP sub-problem into a Markov decision process, and solving the constructed Markov decision process according to a solving sequence through a strategy network to obtain a service path scheme and a service timetable thereof; and integrating a service path scheme obtained by solving each VRP sub-problem with a service timetable to form a planning and scheduling scheme including family medical care paths of all patients. According to the method, through key technologies such as problem decomposition, a neural network parameterization strategy, service embedding and constraint perception mask, the technical problems of the HHCRSP in the aspects of expandability, real-time performance, generalization ability and complex constraint processing are systematically solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical service resource allocation, and particularly relates to a home medical care path planning and scheduling method based on deep reinforcement learning. BACKGROUND

[0002] With the intensification of population aging and the rapid growth of home medical service demand, home medical care routing and scheduling problem (HHCRSP) has gradually become a core problem in the field of medical service resource allocation. This kind of problem usually needs to reasonably arrange medical personnel (such as nurses, nursing staff, etc.) with specific qualifications to provide door-to-door services for patients within a given time window, while meeting the following constraint conditions: service time window limit, service time difference, personnel qualification matching, multi-service dependency relationship, path and scheduling coordination, and multi-team parallel execution.

[0003] HHCRSP essentially belongs to a complex extension of vehicle routing problem (VRP), and has characteristics such as path optimization, scheduling planning and multi-resource allocation, and its combination complexity is much higher than that of traditional traveling salesman problem (TSP) and capacity constrained vehicle routing problem (CVRP). In practical applications, common optimization objectives include minimizing total travel distance, total service cost or response time, while taking into account personnel load balancing and service quality.

[0004] At present, the mainstream methods for solving HHCRSP mainly include mathematical programming model, decomposition strategy, construction heuristic method and meta-heuristic algorithm, etc. These methods mostly rely on artificial design of algorithm process, and can be used for solving HHCRSP of a certain scale, but their universality and scalability are limited, and there are problems such as poor scalability, insufficient solving efficiency, dependence on artificial design, weak generalization ability, and difficulty in handling service and coordination constraints.

[0005] Further, for traveling salesman problem and capacity constrained vehicle routing problem, intelligent solving methods based on deep reinforcement learning (DRL) have gradually appeared. Such methods usually use encoder-decoder architecture or attention mechanism to construct path as a sequence decision process. These methods are mostly applied to relatively standardized problem scenarios such as TSP or CVRP, and the optimization objective is mainly to minimize travel distance, the constraint model is simple, and only single service type and homogeneous personnel resources are considered, so there are still the following deficiencies when used for HHCRSP: (1) Only applicable to single service type or homogeneous resources, without building a task allocation mechanism suitable for multi-team and multi-role; (2) Lack of modeling ability for multi-service dependency relationship, unable to handle simultaneous services, sequential services or multi-person collaboration tasks; (3) The path planning and scheduling constraints are not integrated, and it is difficult to simultaneously process time windows, service order and personnel qualification matching; (4) There is a lack of modeling module for double services and heterogeneous constraints, which cannot be directly migrated to the home medical scene; (5) There is a lack of scalable framework and training mechanism for actual medical scheduling needs; (6) There are obvious defects in dynamic scheduling, real-time response and complex constraint processing.

[0006] Therefore, the existing DRL-VRP method cannot meet the comprehensive needs of path optimization, service scheduling, multi-team collaboration and complex service dependency in home medical scheduling, and it is difficult to be directly promoted in practical applications (the problems and scale solved by the existing DRL method are too simple).

[0007] In summary, when facing large-scale data, complex constraints and dynamic needs, the existing solving method often cannot balance real-time performance and solving quality, especially in the multi-service, multi-team and multi-dependent scene of the medical industry. SUMMARY

[0008] In view of the above problems in the prior art, the home medical care path planning and scheduling method based on deep reinforcement learning provided by the present application solves the problem that the existing method cannot balance real-time performance and solving quality when facing HHCRSP such a large-scale problem, and cannot realize high-quality path planning and scheduling in a complex scene of multi-service, multi-team and multi-dependence.

[0009] In order to achieve the above invention purpose, the technical scheme adopted by the present application is: a home medical care path planning and scheduling method based on deep reinforcement learning, comprising the following steps: S100, obtaining instance data of a home medical care scheduling and path planning problem HHCRSP, and modeling it as a mixed integer linear programming model; S200, decomposing and sorting the mixed integer linear programming model, obtaining a plurality of VRP sub-problems based on the service type required by the patient as the grouping basis and the service dependency relationship as the solving order; S300, modeling the solving process of each VRP sub-problem as a Markov decision process; S400, constructing and training an encoder-decoder neural network based on an attention mechanism as a policy network, and then solving the constructed Markov decision process according to the solving order to obtain the service path scheme and service timetable corresponding to each VRP sub-problem; S500, when all service type corresponding VRP sub-problems are completed to solve, the service path scheme obtained by solving each VRP sub-problem is integrated with the service timetable to form a planning and scheduling scheme containing all patient home medical care paths.

[0010] Further, in the step S100, the objective function of the mixed integer linear programming model is to minimize the total travel distance of all medical staff, which is expressed as: In the formula, denotes the distance between the patient and the patient , denotes the decision variable of the medical staff traveling from the patient to the patient , if yes , otherwise, , denotes all location sets, denotes the set of medical staff, denotes all patient sets; The constraint conditions of the mixed integer linear programming model include: In the formula, denotes the decision variable of the medical staff traveling from the central office to the patient , denotes the decision variable of the medical staff traveling from the patient to the central office , denotes the decision variable of the medical staff providing service at the patient , if providing service , otherwise , denotes a service required number of medical staff, denotes a patient at which a service is needed, if needed , otherwise , denotes a medical staff with the qualification to provide a service, if qualified , otherwise , otherwise , denotes a start time of a service provided by a medical staff at a patient , denotes an earliest service start time of a patient , denotes a duration of a service provided at a patient , denotes a latest service start time of a patient , denotes a large constant used in the constraints, denotes a start time of a service provided by a medical staff at a patient , denotes a start time of a service provided by a medical staff at a patient , denotes a start time of a service provided by a medical staff at a patient , denotes an end time of a service provided by a medical staff at a patient , denotes an end time of a service provided by a medical staff at a patient , denotes a location minimum time interval between two services, denotes a location maximum time interval between two services, denotes a set of double service patients, denotes a set of patients requiring simultaneous service, denotes a set of patients requiring priority order service, denotes a set of service types, subscript denotes different service indices, subscript denotes different location indices, subscript denotes different medical staff indices.

[0011] Further, the step S200 comprises the following sub-steps: S201, dividing the patient set into different groups according to the service types required by the patients, including a single-service patient set and a double-service patient set; S202, dividing the double-service patient set according to service dependency relationship, and obtaining a synchronous double-service patient set and a sequential double-service patient set; S203, sorting all service types, and ensuring that the index of the first service is always smaller than the index of the second service for double-service patients; S204, according to the determined service order, sequentially grouping all patients requiring the same type of service into a VRP sub-problem corresponding to the type of service.

[0012] Further, in the step S300, in the first VRP sub-problem corresponding Markov decision process: the state in the state space contains static information and dynamic information; the static information includes the geographic coordinates of the patients and the central office, the initial time window and the service duration; the dynamic information includes the current last-visited patient location, the set of visited patients and the cumulative travel time; the action in the action space is to select the next patient to be served from the set of currently unvisited patients, or to return to the central office; the set of actions in the action space is a node set , 0 represents the central office, and the rest represent patient nodes; state transition is after performing the action , the state is transferred from to , including updating the current location to the selected patient location, adding the patient to the visited set, and updating the cumulative travel time;

[0013] Further, in the step S400, the encoder in the policy network comprises an initial embedding generation unit and a multi-layer attention update unit; The initial embedding generation unit is used for linear transformation of the initial node features and time window features of each patient, and superimposes the linear projection results to form an initial node embedding; The multi-layer attention updating unit is configured to update the initial node embedding through multiple attention layers, in each of which the node embedding is updated through a multi-head attention mechanism and a feed-forward mechanism, and combined with a residual connection and batch normalization to generate a final node embedding and input to the decoder.

[0014] Further, in the step S400, the decoder in the policy network comprises a context embedding construction unit, a constraint mask mechanism and an action selection unit. The context embedding construction unit is configured to combine the final node embedding output by the encoder, the current service type embedding, the next access node embedding and the service completion time at the last node in the current solution to construct a context embedding reflecting the dynamic information in the current solution. The constraint mask mechanism is configured to select the next access node according to a dynamic constraint mask, wherein the dynamic constraint mask comprises a current patient constraint mask and a central office constraint mask. The action selection unit is configured to calculate the attention score of each node corresponding to the context embedding and apply the constraint mask to select an action.

[0015] Further, in the step S400, in the training process of the policy network: For the first VRP sub-problem corresponding to the policy network, the expected cost of the solution generated by the policy is minimized as the training target, and the gradient descent method is adopted to optimize the parameters of the policy network. The expected cost of the solution generated by the policy is minimized as the training target, and the gradient descent method is adopted to optimize the parameters of the policy network. The training loss function is: In the formula, represents the total length of the service path generated by the policy for the VRP sub-problem , and represents the expected value of the solution cost under the policy . The optimization formula of the parameters of the policy network is: In the formula, represents the gradient, represents the cost obtained by the baseline model for the VRP sub-problem , and represents the derivative of the log policy probability with respect to the parameters, represents the index of the VRP sub-problem, represents the HHCRSP instance, ​representing patients corresponding instance data, representing the number of instances in the training batch.

[0016] Further, in the step S400, when solving the constructed Markov decision process by using the trained policy network, the service path scheme corresponding to each VRP sub-problem is output by greedy decoding or sampling decoding. When the service path scheme is output based on greedy decoding, for the i-th VRP sub-problem In the service path scheme, the selected action at the time step is the node with the highest selection probability in the policy output by the policy network, which is represented as: In the formula, represents the action sequence generated by the policy network from time 0 to time t-1, represents the probability of selecting the patient under the condition of the given sub-problem and the action sequence , represents the node index corresponding to the patient index; When the service path scheme is output based on sampling decoding, for the i-th VRP sub-problem , the policy output by the policy network generates a number of solutions , and the solution with the lowest cost among them is taken as the final generated solution, thereby constituting the service path scheme; wherein, is the solution obtained by sampling according to the policy output by the policy network, represents the index of the generated solution, represents the total travel distance for visiting all patients.

[0017] Further, in the step S400, after solving the current VRP sub-problem, the start time and end time of service at each patient are calculated in sequence according to the generated service path scheme by dynamically updating the time window constraint mechanism, thereby determining the service schedule corresponding to the service path scheme. In the service schedule, for the set of double-service patients, the time window for completing service at the patient is updated according to the preset priority, and the update formula is: In the formula,​ respectively represent the start and end points of the time window at the time of service to the patient, represent the service maintained at the patient time window, represent the planned start time of the first service at the patient , represent the specified time difference of the patient , represent the earliest start time of the first service of the patient , represent the latest start time of the first service of the patient ; wherein the preset priority is and , represent the first service, represent the second service, service or ; the time window of the second service is adjusted according to the planned start time of the previous service and and .

[0018] Further, when the patient is dynamically updated, a rolling optimization strategy is adopted to update the planning and scheduling scheme containing all patient home medical care paths, including: fix the service path and service schedule that have been executed before the current time; merge the current unserved patient set and the updated patient set to form a new patient set, and update the time window of each patient service; re-solve each VRP sub-problem in the updated patient set using the trained policy network, obtain the corresponding service path scheme and its service schedule, and execute until the next patient update.

[0019] The beneficial effects of the present application are: The present application solves the technical problems of HHCRSP in scalability, real-time performance, generalization ability and complex constraint processing by problem decomposition, neural network parameterized strategy, service embedding and constraint perception mask, etc. Compared with the prior art, the present application has the following significant advantages: (1) The existing mathematical programming method and meta-heuristic algorithm have exponential growth in computational complexity when dealing with large-scale HHCRSP, and the number of variables and constraints increases with the number of patients n in O(n 2 ) or O(n 3) growth, resulting in technical defects that the solving time increases sharply with the problem size and even cannot be solved within a reasonable time, the present application provides a construction strategy based on neural network parameterization, which can generate a solution through one forward propagation, reduces the computational complexity of single solving to O(n), and realizes the solving of more than 100 patient scale problems within milliseconds, and can be directly extended to 200-300 patient scale without retraining the model.

[0020] (2) For the technical defects of traditional MILP solvers and meta-heuristic algorithms (such as genetic algorithm, simulated annealing, large neighborhood search, adaptive variable neighborhood search) that need multiple iteration searches or branch and bound processes, the single instance solving time is several minutes to several tens of minutes, which cannot meet the real-time scheduling demand of responding within seconds or even milliseconds in the dynamic patient arrival scene, the present application realizes the millisecond-level solution generation speed in the greedy decoding mode through the pre-training strategy network, and obtains high-quality solutions within seconds in the sampling decoding mode, which is hundreds to thousands of times faster than the traditional method, thereby supporting the immediate path re-planning of patients in real-time arrival in dynamic environment.

[0021] (3) For the technical defects of existing heuristic and meta-heuristic methods that need to manually design neighborhood operators (such as 2-opt, relocate, exchange, Or-opt, cross-exchange), destroy-repair operators, evaluation functions and parameter configurations (such as neighborhood size, temperature coefficient, iteration number, population size, crossover probability, mutation probability, etc.), and these manually designed rules and parameters need to be re-optimized when the problem size changes, the parameter distribution changes or the constraint conditions are adjusted, resulting in poor generalization ability and high development cost of the algorithm, the present application uses an end-to-end deep reinforcement learning framework, learns the neural network parameters of the construction strategy on the training set, automatically extracts problem features and optimization modes, and after training, no manual parameter tuning is needed, and it can be directly applied to different problem sizes (from small to large) and different parameter distributions (uniform distribution, Gaussian distribution, Poisson distribution) test instances, realizing zero-shot generalization ability.

[0022] (4) For the technical defects of the existing method, when modeling and processing double-service synchronization constraints (two medical staffs must arrive at the patient at the same time to start service), sequence constraints (the second service must start within a specified time interval after the first service starts), and cross-person collaboration tasks, a large number of coupling constraints need to be introduced in MILP, or special repair operators need to be designed in meta-heuristic algorithms to maintain feasibility, resulting in significant increase in model complexity and exponential increase in solution difficulty. The present application solves the problem by decomposing the global HHCRSP into a series of sequentially solved VRP sub-problems according to the service type through the problem decomposition framework, and designs a dynamic time window updating mechanism. After solving the first service sub-problem, the time window of the second service is automatically updated according to the determined service start time and the specified time difference constraint. Among them, for the case of synchronous service, the start and end points of the time window of the second service are set to the actual start time of the first service; for the case of sequential service, the start point of the time window of the second service is set to the maximum of the minimum time interval and the original earliest time added to the start time of the first service, and the end point of the time window is set to the minimum of the maximum time interval and the original latest time added to the start time of the first service.

[0023] Through this decomposition-updating mechanism, the complex global coupling constraints are transformed into information transmission between sub-problems, reducing the difficulty of solving a single sub-problem, and greatly reducing the complexity of processing service dependencies.

[0024] (5) For the technical defects of the existing static optimization method, when using rolling optimization or re-solving in a dynamic scenario (patients arrive in real time, executed paths need to be fixed, and unserved patients need to be re-planned), each re-planning needs to start from scratch, and historical information cannot be used, resulting in a significant degradation in solution quality compared to the static scenario. The present application implicitly learns the long-term structural features of the problem when training the strategy network, fixes the paths executed before the current time in the dynamic scenario, and only applies the trained strategy to incrementally construct the paths for unserved patients. Using the global understanding ability of the encoder and the sequential decision-making ability of the decoder, the solution quality in the dynamic scenario can still be maintained similar to that in the static scenario, which has obvious advantages over traditional heuristic construction methods.

[0025] (6) The existing DRL methods (such as attention model, POMO, etc.) for TSP / CVRP only model single service type, homogeneous vehicle resources, and simple capacity constraints, lack modeling modules for multi-team heterogeneous resources (different qualified medical personnel), multiple service types, service dependency relationship (synchronization / precedence), and time window constraint, and cannot be directly migrated to the HHCRSP scene. The present application adds a service embedding module, a dynamic time window encoding module, and a constraint-aware mask module in the neural network architecture, fuses time window information into node embedding through linear projection in the encoder, introduces current service type embedding, next access node embedding, and current service completion time to form context embedding in the decoder, and designs a dynamic mask mechanism to exclude time window constraint violation and visited patients at each decision-making step, so that the DRL framework can effectively handle the complex constraints of HHCRSP. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 A flowchart of a home medical care path planning and scheduling method based on deep reinforcement learning is provided for the present application.

[0027] Figure 2 Single service and double service scenario diagrams in HHCRSP are provided for the present application.

[0028] Figure 3 A strategy network architecture diagram is provided for the present application. DETAILED DESCRIPTION

[0029] The specific embodiments of the present application are described below to facilitate understanding of the present application by those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all applications utilizing the concept of the present application are within the scope of protection.

[0030] The home medical care path planning and scheduling method based on deep reinforcement learning provided by the embodiments of the present application has the core idea of decomposing the complex HHCRSP into a series of vehicle path planning (VRP) sub-problems solved in order of service type, and automatically learning and constructing heuristic methods through a shared encoder-decoder neural network strategy.

[0031] REFERENCE Figure 1 The home medical care path planning and scheduling method comprises the following steps: S100, obtaining instance data of a home medical care scheduling and path planning problem HHCRSP, and modeling it as a mixed integer linear programming model; S200, problem decomposition and sequencing of the mixed integer linear programming model is performed to obtain a plurality of VRP sub-problems taking the service types required by the patients as grouping basis and the service dependency relationship as solving sequence; S300, the solving process of each VRP sub-problem is modeled as a Markov decision process; S400, an encoder-decoder neural network based on attention mechanism is constructed and trained as a policy network, and then the constructed Markov decision process is solved in sequence to obtain the service path scheme and service timetable corresponding to each VRP sub-problem; S500, after the VRP sub-problems corresponding to all service types are solved, the service path scheme and service timetable obtained by solving each VRP sub-problem are integrated to form a planning and scheduling scheme containing the home medical care path of all patients.

[0032] In step S100, the objective function of the mixed integer linear programming model is to minimize the total travel distance of all medical staff, which is expressed as: In the formula, denotes the distance between the patient and the patient , denotes the decision variable of the medical staff traveling from the patient to the patient , if yes , otherwise, , denotes all location sets, denotes the set of medical staff, denotes all patient sets; The constraint conditions of the mixed integer linear programming model include: (1) Ensure that the path of each medical staff starts and ends at the central office (node 0), which is expressed as: (2) If a medical staff visits a patient, it must leave the location, and the flow balance condition is constructed as: (3) Ensure that the service needs of each patient are fully met by assigning an exact number of qualified medical staff ( ), which is expressed as: (4) Ensure that medical staff can only be assigned to provide services when they have the necessary qualifications, which is expressed as: (5) Mandatory service provision and path connection, allowing healthcare professionals to provide services only when accessing the corresponding patient location, is represented as: (6) Each service must begin no earlier than the earliest time specified by the patient, as indicated by: (7) Ensure that each service is completed within the time window specified by the patient, as shown below: (8) Force the service start time of each medical staff member's route to progress according to the travel and service duration to prevent looping, which is expressed as: (9) Ensure that when two healthcare workers are providing services at the same location, they both begin their services simultaneously, as indicated by: (10) Process priority-based services, ensuring that the start time difference between the first and second services remains within a specified range, as expressed as: (11) Define variables: path ( ) and service distribution ( The variable is a binary variable, and the time variable is ( Let be a non-negative continuous variable, which is represented as: In the formula, Indicates medical staff From the central office Drive to the patient Decision variables, Indicates medical staff From the patient Drive to the central office Decision variables, Indicates medical staff In patients The office provides services The decision variables, if providing services, are... ,otherwise , Indicates service The number of medical staff required Indicates the patient The decision variables that need to be served, if necessary ,otherwise , Indicates medical staff Capable of providing services The decision variable is the qualification; if one possesses the qualification... ,otherwise , Indicates medical staff In patients The office provides services The start time, Indicates the patient The earliest service start time, Indicates the patient The office provides services Duration, Indicates the patient The latest service start time, This indicates the large constants used in the constraints. Indicates medical staff In patients The office provides services The start time, Indicates medical staff In patients The office provides services The start time, Indicates medical staff In patients The office provides services The start time, Indicates medical staff In patients The office provides services End time, Indicates medical staff In patients The office provides services End time, Indicates position The minimum time interval between two services Indicates position The maximum time interval between two services This indicates a group of patients receiving dual services. This indicates a group of patients who need to be served simultaneously. This represents a set of patients who require priority services. Represents a set of service types, with subscripts. Indicates different service indexes, subscripts The following table represents different location indices. This represents different indexes of healthcare professionals.

[0033] like Figure 2 As shown, step S200 includes the following sub-steps: S201, divide the patient set into different groups according to the required service types, including a single-service patient set and a double-service patient set; S202, divide the double-service patient set according to service dependency, and obtain a synchronous double-service patient set and a sequential double-service patient set; S203, sort all service types, and ensure that the index of the first service is always less than the index of the second service for double-service patients, so as to ensure the priority of service dependency; S204, according to the determined service order, sequentially form a VRP sub-problem corresponding to the service type for all patients requiring the same service.

[0034] Specifically, in Figure 2 , patients A, B, C and D require single service, while patients E and F require double service. For each double service, the black box represents the first service ( ), and the green box represents the second service ( ). It should be noted that a staff member can provide services to single-service patients and double-service patients at the same time in the same path. For example, staff member 1 first provides service to single-service patient A, and then provides service to double-service patient E. Patient E represents the case of synchronous service (i.e. ), at which time staff members 1 and 6 arrive at the same time. Patient F represents the case of priority double service (i.e. ), in which staff member 2 provides the first service, and then staff member 3 provides the second service. The time window of the second service is updated according to the planned start time of the first service .

[0035] For the single-service patient set , each patient requires only one service; for the double-service patient set , it is further divided into: synchronous double-service patient set : both services must start at the same time, i.e. ; sequential double-service patient set : there is a time sequence dependency between the two services, i.e. .

[0036] In step S300, in the Markov decision process corresponding to the th VRP sub-problem : the state The static information includes the geographic coordinates of the patients and the central office, the initial time window and the service duration; the dynamic information includes the current last visited patient position, the set of visited patients and the cumulative travel time; Actions in the action space To select the next patient to be served from the current set of unvisited patients, or to return to the central office; the set of actions in the action space is the set of nodes , 0 represents the central office, and the rest represent patient nodes; the dynamic mask excludes infeasible choices according to the time window and the sub-loop elimination constraints; The state transition is to perform the action , and the state is transferred from to , including updating the current position to the selected patient position, adding the patient to the visited set, and updating the cumulative travel time; The reward function is the negative value of the total distance after all patients are served; specifically, when all patients are served, the complete solution, i.e., the node sequence , is obtained; the reward is defined as the negative value of the total distance, i.e., , where calculates the total travel distance.

[0037] In step S400, a policy is parameterized constructed based on a policy network, which is shared across all sub-problems. Given an HHCRSP instance , the sub-problem corresponds to a VRP in which the healthcare team provides a specific service. The probability of generating a solution (i.e., a service path) can be factorized into the product of conditional probabilities: where the symbol represents the product of terms from time step to (i.e., the total number of steps to generate a complete solution). Each term represents the conditional probability of selecting the next action given the sub-problem and the previously generated partial solution . In other words, the policy constructs a complete solution step by step by dynamically calculating the probability of selecting the next patient at each decision step (conditional on the current partial solution).

[0038] As shown in Figure 3 , the encoder in the policy network includes an initial embedding generation unit and a multi-layer attention update unit; The initial embedding generation unit is configured to perform linear transformation on the initial node features and the time window features of each patient, and stack the linear projection results to form initial node embeddings; The multi-layer attention update unit is configured to update the initial node embeddings through multiple attention layers. In each attention layer, the node embeddings are updated through a multi-head attention mechanism and a feed-forward mechanism, and combined with a residual connection and batch normalization to generate final node embeddings, which are input to the decoder.

[0039] Specifically, in the encoder, the node raw features (including patient coordinates, service types, and time windows, etc.) are first mapped to initial embeddings through linear transformation. Specifically, for the central office node (i.e., depot), a learnable embedding is used as its initial embedding. For each patient node , the initial embedding is a linear projection of the raw features (i.e., patient coordinates and service types) plus linear projections of the start and end of the time window, which is expressed as: In the formula, , is a trainable parameter, denotes a concatenation operator.

[0040] In the multi-layer attention update unit, in each attention layer , the node embeddings are updated through a multi-head self-attention (MHA) layer and a feed-forward (FF) layer, each combined with a residual connection and batch normalization (BN): MHA allows the model to attend to different positions of the input sequence in parallel between different attention heads. In each head, information is extracted by computing query, key, and value vectors. Specifically, given the -th head , the attention score of node to node is computed as follows: where the layer index is omitted for brevity. and are trainable parameters that convert the node embeddings into query and key vectors. Given the attention scores, the aggregated node embeddings from different heads are obtained by: in, These are trainable parameters that convert node embeddings into value vectors. These are trainable parameters used for linear transformation. After MHA, residual connections and batch normalization are applied to process the node embeddings. Furthermore, the FF layer further updates the node embeddings through two fully connected layers: The ReLU function is used to activate the hidden representation. After the wheels are processed by the MHA and FF layers, the final node is embedded. Passed to the decoder.

[0041] like Figure 3 As shown, the decoder in the policy network includes a context embedding construction unit, a constraint masking mechanism, and an action selection unit; The context embedding construction unit is used to combine the final node embedding, current service embedding, next access node embedding, and service completion time at the last node of the current solution output by the encoder to construct a context embedding that reflects the dynamic information in the current solution. The constraint mask mechanism is used to select the next access node based on the dynamic constraint mask; the dynamic constraint mask includes the current patient constraint mask and the central office constraint mask. The action selection unit is used to calculate the attention score of the context embedding for each node and apply a constraint mask to select the action.

[0042] Specifically, in the context embedding construction unit, the decoder at each step Select the next node to visit based on the current state (i.e., the current partial solution). (i.e., patient nodes or depot nodes). To reflect the dynamic information in partial solutions, context embedding is used. Combining the encoder output and dynamic states (such as the current service type, last accessed node, and current time), the context embedding representation is obtained as follows: in It is the instance embedding (i.e., the average value of encoder node embeddings). It is the embedding of the current service (i.e., the corresponding team) (obtained through linear projection); It is the embedding of the last selected node (i.e., the most recently served patient); This is the service completion time at the last node of the current partial solution. The completion time is dynamically updated as follows: In the formula, It is the node selected in the current step. From arrive Travel time, In the patient Service hours at the location It is the team in the patient The office provides services The earliest start time.

[0043] In the constraint masking mechanism, to ensure the feasibility of the generated path, a dynamic masking scheme is applied when selecting the next node, including... (For patients) )and (For depot). The mask is dynamically calculated at each time step, eliminating infeasible options based on the following constraints: TRUE indicates that the node should be masked.

[0044] In the action selection unit, the decoder computes each node. Attention score And apply constraint masks. If the node Masked ( Its fraction is set as : Among them, nodes The final selection probability is calculated using the Softmax function: Using the above structure, a probabilistic policy learning framework based on an encoder-decoder neural network is implemented to construct routing solutions for healthcare workers providing services. Given each subproblem This can be summarized as follows: in, This represents the total number of steps required to construct a complete solution that resolves all services.

[0045] In this embodiment, the REINFORCE algorithm combined with shared baseline training is used to optimize performance. The shared baseline is used to reduce the variance of gradient estimation and accelerate training convergence. The training process mainly involves the periodic evaluation and updating of the policy network. Specifically, at the end of each training cycle, the current policy network and the baseline model (i.e., the current best policy network) are evaluated on the validation set. This evaluation compares the average cost achieved by the two policy networks on the validation set. If the average cost of the current policy network is lower than that of the baseline model, the current policy network replaces the baseline model as the new current best model; this process ensures that the baseline model always achieves the best performance relative to all historical models, thus providing a more accurate reference for policy gradients.

[0046] Based on this, in step S400, during the training process of the policy network: For the VRP subproblems The corresponding policy network is used to minimize the policy network policy. The expected cost of generating the solution is the training objective, and gradient descent is used to optimize the policy network parameters. Among them, the training loss function for: In the formula, The strategy is represented as a VRP subproblem. generated service path Total length, Indicating in strategy The expected value of the cost of the solution; where the total length, i.e. the cost is calculated as the total Euclidean distance of the solution (i.e. the employee path), and by minimizing this loss function, the model is incentivized to learn to generate HHCRSP instance solutions that effectively reduce the total travel distance.

[0047] Policy network parameters The optimized formula is: In the formula, Represents the gradient. The baseline model is represented as a VRP subproblem. The cost of obtaining This represents the derivative of the logarithmic policy probability with respect to the parameters. Indicates the VRP subproblem index. Indicates an HHCRSP instance, Indicates the patient Corresponding instance data, This indicates the number of instances in the training batch.

[0048] Among them, item The difference between the observed performance and the expected performance constitutes the advantage function. When the observation cost is lower than the baseline estimate, the advantage takes a negative value, thereby reducing the probability of generating the corresponding solution through strategy; conversely, when the observation cost exceeds the baseline estimate, the probability increases and encourages the generation of solutions.

[0049] During training iterations, we compute the product of the advantage function of each service and the corresponding policy log probability gradient for each instance, and then average these products across all services and the entire batch. The aggregated gradients are used by the optimizer to update the policy network.

[0050] In step S400, when solving the constructed Markov decision process using the trained policy network, the service path schemes corresponding to each VRP subproblem are output through greedy decoding or sampling decoding. When using a greedy decoding output service path scheme, for the first... VRP subproblems In the service path scheme, time step The following selection action The policy network outputs a policy that selects the node with the highest probability, which is represented as: In the formula, This represents the sequence of actions generated by the policy network from time 0 to time t-1. Indicates that in a given subproblem and action sequence Under these conditions, select patients The probability, This represents the node index, which corresponds to the patient index; This method has a computation time in the millisecond range, making it suitable for scenarios with high real-time requirements.

[0051] When using a sampling-decoding output service path scheme, for the first... VRP subproblems The policy network outputs the policy. Generate the lowest cost solution among several solutions. This final solution forms the service path scheme; where... To output policies based on policy networks The solution obtained from sampling Indicates the index of the generated solution. This indicates the total travel distance to visit all patients.

[0052] Based on the above policy network, and according to the determined service order, the corresponding VRP subproblems are solved sequentially for each type of service; for each service type... The problem involves grouping all patients requiring the service into sub-problems and planning access routes for the healthcare team providing the service. The route construction employs a patient-by-patient selection method, choosing one patient from those not currently receiving services to join the route each time, until all patients have been served.

[0053] In step S400, after solving the current VRP subproblem, the start and end times of the service at each patient are calculated sequentially according to the generated service path scheme through a dynamic time window constraint mechanism, thereby determining the service timetable of the corresponding service path scheme. In the service schedule, for a set of patients requiring dual services, in the patient Complete the service Time window Update according to the preset priority, the update formula is: In the formula, These respectively represent the patients The start and end points of the service time window. Indicates service In patients Maintenance window Indicates first service In patients The planned start time at the location, Indicates the patient The specified time difference Indicates the patient First service The earliest start time, Indicates the patient First service The latest start time; where the preset priority is and , This indicates it's the first time providing service. This indicates the second service. or Second service The time window is based on the planned start time of the preceding services. as well as and The value is adjusted.

[0054] Specifically, when In this case, the start time of subsequent services must be precisely matched with the planned start time of preceding services to achieve synchronized service.

[0055] In the embodiment, when the patient is dynamically updated, a rolling optimization strategy is adopted to update the planning and scheduling scheme containing all patient home medical care paths, including: fixing the service path and service schedule that have been executed before the current time, i.e., all start times of services remain unchanged; merging the current unserved patient set and the updated patient set to form a new patient set , and updating the time window of each patient service to ensure that the earliest service time is not earlier than the current time, which is expressed as: re-solve each VRP sub-problem in the updated patient set by using the trained policy network to obtain the corresponding service path scheme and service schedule and execute them until the next patient update; Specifically, the trained policy network is applied to re-solve the updated problem instance to generate a new service path scheme; the global understanding of the current state by the encoder and the sequential decision-making ability of the decoder are utilized to quickly generate a feasible scheme and execute it until the next batch of patients arrives or needs to be re-planned.

[0056] The rolling optimization strategy provided in the embodiment has the advantages that the generalization ability of the trained policy network is fully utilized, re-training is not required, the continuity and executability of the scheme are guaranteed by fixing the executed part, and the millisecond-level response time meets the real-time decision-making requirement.

[0057] The principles and implementation manners of the present application are described in the embodiments, and the above embodiment descriptions are only used to help understand the method and core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed, and the above description should not be understood as limiting the present application.

[0058] Those skilled in the art will realize that the embodiments described herein are for the purpose of helping the reader understand the principles of the present application and should be understood as not limiting the protection scope of the present application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspirations disclosed in the present application without departing from the essence of the present application, and these modifications and combinations still fall within the protection scope of the present application.

Claims

1. A home medical care path planning and dispatching method based on deep reinforcement learning, characterized in that, The method comprises the following steps: S100, obtaining instance data of a home health care routing and scheduling problem (HHCRSP) and modeling the instance data into a mixed integer linear programming model; S200, decomposing and sorting the mixed integer linear programming model to obtain a plurality of VRP sub-problems based on service type and service dependency; S300, modeling the solving process of each VRP sub-problem as a Markov decision process; S400, constructing and training an encoder-decoder neural network based on an attention mechanism as a policy network, and then solving the constructed Markov decision process according to the solving order to obtain the service path scheme and service timetable corresponding to each VRP sub-problem; S500, after all the VRP sub-problems corresponding to the service types are solved, integrating the service path scheme and service timetable obtained by solving the VRP sub-problems to form a planning and scheduling scheme including the home health care paths of all patients. 2.The home medical care path planning and dispatching method based on deep reinforcement learning according to claim 1, characterized in that, In the step S100, the objective function of the mixed integer linear programming model is to minimize the total travel distance of all medical staff, which is expressed as: wherein denotes the patient and the distance between the patient, denotes the medical staff driving to the patient from the patient the decision variable, if yes , otherwise, , denotes the set of all locations, denotes the set of medical staff, denotes the set of all patients; The constraint conditions of the mixed integer linear programming model include: In the formula, Indicates medical staff From the central office Drive to the patient Decision variables, Indicates medical staff From the patient Drive to the central office Decision variables, Indicates medical staff In patients The office provides services The decision variables, if providing services, are... ,otherwise , Indicates service The number of medical staff required Indicates the patient The decision variables that need to be served, if necessary ,otherwise , Indicates medical staff Capable of providing services The decision variable is the qualification; if one possesses the qualification... ,otherwise , Indicates medical staff In patients The office provides services The start time, Indicates the patient The earliest service start time, Indicates the patient The office provides services Duration, Indicates the patient The latest service start time, This indicates the large constants used in the constraints. Indicates medical staff In patients The office provides services The start time, Indicates medical staff In patients The office provides services The start time, Indicates medical staff In patients The office provides services The start time, Indicates medical staff In patients The office provides services End time, denotes a healthcare worker at a patient provides service end time, denotes a location minimum time interval between two services, denotes a location maximum time interval between two services, denotes a set of double-service patients, denotes a set of patients requiring simultaneous service, denotes a set of patients requiring priority order service, denotes a set of service types, subscript denotes a different service index, subscript denotes a different location index, subscript denotes a different healthcare worker index. 3.The home medical care path planning and dispatching method based on deep reinforcement learning according to claim 1, characterized in that, The step S200 comprises the following sub-steps: S201, dividing the patient set into different groups according to the service types required by the patients, including a single-service patient set and a double-service patient set; S202, dividing the double-service patient set according to the service dependency relationship to obtain a synchronous double-service patient set and a sequential double-service patient set; S203, sorting all service types and ensuring that the index of the first service is always less than the index of the second service for double-service patients; S204, according to the determined service order, sequentially grouping all patients requiring the same type of service into a VRP sub-problem corresponding to the type of service. 4.The home medical care path planning and dispatching method based on deep reinforcement learning according to claim 1, characterized in that, In the step S300, in the first VRP sub-problem corresponding Markov decision process: States in state space comprising static information and dynamic information; the static information including geographical coordinates of the patient and the central office, an initial time window and a service duration; the dynamic information including a current last visited patient location, a set of visited patients and a cumulative travel time; Actions in the action space To select the next patient to be served from the current set of unvisited patients, or to select to return to the central office; the set of actions in the action space is the set of nodes , 0 represents the central office, the rest represent patient nodes; State transitions for performing actions After, the state is transitioned from to , including updating the current location to the selected patient location, adding the patient to the visited set, and updating the cumulative travel time; The reward function is the negative value of the total distance traveled after all patients are served. 5.The home medical care path planning and dispatching method based on deep reinforcement learning according to claim 1, characterized in that, In the step S400, the encoder in the policy network comprises an initial embedding generation unit and a multi-layer attention update unit. The initial embedding generation unit is used for linear transformation of the initial node features and time window features of each patient, and the linear projection results are stacked to form initial node embeddings. The multi-layer attention update unit is used for updating the initial node embeddings through multiple attention layers. In each attention layer, the node embeddings are updated through multi-head attention mechanism and feedforward mechanism, and the final node embeddings are generated by combining residual connection and batch normalization and input to the decoder. 6.The home medical care path planning and dispatching method based on deep reinforcement learning according to claim 5, characterized in that, In the step S400, the decoder in the policy network comprises a context embedding construction unit, a constraint mask mechanism and an action selection unit. The context embedding construction unit is used to construct a context embedding reflecting the dynamic information in the current solution by combining the final node embeddings output by the encoder, the current service type embedding, the next access node embedding and the service completion time at the current solution node. The constraint mask mechanism is used to select the next access node according to the dynamic constraint mask; wherein the dynamic constraint mask includes a current patient constraint mask and a central office constraint mask. The action selection unit is configured to calculate attention scores of context embeddings corresponding to each node and apply a constraint mask, and then select an action. 7.The home medical care path planning and dispatching method based on deep reinforcement learning according to claim 4, characterized in that, In the step S400, during the training of the policy network: For the first VRP sub-problem corresponding policy network, to minimize the expected cost of the policy network policy generating a solution, the expected cost is the training target, and the gradient descent method is used to optimize the policy network parameters; where the training loss function is: wherein denotes the strategy is a VRP sub-problem generated service path total length of the denotes the expected value of the cost under the strategy solution Policy network parameters The optimization formula is: wherein, denotes a gradient, denotes that the baseline model is a VRP subproblem the cost obtained, denotes the derivative of the log policy probability with respect to the parameters, denotes the VRP subproblem index, denotes the HHCRSP instance, denotes the patient corresponding instance data, denotes the number of instances in the training batch.

8. The deep reinforcement learning based home healthcare pathway planning and dispatching method according to claim 4, characterized in that, In the step S400, when the constructed Markov decision process is solved by using the trained policy network, a service path scheme corresponding to each VRP sub-problem is output by using a greedy decoding or a sampling decoding; When the service path scheme is output based on the greedy decoding, for the first VRP sub-problem , the selection action at the time step in the service path scheme is the node with the highest selection probability in the policy output by the policy network, which is represented as: wherein denotes the sequence of actions generated by the policy network from time 0 to time t-1, denotes the probability of selecting a patient and action sequence given the sub-problem and the sequence of actions denotes the node index, which corresponds to the patient index; When the service path scheme is decoded based on sampling, for the first VRP sub-problem the policy network outputs a policy The lowest cost solution among several solutions is generated as the final generated solution, thereby forming a service path scheme; wherein, the policy network outputs a policy according to the solution obtained by sampling, indicates the index of the generated solution, indicates the total travel distance for visiting all patients. 9.The home medical care path planning and dispatching method based on deep reinforcement learning according to claim 1, wherein, In the step S400, after the solution of the current VRP sub-problem is completed, the start time and the end time of service at each patient are calculated in sequence according to the generated service path scheme by using a dynamically updated time window constraint mechanism, and then a service schedule corresponding to the service path scheme is determined; In the service schedule, for a double-service patient set, the time window in which the patient is served is updated according to a pre-set priority update, with the formula: ​ In the formula, respectively represent the start and end points of the time window at the time of service to the patient, represent the service maintained at the patient , represent the planned start time of the first service at the patient , represent the specified time difference of the patient , represent the earliest start time of the first service of the patient , represent the latest start time of the first service of the patient ; wherein the preset priority is and , represent the first service, represent the second service, the service or ; the time window of the second service is adjusted according to the planned start time of the previous service and the value of . 10.The home medical care path planning and dispatching method based on deep reinforcement learning according to claim 1, wherein, When the patients are dynamically updated, a rolling optimization strategy is used to update the planning and scheduling scheme of the home medical care path of all patients, including: Fixing the service path and the service schedule of the service performed before the current time; Merging the current unserved patient set and the updated patient set to form a new patient set, and updating the time window of service for each patient; Using the trained policy network to re-solve each VRP sub-problem in the updated patient set, obtaining the corresponding service path scheme and service schedule and executing them until the next patient update.

Images