A Multi-Mode Queue Cooperative Processing Method Based on Reinforcement Learning

Abstract

This invention discloses a multi-mode queue collaborative processing method based on reinforcement learning, comprising: S1, collecting multi-source heterogeneous data from a physical examination center; S2, performing spatiotemporal alignment and feature extraction on the data to construct a high-dimensional state feature vector; S3, constructing a high-fidelity digital twin simulation environment, initializing an agent based on an improved H-PPO algorithm, and defining a heterogeneous topology; S4, constructing a multi-objective weighted fusion reward function and introducing a cross-modal action consistency constraint factor to guide the agent in optimization; S5, outputting parameterized mixed actions through an Actor network and performing legality verification; S6, implementing differentiated collaborative scheduling strategies for different queue modes; and S7, updating network parameters and completing deployment through an online fine-tuning mechanism. This invention achieves dynamic resource allocation and closed-loop process optimization in the complex environment of a physical examination center, effectively shortening waiting time and improving equipment utilization.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical management and operations optimization, and in particular to a multi-mode queue collaborative processing method based on reinforcement learning. Background Technology

[0002] Reinforcement learning, with its self-learning and strategy optimization capabilities in complex dynamic decision-making environments, has been widely applied in recent years in fields such as intelligent manufacturing scheduling, traffic signal control, and medical resource allocation, becoming an important development direction for intelligent collaborative processing. However, in practical applications, the medical examination center scenario faces many challenges, such as heterogeneous queue patterns, complex resource constraints, and strict process dependencies, and the deployment effectiveness of reinforcement learning is still constrained by many factors.

[0003] Most current scheduling methods rely on single rule-based algorithms or static heuristics, making it difficult to fully utilize multi-source heterogeneous information such as the number of people waiting in each queue, the busy / idle status of medical staff, and the progress of patients undergoing physical examinations. This results in a lack of comprehensive modeling of the system's global state. Some systems only use a single type of action space to output scheduling instructions, ignoring the strong coupling between queue type selection and resource pre-allocation parameters. This makes it difficult to finely guide resource allocation through the coordination of discrete and continuous actions, limiting the adaptive optimization capability of scheduling strategies in multi-mode queues. Furthermore, the scheduling decision-making process lacks a real-time verification mechanism for conflicts between medical dependency sequences and physical paths, making it difficult to provide managers with a safe and reliable basis for decision-making, thus affecting the credibility and usability of scheduling results.

[0004] Furthermore, existing reinforcement learning algorithms have relatively simple reward mechanisms in scheduling tasks. They have failed to construct a multi-objective weighted fusion mechanism that includes process completion, waiting time savings, and resource idle status. They have also failed to introduce cross-modal action consistency constraint factors to penalize the semantic distance between discrete decision features and continuous parameter features. This makes it difficult for the agent to find a Pareto optimal solution between efficiency, fairness, and resource utilization, which seriously restricts the practical value and stability of the model in real physical examination environments.

[0005] Therefore, how to provide a multi-mode queue collaborative processing method based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a multi-modal queue collaborative processing method based on reinforcement learning. This invention fully integrates key steps such as multi-source heterogeneous data feature extraction, high-fidelity digital twin simulation, improved H-PPO algorithm decision-making, and multi-objective weighted reward optimization. It constructs an intelligent collaborative processing flow with differentiated scheduling of heterogeneous queues, parameterized hybrid action legality verification, cross-modal decision consistency constraints, and online fine-tuning mechanisms. This enables comprehensive perception of queue status, dynamic resource allocation, and continuous strategy optimization in complex physical examination environments. This invention possesses advantages such as comprehensive state feature representation, accurate hybrid action decision-making, high resource collaboration efficiency, and fast training convergence speed, significantly improving the service efficiency and resource utilization of physical examination centers. In particular, by introducing a cross-modal action consistency constraint factor, it effectively solves the semantic conflict between discrete and continuous action components in decision-making. Combined with the improved H-PPO algorithm, it achieves proximal optimization iteration of the policy gradient, thereby effectively addressing problems such as poor adaptability of scheduling strategies, fragmented multi-modal action decision-making, and uneven resource allocation in existing methods.

[0007] A multi-mode queue collaborative processing method based on reinforcement learning according to an embodiment of the present invention includes the following steps: S1. Real-time collection of multi-source heterogeneous data from the physical examination center; S2. Spatiotemporal alignment and feature extraction are performed on multi-source heterogeneous data. The item dependency relationship of the examinee is extracted to construct a directed acyclic graph encoding. The queue state, department state and system global indicators are integrated to construct a high-dimensional state feature vector and map it to the input state. S3. Construct a high-fidelity digital twin simulation environment for the physical examination center, initialize an intelligent agent based on the improved H-PPO algorithm, and define a heterogeneous topology including ordinary queues, bottleneck queues, independent queues, and parallel queues. S4. Construct a multi-objective weighted fusion reward function mechanism, introduce a cross-modal action consistency constraint factor, calculate the semantic distance between discrete decision features and continuous parameter features in the Actor network, impose penalty constraints on inconsistent decisions between discrete and continuous actions, and determine positive reward terms and negative penalty terms to guide the agent to seek Pareto optimal solutions between efficiency, fairness and resource utilization. S5. Based on the current input state, output parameterized hybrid actions through the Actor network, and perform legality verification through preset resource mutual exclusion and process constraint rules to eliminate unacceptable actions and output the verified parameterized hybrid actions. S6. The agent executes the parameterized hybrid action after verification, implements differentiated collaborative scheduling strategies for different queue modes, performs peak-valley regulation for ordinary queues, implements logical virtual occupancy for bottleneck queues and combines it with the FCFS algorithm for dynamic scheduling, strictly guarantees the FCFS algorithm for independent queues and combines it with physical virtual occupancy for intermittent service, and implements dynamic snake-shaped queuing for parallel queues based on comprehensive service time and health check type. S7. After the agent executes the parameterized hybrid action after verification, it obtains the immediate reward and the state of the next moment and stores them in the experience replay buffer. Based on the near-end policy, it optimizes the target calculation gradient to update the parameters of the Actor network and Critic network. Through the online fine-tuning mechanism, it adapts to the dynamic changes of the environment and is deployed to the actual scheduling system to complete the collaborative processing.

[0008] Optionally, S1 specifically includes: S11. By deploying sensor groups at the entrances, triage desks and examination rooms of each department in the physical examination center, the real-time number of people waiting in each queue and the estimated waiting time are collected at a preset frequency. The busy and idle status of medical staff and equipment in the department is read in real time through the HIS system interface, and the current service speed is calculated based on historical service records. S12. Obtain the individual attributes of the examinee through an ID card reader or barcode scanner, determine the examinee's progress by querying the list of physical examination package items, summarize the queue length data and equipment utilization data of each department, and calculate the global load index of the system. S13. Align the real-time number of people waiting in each queue with the estimated waiting time, the busy / idle status of medical staff and equipment in the department and the service speed, the individual attributes and process progress of the examinees, and the global load index of the system according to a unified timestamp, and output multi-source heterogeneous data.

[0009] Optionally, S2 specifically includes: S21. Read the timestamps carried by all data in the multi-source heterogeneous data, unify the timestamps of all data to the preset standard time, classify the data into the corresponding spatial regions according to the physical location coordinates of the department, and complete the spatiotemporal alignment. S22. Read the list of physical examination items of the examinee, define each item as a node, connect the nodes with dependencies according to the order of medical examinations, and generate a directed acyclic graph encoding. S23. Calculate the statistical average of the real-time number of people waiting and the estimated waiting time for each queue, generate queue status characteristics, and calculate the average of the busy / idle status duration and service speed of medical staff and equipment in the department, and generate department status characteristics. S24. Perform numerical normalization calculation on the global load index of the system to obtain the global index of the system. Extract the adjacency matrix of the directed acyclic graph encoding and input it into the preset fully connected layer for vector transformation to generate a graph embedding vector. Concatenate the queue state features, department state features, global index of the system and the graph embedding vector in a preset order to generate a high-dimensional state feature vector. S25. Perform a linear transformation operation on the high-dimensional state feature vector to map the numerical range of the high-dimensional state feature vector to between 0 and 1, and output the input state.

[0010] Optionally, S3 specifically includes: S31. Read the building floor plan data of the physical examination center, map each department to a service node in the simulation environment, calculate the average service time and variance of each department based on historical physical examination records, construct a service time generation function that conforms to a normal distribution, and build a high-fidelity digital twin simulation environment for the physical examination center. S32. Initialize the agent based on the improved H-PPO algorithm, construct the Actor network and Critic network containing the input layer, hidden layer and output layer, and divide the output of the Actor network into two branches: discrete action head and continuous action head. S33. Calculate the average service time and average arrival rate of each department within the preset historical time window, calculate the theoretical maximum throughput of the department, use the ratio of average waiting time to average service time as the load intensity factor, and combine it with the ratio of actual throughput to theoretical maximum throughput to perform a weighted sum with preset weights to obtain the load ratio of the service capacity of each department. Define the queue corresponding to the department with a load ratio greater than the first preset threshold as the bottleneck queue, and define the queue corresponding to the department with no project dependency relationship as the independent queue. S34. Departments whose statistical service capabilities meet the preset standards and have no resource conflicts will have their corresponding queues defined as ordinary queues. Departments with multiple parallel service windows will have their corresponding queues defined as parallel queues. S35. The bottleneck queue, independent queue, ordinary queue and parallel queue are respectively used as nodes. The nodes are connected according to the physical path of the physical examination process, and the resource mutual exclusion passage rules are preset for each connection path to construct a heterogeneous topology structure including ordinary queue, bottleneck queue, independent queue and parallel queue.

[0011] Optionally, S4 specifically includes: S41. Calculate the number of items completed by all examinees at the current moment, divide it by the total number of items for examinees to obtain the completion rate, calculate the arithmetic mean of the completion rates of all examinees, multiply the arithmetic mean by the first preset weighting coefficient, and calculate the process completion reward item. S42. Read the estimated waiting time of each queue at the current moment, subtract the estimated waiting time of each queue at the previous moment to obtain the change in waiting time, sum up the changes in waiting time of all queues and take the negative value, multiply the negative result by the second preset weight coefficient, and calculate the waiting time saving reward item. S43. Count the number of medical personnel and equipment that are currently idle, divide the number by the total number of medical personnel and equipment to obtain the resource idle rate, multiply the resource idle rate by the third preset weight coefficient, and calculate the resource idle penalty item. S44. Read the output matrix from the last hidden layer of the Actor network, multiply the output matrix by the preset weight matrix corresponding to the discrete action head and add the bias vector to obtain the logits numerical vector of the discrete action. Input the logits numerical vector into the Softmax function for exponential normalization operation, calculate the probability value corresponding to each queue type, and output the class probability distribution. S45. Read the same output matrix from the last hidden layer of the Actor network, multiply the output matrix by the preset first set of weight parameters corresponding to the continuous action head, calculate the mean vector of the Gaussian distribution, multiply the output matrix by the preset second set of weight parameters corresponding to the continuous action head, calculate the variance vector of the Gaussian distribution, and take the logarithm of the variance vector to ensure numerical stability, and output the Gaussian distribution parameters. S46. Read the category probability distribution output by the discrete action head, expand the dimension to the same dimension as the Gaussian distribution parameters, and denote it as the first distribution component. Read the mean vector and variance vector of the Gaussian distribution output by the continuous action head, and concatenate them column by column to generate a continuous parameter distribution feature, which is denoteed as the second distribution component. Perform vector concatenation operation on the first distribution component and the second distribution component according to the preset dimension order to generate a joint feature vector, and output it as a mixed action probability distribution. S47. Read the output values ​​of the discrete action head fully connected layer of the Actor network before the activation function, and arrange them into a one-dimensional row vector as the discrete decision feature vector. Read the output values ​​of the continuous action head fully connected layer of the Actor network before the activation function, and arrange them into a one-dimensional row vector as the continuous parameter feature vector. S48. Calculate the dot product of the discrete decision feature vector and the continuous parameter feature vector and their respective L2 norm products. Divide the dot product by the L2 norm product to obtain the cosine similarity. Subtract the cosine similarity from the value 1 to obtain the semantic distance. Multiply the semantic distance by the preset consistency penalty coefficient to calculate the cross-modal action consistency constraint factor. When the discrete action and the continuous action decision are inconsistent, it is used as the penalty value. If they are consistent, the penalty value is set to zero. S49. The process completion reward item, waiting time saving reward item, resource idle penalty item and penalty value are weighted and summed according to preset weights to calculate the instant reward value, and the advantage function estimate is calculated in combination with the time difference error. S410. Read the instant reward value and the input state at the next moment from the experience playback buffer. Add the instant reward value to the product of the Critic network's value estimate of the input state at the next moment multiplied by a preset discount factor. Subtract the Critic network's value estimate of the input state at the current moment to calculate the time difference error. Subtract the corresponding mean from the time difference error and divide by the standard deviation to calculate the dominance function. S411. Read the mixed action probability distribution generated in step S46, calculate the probability ratio of the current policy and the old policy on the corresponding actions, introduce a preset truncation coefficient, determine whether the probability ratio exceeds the preset interval, if it does, replace the probability ratio with the boundary value of the preset interval to obtain the truncated probability ratio, multiply the advantage function by the truncated probability ratio to construct the objective function, calculate the gradient based on the objective function, and backpropagate to update the parameters of the Actor network and the Critic network.

[0012] Optionally, S5 specifically includes: S51. Read the current input state and input it into the hidden layer of the Actor network for forward propagation calculation. Extract state features through linear transformation and nonlinear activation function, and output the intermediate state feature matrix. S52. Input the intermediate state feature matrix into the discrete action head, multiply it by the preset weight matrix corresponding to the discrete action head and add the bias vector. After normalization by the Softmax function, output the probability value of selecting each queue type and take the queue type with the highest probability value as the discrete action component. S53. Input the intermediate state feature matrix into the continuous action head, multiply it by the preset first set of weight parameters corresponding to the continuous action head to obtain the mean vector of the Gaussian distribution, multiply it by the preset second set of weight parameters corresponding to the continuous action head to obtain the variance vector of the Gaussian distribution, randomly sample a noise vector from the standard normal distribution, multiply the mean vector and the noise vector element by element and then linearly combine them with the variance vector to generate the specific value of resource pre-allocation as the continuous action component. S54. Combine discrete action components with continuous action components to generate parameterized hybrid actions, read preset resource mutual exclusion and process constraint rules, check whether the queue type determined by the discrete action components has a physical path conflict with the unfinished items of the current examinee, check whether the specific value of the pre-allocated resources determined by the continuous action components exceeds the range of the number of medical personnel and equipment currently available in the corresponding department, and at the same time verify whether the medical dependency order of the physical examination items is met. S57. If the parameterized mixed action passes all checks, it is determined to be an actionable action and is retained. If any conflict exists, it is determined to be an actionable action, which is then removed from the action space. The discrete action components are forcibly replaced with the default queue type that conforms to the process constraints, and the verified parameterized mixed action is output.

[0013] Optionally, S6 specifically includes: S61. Read the parameterized mixed action after verification, parse the discrete action components to obtain the type label of the target queue, and parse the continuous action components to obtain the specific value of resource pre-allocation. S62. When the target queue type label is ordinary queue, calculate the current system global load index value. If the value is greater than the preset high load threshold, calculate the reciprocal of the number of people waiting in real time in each ordinary queue branch as the selection probability, and randomly select the branch with fewer people for diversion according to the probability. If the value is less than the preset low load threshold, directly add the examinee to the end of the queue to complete peak and valley control. S63. When the target queue type label is bottleneck queue, read the current waiting sequence of the bottleneck queue, calculate the service duration requirement of the logical virtual placeholder based on the specific value of resource pre-allocation multiplied by the average service duration of the bottleneck queue, generate a logical virtual placeholder mark carrying the service duration requirement, traverse all existing placeholder marks in the bottleneck queue, and calculate the time gap width between two adjacent placeholder marks. S64. Compare the service duration requirement of the logical virtual placeholder with the width of each time slot one by one, filter out the smallest time slot that is greater than the service duration requirement, insert the logical virtual placeholder into the starting position of the smallest time slot, update the expected start time of all subsequent examinees in the queue sequence, and for other examinees who have not inserted the logical virtual placeholder, read the arrival timestamp and rearrange the service order according to the order of the timestamps using the FCFS algorithm. S65. When the target queue's type label is independent queue, read the start time and average service duration of the patients currently receiving services in the queue, add the two together to get the expected end time, and subtract the current system time from the expected end time to get the remaining service time. S66. Compare the remaining service time with the preset gap threshold. If the remaining service time is less than the gap threshold, it is determined that the current service is about to end and a service gap is generated. Calculate the service duration of the person to be inserted for physical examination based on the specific value of the pre-allocated resources. If the service duration is less than or equal to the gap corresponding to the remaining service time, a physical virtual placeholder is generated. S67. Map the physical virtual placeholder to the specific identity information of the examinee, insert it into the time node after the current service ends, update the queuing number of the subsequent examinees, and for regular examinees who are not inserted, strictly arrange the service order according to the arrival timestamp to complete the intermittent service. S68. When the target queue type label is parallel queue, read the current queue length and average service time of each parallel service window, calculate the expected waiting time of each window, select the window with the smallest expected waiting time as the target window, read the physical examination type of the examinee, and assign examinees of the same physical examination type to different parallel windows in an S-shaped order to complete the dynamic snake-shaped queue.

[0014] Optionally, S7 specifically includes: S71. After the agent executes the verified parameterized hybrid action, it reads the instant reward value calculated in step S49 and obtains the input state of the next moment from the simulation environment. It combines the instant reward value, the current input state, the verified parameterized hybrid action, and the input state of the next moment into an experience data unit and stores it in the experience playback buffer. S72. Monitor the amount of data stored in the experience playback buffer. When the amount of data stored reaches the preset batch size, randomly select a set of experience data units, read the instant reward value and status information, call the method described in step S410 to calculate the time difference error and advantage function, and update the parameters of the Actor network and Critic network to complete one proximal optimization iteration. S73. Determine whether the current training round has reached the preset convergence condition. If not, continue to collect data and execute steps S1 to S72. Otherwise, determine that the agent training is complete. After a preset online fine-tuning period, read multi-source heterogeneous data from the real environment to construct the real-time input state. Input the trained agent to obtain actions and execute them. Based on the reward value of the real feedback, fine-tune and update the network parameters according to the method of step S72. S74. The weight matrices and bias vectors of the finally updated Actor network and Critic network are stored in the control terminal of the actual scheduling system to complete the collaborative processing of agent deployment and health check cycle.

[0015] The beneficial effects of this invention are: This invention addresses the challenges of heterogeneous queue patterns, complex resource constraints, and strict process dependencies in health checkup centers by constructing a high-fidelity digital twin simulation environment and a multi-source heterogeneous data feature extraction mechanism. It employs directed acyclic graph (DAG) encoding to represent item dependencies and combines spatiotemporal alignment and feature fusion operations to construct a high-dimensional state feature vector, generating the input state. An agent based on an improved H-PPO algorithm is initialized, and parameterized hybrid actions containing discrete and continuous action components are output via the Actor network. These actions are then validated using preset resource mutual exclusion and process constraint rules, resulting in the validated parameterized hybrid actions. During the strategy optimization phase, a cross-modal action consistency constraint factor is introduced to calculate the semantic distance between discrete decision features and continuous parameter features. Combined with process completion, waiting time savings, and resource idle status, an instant reward value is calculated, guiding the agent to seek a Pareto optimal solution between efficiency, fairness, and resource utilization. Furthermore, an online fine-tuning mechanism dynamically updates the parameters of the Actor and Critic networks, and differentiated collaborative scheduling strategies for ordinary queues, bottleneck queues, independent queues, and parallel queues are used to complete logical virtual occupancy and interleaved services. Ultimately, it enables intelligent collaborative processing, dynamic resource allocation, and closed-loop optimization of multi-mode queues in the health checkup center, effectively improving the comprehensiveness of state perception, the consistency of mixed action decision-making, and the adaptive capability of the scheduling system in a multi-source heterogeneous data environment. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a multi-mode queue collaborative processing method based on reinforcement learning proposed in this invention; Figure 2 This is a flowchart of the hybrid action generation and multi-mode queue differential scheduling based on the improved H-PPO algorithm proposed in this invention. Figure 3 This is a flowchart of the reward function calculation and cross-modal action consistency constraint based on multi-objective weighted fusion proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figures 1-3 A multi-mode queue collaborative processing method based on reinforcement learning includes the following steps: S1. Real-time collection of multi-source heterogeneous data from the physical examination center, including the real-time number of people waiting in each queue and the estimated waiting time, the busy / idle status of medical staff and equipment in the department and the service speed, the individual attributes and process progress of the examinees, and the global load index of the system. S2. Perform spatiotemporal alignment and feature extraction on multi-source heterogeneous data, extract the item dependency relationship of the examinee to construct a directed acyclic graph encoding, integrate queue state, department state and system global indicators, construct a high-dimensional state feature vector containing time and space dimensions, and map it to the input state. S3. Construct a high-fidelity digital twin simulation environment for the physical examination center, initialize an intelligent agent based on the improved H-PPO algorithm, the intelligent agent includes an Actor network and a Critic network, and define a heterogeneous topology structure including ordinary queues, bottleneck queues, independent queues and parallel queues. S4. Construct a multi-objective weighted fusion reward function mechanism, introduce a cross-modal action consistency constraint factor, calculate the semantic distance between discrete decision features and continuous parameter features in the Actor network, impose penalty constraints on inconsistent decisions between discrete and continuous actions, and determine positive reward terms and negative penalty terms by combining process completion degree, waiting time savings and resource idle state, so as to guide the agent to seek Pareto optimal solution between efficiency, fairness and resource utilization. S5. Based on the current input state, output parameterized hybrid actions through the Actor network, and perform legality verification through preset resource mutual exclusion and process constraint rules to eliminate unacceptable actions and output the verified parameterized hybrid actions. S6. The agent executes the parameterized hybrid action after verification, implements differentiated collaborative scheduling strategies for different queue modes, performs peak-valley regulation for ordinary queues, implements logical virtual occupancy for bottleneck queues and combines it with the FCFS algorithm for dynamic scheduling, strictly guarantees the FCFS algorithm for independent queues and combines it with physical virtual occupancy for intermittent service, and implements dynamic snake-shaped queuing for parallel queues based on comprehensive service time and health check type. S7. After the agent executes the parameterized hybrid action after verification, it obtains the immediate reward and the state of the next moment and stores them in the experience replay buffer. Based on the near-end policy, it optimizes the target calculation gradient to update the parameters of the Actor network and Critic network. Through the online fine-tuning mechanism, it adapts to the dynamic changes of the environment and is deployed to the actual scheduling system to complete the collaborative processing.

[0019] This invention significantly improves the overall operational efficiency and resource utilization of health checkup centers. By constructing a high-fidelity digital twin simulation environment and using directed acyclic graph encoding, it achieves accurate modeling of the complex dependencies and dynamic queue states in the health checkup process, effectively solving the problem that traditional static rules are difficult to adapt to dynamic changes in passenger flow. Utilizing an improved H-PPO algorithm to drive the agent to output parameterized hybrid actions, it can implement differentiated collaborative scheduling strategies for ordinary, bottleneck, independent, and parallel queues, such as logical virtual occupancy of bottleneck equipment and dynamic serpentine queuing for parallel queues, significantly shortening the average waiting time and dwell time of examinees and effectively alleviating morning peak congestion. Introducing a cross-modal action consistency constraint factor ensures the semantic unity of discrete decisions and continuous parameters, avoiding ineffective scheduling and greatly improving the stability and security of the scheduling system. Through a multi-objective weighted fusion reward mechanism, it guides the system to seek a Pareto optimal solution among efficiency, fairness, and resource utilization, resulting in a significant increase in equipment utilization and completely eliminating scheduling conflicts. In addition, the online fine-tuning mechanism ensures the continuous evolution capability of the intelligent agent in the real environment, significantly improves the response speed and handling capability of the health check center to emergencies, and realizes intelligent and refined full-process collaborative management.

[0020] In this embodiment, S1 specifically includes: S11. By deploying sensor groups at the entrances, triage desks and examination rooms of each department in the physical examination center, the real-time number of people waiting in each queue and the estimated waiting time are collected at a preset frequency. The busy and idle status of medical staff and equipment in the department is read in real time through the HIS system interface, and the current service speed is calculated based on historical service records. S12. Obtain the individual attributes of the examinee through an ID card reader or barcode scanner, determine the examinee's progress by querying the list of physical examination package items, summarize the queue length data and equipment utilization data of each department, and calculate the global load index of the system. S13. Align the real-time number of people waiting in each queue with the estimated waiting time, the busy / idle status of medical staff and equipment in the department and the service speed, the individual attributes and process progress of the examinees, and the global load index of the system according to a unified timestamp, and output multi-source heterogeneous data.

[0021] In this embodiment, S2 specifically includes: S21. Read the timestamps carried by all data in the multi-source heterogeneous data, unify the timestamps of all data to the preset standard time, classify the data into the corresponding spatial regions according to the physical location coordinates of the department, and complete the spatiotemporal alignment. S22. Read the list of physical examination items for the examinee, define each item as a node, connect the nodes with dependencies according to the order of the medical examinations, and generate a directed acyclic graph encoding of the item dependencies. S23. Calculate the statistical average of the real-time number of people waiting and the estimated waiting time for each queue, generate queue status characteristics, and calculate the average of the busy / idle status duration and service speed of medical staff and equipment in the department, and generate department status characteristics. S24. Perform numerical normalization calculation on the global load index of the system to obtain the global index of the system. Extract the adjacency matrix of the directed acyclic graph encoding and input it into the preset fully connected layer for vector transformation to generate a graph embedding vector. Concatenate the queue state features, department state features, global index of the system and the graph embedding vector in a preset order to generate a high-dimensional state feature vector containing time and space dimensions. S25. Perform a linear transformation operation on the high-dimensional state feature vector to map the numerical range of the high-dimensional state feature vector to between 0 and 1, and output the input state of the agent.

[0022] In this embodiment, S3 specifically includes: S31. Read the building floor plan data of the physical examination center, map each department to a service node in the simulation environment, calculate the average service time and variance of each department based on historical physical examination records, construct a service time generation function that conforms to a normal distribution, and build a high-fidelity digital twin simulation environment for the physical examination center. S32. Initialize the agent based on the improved H-PPO algorithm, construct the Actor network and Critic network containing the input layer, hidden layer and output layer, and divide the output of the Actor network into two branches: discrete action head and continuous action head. S33. Calculate the average service time and average arrival rate of each department within the preset historical time window, calculate the theoretical maximum throughput of the department, use the ratio of average waiting time to average service time as the load intensity factor, and combine it with the ratio of actual throughput to theoretical maximum throughput to perform a weighted sum with preset weights to obtain the load ratio of the service capacity of each department. Define the queue corresponding to the department with a load ratio greater than the first preset threshold as the bottleneck queue, and define the queue corresponding to the department with no project dependency relationship as the independent queue. S34. Departments whose statistical service capabilities meet the preset standards and have no resource conflicts will have their corresponding queues defined as ordinary queues. Departments with multiple parallel service windows will have their corresponding queues defined as parallel queues. S35. The bottleneck queue, independent queue, ordinary queue and parallel queue are respectively used as nodes. The nodes are connected according to the physical path of the physical examination process, and the resource mutual exclusion passage rules are preset for each connection path to construct a heterogeneous topology structure including ordinary queue, bottleneck queue, independent queue and parallel queue.

[0023] In this embodiment, S4 specifically includes: S41. Calculate the number of items completed by all examinees at the current moment, divide it by the total number of items for examinees to obtain the completion rate, calculate the arithmetic mean of the completion rates of all examinees, multiply the arithmetic mean by the first preset weighting coefficient, and calculate the process completion reward item. S42. Read the estimated waiting time of each queue at the current moment, subtract the estimated waiting time of each queue at the previous moment to obtain the change in waiting time, sum up the changes in waiting time of all queues and take the negative value, multiply the negative result by the second preset weight coefficient, and calculate the waiting time saving reward item. S43. Count the number of medical personnel and equipment that are currently idle, divide the number by the total number of medical personnel and equipment to obtain the resource idle rate, multiply the resource idle rate by the third preset weight coefficient, and calculate the resource idle penalty item. S44. Read the output matrix from the last hidden layer of the Actor network, multiply the output matrix by the preset weight matrix corresponding to the discrete action head and add the bias vector to obtain the logits numerical vector of the discrete action. Input the logits numerical vector into the Softmax function for exponential normalization operation, calculate the probability value corresponding to each queue type, and output the class probability distribution. S45. Read the same output matrix from the last hidden layer of the Actor network, multiply the output matrix by the preset first set of weight parameters corresponding to the continuous action head, calculate the mean vector of the Gaussian distribution, multiply the output matrix by the preset second set of weight parameters corresponding to the continuous action head, calculate the variance vector of the Gaussian distribution, and take the logarithm of the variance vector to ensure numerical stability, and output the Gaussian distribution parameters. S46. Read the category probability distribution output by the discrete action head, expand the dimension to the same dimension as the Gaussian distribution parameters, and denote it as the first distribution component. Read the mean vector and variance vector of the Gaussian distribution output by the continuous action head, and concatenate them column by column to generate a continuous parameter distribution feature, which is denoteed as the second distribution component. Perform vector concatenation operation on the first distribution component and the second distribution component according to the preset dimension order to generate a joint feature vector that contains both discrete probability values ​​and continuous distribution parameters, and output it as a mixed action probability distribution. S47. Read the output values ​​of the discrete action head fully connected layer of the Actor network before the activation function, and arrange them into a one-dimensional row vector as the discrete decision feature vector. Read the output values ​​of the continuous action head fully connected layer of the Actor network before the activation function, and arrange them into a one-dimensional row vector as the continuous parameter feature vector. S48. Calculate the dot product of the discrete decision feature vector and the continuous parameter feature vector and their respective L2 norm products. Divide the dot product by the L2 norm product to obtain the cosine similarity. Subtract the cosine similarity from the value 1 to obtain the semantic distance. Multiply the semantic distance by the preset consistency penalty coefficient to calculate the cross-modal action consistency constraint factor. When the discrete action and the continuous action decision are inconsistent, it is used as the penalty value. If they are consistent, the penalty value is set to zero. S49. The process completion reward item, waiting time saving reward item, resource idle penalty item and penalty value are weighted and summed according to preset weights to calculate the instant reward value, and the advantage function estimate is calculated in combination with the time difference error. S410. Read the instant reward value and the input state at the next moment from the experience playback buffer. Add the instant reward value to the product of the Critic network's value estimate of the input state at the next moment multiplied by a preset discount factor. Subtract the Critic network's value estimate of the input state at the current moment to calculate the time difference error. Subtract the corresponding mean from the time difference error and divide by the standard deviation to calculate the dominance function. S411. Read the mixed action probability distribution generated in step S46, calculate the probability ratio of the current policy and the old policy on the corresponding actions, introduce a preset truncation coefficient, and determine whether the probability ratio exceeds the preset interval. If it does, replace the probability ratio with the boundary value of the preset interval to obtain the truncated probability ratio. Multiply the advantage function by the truncated probability ratio to construct the objective function, calculate the gradient based on the objective function, and backpropagate to update the parameters of the Actor network and the Critic network to realize the proximal optimization iteration of the H-PPO algorithm.

[0024] This invention achieves dynamic modeling and precise optimization of hybrid action strategies for complex queues in a health checkup center by constructing a high-fidelity digital twin environment and improving the H-PPO algorithm framework. Based on historical data, it accurately identifies bottleneck queues and independent queues and constructs a heterogeneous topology, providing an interactive environment that approximates reality for the agent. Utilizing the dual-branch structure of the Actor network to output discrete category probabilities and continuous parameter distributions, it generates parameterized hybrid actions and constructs a consistency constraint factor by calculating the semantic distance of cross-modal feature vectors, effectively solving the problem of mismatch between discrete decision-making and continuous parameters. Combining multi-objective weighted rewards and a proximal policy optimization mechanism, it truncates probability ratios and calculates an advantage function to guide network parameter updates. This invention maintains stable convergence of the policy gradient in multi-modal queue collaboration scenarios, significantly improving the decision-making accuracy and environmental adaptability of the scheduling system under resource conflicts and passenger flow fluctuations.

[0025] The improved H-PPO algorithm of this invention shares similarities with the original H-PPO algorithm in that both retain the core framework of proximal policy optimization, namely, policy interaction through an Actor-Critic architecture, evaluation of action value using a dominance function, and the introduction of a truncation coefficient to constrain the probability ratio of policy updates, preventing policy collapse due to excessive step size. Both also rely on iterative updates of the value network based on temporal difference errors. The difference lies in that this invention breaks the limitations of the original H-PPO algorithm, which features a relatively independent hybrid action generation mechanism and a lack of deep semantic association. In steps S44 to S46, a joint output structure of parameterized hybrid actions is constructed, rather than simply outputting discrete and continuous distributions side-by-side. More importantly, this invention incorporates a cross-modal action consistency constraint mechanism in steps S47 and S48. This mechanism, as an independent penalty module, can deeply analyze the geometric distance between discrete decision features and continuous parameter features in the latent semantic space within the Actor network.

[0026] The beneficial effects of the improvements lie in the fact that by calculating the cosine similarity between the discrete decision feature vector and the continuous parameter feature vector and constructing a consistency constraint factor, the improved H-PPO algorithm can force discrete actions, such as queue selection, to maintain a high degree of consistency with continuous parameters, such as time, in terms of decision logic. This effectively solves the semantic conflict that may occur between discrete decisions and continuous parameters in the original algorithm, such as the ineffective scheduling problem caused by selecting a queue that does not require waiting but allocating a long service time. This design significantly enhances the agent's ability to model the complex scheduling logic of the physical examination center and can more accurately capture the inherent coupling relationship between mixed actions. The introduction of the consistency constraint factor improves the accuracy of policy decisions while effectively avoiding infeasible or inefficient action space exploration, enhancing the convergence stability and robustness of the system in resource-constrained and high-concurrency physical examination environments.

[0027] In this embodiment, S5 specifically includes: S51. Read the current input state and input it into the hidden layer of the Actor network for forward propagation calculation. Extract state features through linear transformation and nonlinear activation function, and output the intermediate state feature matrix. S52. Input the intermediate state feature matrix into the discrete action head, multiply it by the preset weight matrix corresponding to the discrete action head and add the bias vector. After normalization by the Softmax function, output the probability value of selecting each queue type and take the queue type with the highest probability value as the discrete action component. S53. Input the intermediate state feature matrix into the continuous action head, multiply it by the preset first set of weight parameters corresponding to the continuous action head to obtain the mean vector of the Gaussian distribution, multiply it by the preset second set of weight parameters corresponding to the continuous action head to obtain the variance vector of the Gaussian distribution, randomly sample a noise vector from the standard normal distribution, multiply the mean vector and the noise vector element by element and then linearly combine them with the variance vector to generate the specific value of resource pre-allocation as the continuous action component. S54. Combine discrete action components with continuous action components to generate parameterized hybrid actions, read preset resource mutual exclusion and process constraint rules, check whether the queue type determined by the discrete action components has a physical path conflict with the unfinished items of the current examinee, check whether the specific value of the pre-allocated resources determined by the continuous action components exceeds the range of the number of medical personnel and equipment currently available in the corresponding department, and at the same time verify whether the medical dependency order of the physical examination items is met. S57. If the parameterized mixed action passes all checks, it is determined to be an actionable action and is retained. If any conflict exists, it is determined to be an actionable action, which is then removed from the action space. The discrete action components are forcibly replaced with the default queue type that conforms to the process constraints, and the verified parameterized mixed action is output.

[0028] In this embodiment, S6 specifically includes: S61. Read the parameterized mixed action after verification, parse the discrete action components to obtain the type label of the target queue, and parse the continuous action components to obtain the specific value of resource pre-allocation. S62. When the target queue type label is ordinary queue, calculate the current system global load index value. If the value is greater than the preset high load threshold, calculate the reciprocal of the number of people waiting in real time in each ordinary queue branch as the selection probability, and randomly select the branch with fewer people for diversion according to the probability. If the value is less than the preset low load threshold, directly add the examinee to the end of the queue to complete peak and valley control. S63. When the target queue type label is bottleneck queue, read the current waiting sequence of the bottleneck queue, calculate the service duration requirement of the logical virtual placeholder based on the specific value of resource pre-allocation multiplied by the average service duration of the bottleneck queue, generate a logical virtual placeholder mark carrying the service duration requirement, traverse all existing placeholder marks in the bottleneck queue, and calculate the time gap width between two adjacent placeholder marks. S64. Compare the service duration requirement of the logical virtual placeholder with the width of each time slot one by one, filter out the smallest time slot that is greater than the service duration requirement, insert the logical virtual placeholder into the starting position of the smallest time slot, update the expected start time of all subsequent examinees in the queue sequence, and for other examinees who have not inserted the logical virtual placeholder, read the arrival timestamp and rearrange the service order according to the order of the timestamps using the FCFS algorithm. S65. When the target queue's type label is independent queue, read the start time and average service duration of the patients currently receiving services in the queue, add the two together to get the expected end time, and subtract the current system time from the expected end time to get the remaining service time. S66. Compare the remaining service time with the preset gap threshold. If the remaining service time is less than the gap threshold, it is determined that the current service is about to end and a service gap is generated. Calculate the service duration of the person to be inserted for physical examination based on the specific value of the pre-allocated resources. If the service duration is less than or equal to the gap corresponding to the remaining service time, a physical virtual placeholder is generated. S67. Map the physical virtual placeholder to the specific identity information of the examinee, insert it into the time node after the current service ends, update the queuing number of the subsequent examinees, and for regular examinees who are not inserted, strictly arrange the service order according to the arrival timestamp to complete the intermittent service. S68. When the target queue type label is parallel queue, read the current queue length and average service time of each parallel service window, calculate the expected waiting time of each window, select the window with the smallest expected waiting time as the target window, read the physical examination type of the examinee, and assign examinees of the same physical examination type to different parallel windows in an S-shaped order to complete the dynamic snake-shaped queue.

[0029] In this embodiment, S7 specifically includes: S71. After the agent executes the verified parameterized hybrid action, it reads the instant reward value calculated in step S49 and obtains the input state of the next moment from the simulation environment. It combines the instant reward value, the current input state, the verified parameterized hybrid action, and the input state of the next moment into an experience data unit and stores it in the experience playback buffer. S72. Monitor the amount of data stored in the experience playback buffer. When the amount of data stored reaches the preset batch size, randomly select a set of experience data units, read the instant reward value and status information, call the method described in step S410 to calculate the time difference error and advantage function, and update the parameters of the Actor network and Critic network to complete one proximal optimization iteration. S73. Determine whether the current training round has reached the preset convergence condition. If not, continue to collect data and execute steps S1 to S72. Otherwise, determine that the agent training is complete. After a preset online fine-tuning period, read multi-source heterogeneous data from the real environment to construct the real-time input state. Input the trained agent to obtain actions and execute them. Based on the reward value of the real feedback, fine-tune and update the network parameters according to the method of step S72. S74. The weight matrices and bias vectors of the finally updated Actor network and Critic network are stored in the control terminal of the actual scheduling system to complete the collaborative processing of agent deployment and health check cycle.

[0030] Example 1: To verify the feasibility of applying this invention in the field of intelligent scheduling and resource optimization in health checkup centers, the invention was deployed in the health management center of a large tertiary general hospital in a certain city. This center receives over 1200 people for health checkups daily, covering 35 sub-items including internal medicine, surgery, radiology, and laboratory medicine, involving over 80 medical staff, triage staff, and support staff, with an average daily operating time exceeding 10 hours. The scheduling challenges faced by the health checkup center are mainly manifested in: severe congestion during the morning peak for fasting procedures (such as color Doppler ultrasound and blood collection), resource allocation conflicts between regular and VIP queues, uneven utilization of bottleneck equipment such as CT and MRI, and chaotic patient flow paths between multiple departments, resulting in excessively long average checkup completion times and significant differences in workload between departments.

[0031] The center generates daily data including: appointment registration information, arrival records by time slot, queue and call logs for each department, equipment status logs, doctor schedules, and historical behavioral patterns of examinees, with an average daily data increment exceeding 800MB. Traditional physical examination queuing systems often employ a simple FIFO (First-In, First-Out) logic or static rule-based partitioned scheduling, lacking the dynamic adaptability to individual differences among examinees, item dependencies (such as pre- and post-meal item distinctions), and real-time resource fluctuations. This often leads to uneven workloads—long queues in some departments while adjacent departments are deserted, resulting in low examinee satisfaction and an inability to cope with scheduling disruptions caused by sudden additions of items or equipment malfunctions.

[0032] In practical deployment, the method of this invention first constructs a high-fidelity digital twin simulation environment to map the aforementioned multi-source heterogeneous data into a real-time mirror of the medical examination center. It utilizes a directed acyclic graph (DAG) to precisely encode the strict medical process dependencies such as "blood collection → breakfast → abdominal ultrasound," and combines spatiotemporal alignment and feature fusion operations to construct a high-dimensional state feature vector containing dimensions such as queue length, waiting time, and resource idle rate, which serves as the input state for the agent. Subsequently, an agent based on an improved H-PPO algorithm is introduced for real-time decision-making. This agent outputs a parameterized hybrid action via an Actor network, containing discrete action components (such as choosing which queue branch to assign the examinee to) and continuous action components (such as reserving a certain proportion of buffer resources). The action is then validated using preset resource mutual exclusion and process constraint rules to ensure that the scheduling instructions comply with medical safety regulations. During the strategy optimization phase, the system introduces a cross-modal action consistency constraint factor, calculates the semantic distance between discrete decision features and continuous parameter features, and calculates immediate reward values ​​based on process completion, waiting time savings, and resource idle status, guiding the agent to seek a Pareto optimal solution among efficiency, fairness, and resource utilization. Furthermore, for ordinary queues, bottleneck queues, independent queues, and parallel queues, the system automatically matches differentiated collaborative scheduling strategies. For example, it implements logical virtual occupancy at bottleneck equipment (such as CT scanners) and interleaved services between independent departments, thereby achieving closed-loop optimization of the entire process. Table 1 below shows the comparison data of the method of this invention and the traditional FIFO queuing system on key performance indicators of scheduling in the medical examination center during a three-month trial period: Table 1. Performance Comparison Data of the Invention and Traditional FIFO Methods in Intelligent Scheduling of Medical Examination Centers

[0033] Based on the comparative data shown in Table 1, it can be seen that the intelligent scheduling method based on the improved H-PPO algorithm proposed in this invention shows significant performance advantages over the traditional FIFO method in the complex dynamic environment of the physical examination center. In particular, it has achieved comprehensive improvement in key indicators such as waiting time control, dwell time optimization, equipment resource utilization improvement and scheduling conflict elimination.

[0034] Regarding average waiting time and dwell time, this invention achieved significant reductions in all four typical test scenarios. For example, in the "morning rush hour fasting project" scenario, the traditional FIFO method lacks dynamic diversion capabilities to handle the morning surge of people arriving for testing, resulting in an average waiting time of up to 42.5 minutes and an average dwell time exceeding 2 hours. However, this invention, by integrating a hybrid action generation strategy and a real-time load balancing mechanism, reduced the average waiting time to 18.6 minutes and the average dwell time to 98.5 minutes, representing reductions of 56.2% and 27.2% respectively, effectively alleviating the physiological anxiety and queuing pressure of those being tested.

[0035] Regarding equipment resource utilization, this invention achieves accurate prediction and dynamic allocation of bottleneck resources through a deep reinforcement learning feedback mechanism. In the "bottleneck equipment inspection" scenario, traditional methods, unable to adapt to fluctuations in equipment load and differences in inspection duration, have a utilization rate of only 65.3%, often resulting in idle "people waiting for equipment" or "equipment waiting for people" idling. This invention, however, utilizes a logical virtual occupancy and intermittent service strategy to significantly increase the average equipment utilization rate to 92.1%, fully tapping the service potential of expensive medical equipment.

[0036] In terms of scheduling conflict control, this invention demonstrates an absolute advantage. Traditional methods, lacking a global perspective and process constraint verification, experience an average of up to 22 conflicts per day in "VIP and ordinary queue resource conflicts" scenarios, which can easily lead to medical disputes. This invention, however, introduces a resource mutual exclusion and process constraint legality verification mechanism, uniformly reducing the number of scheduling conflicts in all scenarios to 0, completely eliminating process logic errors and resource preemption issues.

[0037] Overall, this invention achieves efficient and conflict-free intelligent scheduling in key scenarios such as "fasting projects during morning rush hour" and "bottleneck equipment inspection" by integrating multi-source data to construct a high-dimensional state space and combining it with an improved H-PPO algorithm. In particular, it reduces the completion time of fasting projects from 112.4 minutes to 65.3 minutes, demonstrating broad practical value and filling the technical shortcomings of traditional methods in dynamic resource allocation and complex process constraints.

[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-mode queue collaborative processing method based on reinforcement learning, characterized in that, Includes the following steps: S1. Real-time collection of multi-source heterogeneous data from the physical examination center; S2. Spatiotemporal alignment and feature extraction are performed on multi-source heterogeneous data. The item dependency relationship of the examinee is extracted to construct a directed acyclic graph encoding. The queue state, department state and system global indicators are integrated to construct a high-dimensional state feature vector and map it to the input state. S3. Construct a high-fidelity digital twin simulation environment for the physical examination center, initialize an intelligent agent based on the improved H-PPO algorithm, and define a heterogeneous topology including ordinary queues, bottleneck queues, independent queues, and parallel queues. S4. Construct a multi-objective weighted fusion reward function mechanism, introduce a cross-modal action consistency constraint factor, calculate the semantic distance between discrete decision features and continuous parameter features in the Actor network, impose penalty constraints on inconsistent decisions between discrete and continuous actions, and determine positive reward terms and negative penalty terms to guide the agent to seek Pareto optimal solutions between efficiency, fairness and resource utilization. S5. Based on the current input state, output parameterized hybrid actions through the Actor network, and perform legality verification through preset resource mutual exclusion and process constraint rules to eliminate unacceptable actions and output the verified parameterized hybrid actions. S6. The agent executes the parameterized hybrid action after verification, implements differentiated collaborative scheduling strategies for different queue modes, performs peak-valley regulation for ordinary queues, implements logical virtual occupancy for bottleneck queues and combines it with the FCFS algorithm for dynamic scheduling, strictly guarantees the FCFS algorithm for independent queues and combines it with physical virtual occupancy for intermittent service, and implements dynamic snake-shaped queuing for parallel queues based on comprehensive service time and health check type. S7. After the agent executes the parameterized hybrid action after verification, it obtains the immediate reward and the state of the next moment and stores them in the experience replay buffer. Based on the near-end policy, it optimizes the target calculation gradient to update the parameters of the Actor network and Critic network. Through the online fine-tuning mechanism, it adapts to the dynamic changes of the environment and is deployed to the actual scheduling system to complete the collaborative processing.

2. The multi-mode queue collaborative processing method based on reinforcement learning according to claim 1, characterized in that, S1 specifically includes: S11. By deploying sensor groups at the entrances, triage desks and examination rooms of each department in the physical examination center, the real-time number of people waiting in each queue and the estimated waiting time are collected at a preset frequency. The busy and idle status of medical staff and equipment in the department is read in real time through the HIS system interface, and the current service speed is calculated based on historical service records. S12. Obtain the individual attributes of the examinee through an ID card reader or barcode scanner, determine the examinee's progress by querying the list of physical examination package items, summarize the queue length data and equipment utilization data of each department, and calculate the global load index of the system. S13. Align the real-time number of people waiting in each queue with the estimated waiting time, the busy / idle status of medical staff and equipment in the department and the service speed, the individual attributes and process progress of the examinees, and the global load index of the system according to a unified timestamp, and output multi-source heterogeneous data.

3. The multi-mode queue collaborative processing method based on reinforcement learning according to claim 1, characterized in that, S2 specifically includes: S21. Read the timestamps carried by all data in the multi-source heterogeneous data, unify the timestamps of all data to the preset standard time, classify the data into the corresponding spatial regions according to the physical location coordinates of the department, and complete the spatiotemporal alignment. S22. Read the list of physical examination items of the examinee, define each item as a node, connect the nodes with dependencies according to the order of medical examinations, and generate a directed acyclic graph encoding. S23. Calculate the statistical average of the real-time number of people waiting and the estimated waiting time for each queue, generate queue status characteristics, and calculate the average of the busy / idle status duration and service speed of medical staff and equipment in the department, and generate department status characteristics. S24. Perform numerical normalization calculation on the global load index of the system to obtain the global index of the system. Extract the adjacency matrix of the directed acyclic graph encoding and input it into the preset fully connected layer for vector transformation to generate a graph embedding vector. Concatenate the queue state features, department state features, global index of the system and the graph embedding vector in a preset order to generate a high-dimensional state feature vector. S25. Perform a linear transformation operation on the high-dimensional state feature vector to map the numerical range of the high-dimensional state feature vector to between 0 and 1, and output the input state.

4. The multi-mode queue collaborative processing method based on reinforcement learning according to claim 1, characterized in that, S3 specifically includes: S31. Read the building floor plan data of the physical examination center, map each department to a service node in the simulation environment, calculate the average service time and variance of each department based on historical physical examination records, construct a service time generation function that conforms to a normal distribution, and build a high-fidelity digital twin simulation environment for the physical examination center. S32. Initialize the agent based on the improved H-PPO algorithm, construct the Actor network and Critic network containing the input layer, hidden layer and output layer, and divide the output of the Actor network into two branches: discrete action head and continuous action head. S33. Calculate the average service time and average arrival rate of each department within the preset historical time window, calculate the theoretical maximum throughput of the department, use the ratio of average waiting time to average service time as the load intensity factor, and combine it with the ratio of actual throughput to theoretical maximum throughput to perform a weighted sum with preset weights to obtain the load ratio of the service capacity of each department. Define the queue corresponding to the department with a load ratio greater than the first preset threshold as the bottleneck queue, and define the queue corresponding to the department with no project dependency relationship as the independent queue. S34. Departments whose statistical service capabilities meet the preset standards and have no resource conflicts will have their corresponding queues defined as ordinary queues. Departments with multiple parallel service windows will have their corresponding queues defined as parallel queues. S35. The bottleneck queue, independent queue, ordinary queue and parallel queue are respectively used as nodes. The nodes are connected according to the physical path of the physical examination process, and the resource mutual exclusion passage rules are preset for each connection path to construct a heterogeneous topology structure including ordinary queue, bottleneck queue, independent queue and parallel queue.

5. The multi-mode queue collaborative processing method based on reinforcement learning according to claim 1, characterized in that, S4 specifically includes: S41. Calculate the number of items completed by all examinees at the current moment, divide it by the total number of items for examinees to obtain the completion rate, calculate the arithmetic mean of the completion rates of all examinees, multiply the arithmetic mean by the first preset weighting coefficient, and calculate the process completion reward item. S42. Read the estimated waiting time of each queue at the current moment, subtract the estimated waiting time of each queue at the previous moment to obtain the change in waiting time, sum up the changes in waiting time of all queues and take the negative value, multiply the negative result by the second preset weight coefficient, and calculate the waiting time saving reward item. S43. Count the number of medical personnel and equipment that are currently idle, divide the number by the total number of medical personnel and equipment to obtain the resource idle rate, multiply the resource idle rate by the third preset weight coefficient, and calculate the resource idle penalty item. S44. Read the output matrix from the last hidden layer of the Actor network, multiply the output matrix by the preset weight matrix corresponding to the discrete action head and add the bias vector to obtain the logits numerical vector of the discrete action. Input the logits numerical vector into the Softmax function for exponential normalization operation, calculate the probability value corresponding to each queue type, and output the class probability distribution. S45. Read the same output matrix from the last hidden layer of the Actor network, multiply the output matrix by the preset first set of weight parameters corresponding to the continuous action head, calculate the mean vector of the Gaussian distribution, multiply the output matrix by the preset second set of weight parameters corresponding to the continuous action head, calculate the variance vector of the Gaussian distribution, and take the logarithm of the variance vector to ensure numerical stability, and output the Gaussian distribution parameters. S46. Read the category probability distribution output by the discrete action head, expand the dimension to the same dimension as the Gaussian distribution parameters, and denote it as the first distribution component. Read the mean vector and variance vector of the Gaussian distribution output by the continuous action head, and concatenate them column by column to generate a continuous parameter distribution feature, which is denoteed as the second distribution component. Perform vector concatenation operation on the first distribution component and the second distribution component according to the preset dimension order to generate a joint feature vector, and output it as a mixed action probability distribution. S47. Read the output values ​​of the discrete action head fully connected layer of the Actor network before the activation function, and arrange them into a one-dimensional row vector as the discrete decision feature vector. Read the output values ​​of the continuous action head fully connected layer of the Actor network before the activation function, and arrange them into a one-dimensional row vector as the continuous parameter feature vector. S48. Calculate the dot product of the discrete decision feature vector and the continuous parameter feature vector and their respective L2 norm products. Divide the dot product by the L2 norm product to obtain the cosine similarity. Subtract the cosine similarity from the value 1 to obtain the semantic distance. Multiply the semantic distance by the preset consistency penalty coefficient to calculate the cross-modal action consistency constraint factor. When the discrete action and the continuous action decision are inconsistent, it is used as the penalty value. If they are consistent, the penalty value is set to zero. S49. The process completion reward item, waiting time saving reward item, resource idle penalty item and penalty value are weighted and summed according to preset weights to calculate the instant reward value, and the advantage function estimate is calculated in combination with the time difference error. S410. Read the instant reward value and the input state at the next moment from the experience playback buffer. Add the instant reward value to the product of the Critic network's value estimate of the input state at the next moment multiplied by a preset discount factor. Subtract the Critic network's value estimate of the input state at the current moment to calculate the time difference error. Subtract the corresponding mean from the time difference error and divide by the standard deviation to calculate the dominance function. S411. Read the mixed action probability distribution generated in step S46, calculate the probability ratio of the current policy and the old policy on the corresponding actions, introduce a preset truncation coefficient, determine whether the probability ratio exceeds the preset interval, if it does, replace the probability ratio with the boundary value of the preset interval to obtain the truncated probability ratio, multiply the advantage function by the truncated probability ratio to construct the objective function, calculate the gradient based on the objective function, and backpropagate to update the parameters of the Actor network and the Critic network.

6. The multi-mode queue collaborative processing method based on reinforcement learning according to claim 1, characterized in that, S5 specifically includes: S51. Read the current input state and input it into the hidden layer of the Actor network for forward propagation calculation. Extract state features through linear transformation and nonlinear activation function, and output the intermediate state feature matrix. S52. Input the intermediate state feature matrix into the discrete action head, multiply it by the preset weight matrix corresponding to the discrete action head and add the bias vector. After normalization by the Softmax function, output the probability value of selecting each queue type and take the queue type with the highest probability value as the discrete action component. S53. Input the intermediate state feature matrix into the continuous action head, multiply it by the preset first set of weight parameters corresponding to the continuous action head to obtain the mean vector of the Gaussian distribution, multiply it by the preset second set of weight parameters corresponding to the continuous action head to obtain the variance vector of the Gaussian distribution, randomly sample a noise vector from the standard normal distribution, multiply the mean vector and the noise vector element by element and then linearly combine them with the variance vector to generate the specific value of resource pre-allocation as the continuous action component. S54. Combine discrete action components with continuous action components to generate parameterized hybrid actions, read preset resource mutual exclusion and process constraint rules, check whether the queue type determined by the discrete action components has a physical path conflict with the unfinished items of the current examinee, check whether the specific value of the pre-allocated resources determined by the continuous action components exceeds the range of the number of medical personnel and equipment currently available in the corresponding department, and at the same time verify whether the medical dependency order of the physical examination items is met. S57. If the parameterized mixed action passes all checks, it is determined to be an actionable action and is retained. If any conflict exists, it is determined to be an actionable action, which is then removed from the action space. The discrete action components are forcibly replaced with the default queue type that conforms to the process constraints, and the verified parameterized mixed action is output.

7. The multi-mode queue collaborative processing method based on reinforcement learning according to claim 1, characterized in that, S6 specifically includes: S61. Read the parameterized mixed action after verification, parse the discrete action components to obtain the type label of the target queue, and parse the continuous action components to obtain the specific value of resource pre-allocation. S62. When the target queue type label is ordinary queue, calculate the current system global load index value. If the value is greater than the preset high load threshold, calculate the reciprocal of the number of people waiting in real time in each ordinary queue branch as the selection probability, and randomly select the branch with fewer people for diversion according to the probability. If the value is less than the preset low load threshold, directly add the examinee to the end of the queue to complete peak and valley control. S63. When the target queue type label is bottleneck queue, read the current waiting sequence of the bottleneck queue, calculate the service duration requirement of the logical virtual placeholder based on the specific value of resource pre-allocation multiplied by the average service duration of the bottleneck queue, generate a logical virtual placeholder mark carrying the service duration requirement, traverse all existing placeholder marks in the bottleneck queue, and calculate the time gap width between two adjacent placeholder marks. S64. Compare the service duration requirement of the logical virtual placeholder with the width of each time slot one by one, filter out the smallest time slot that is greater than the service duration requirement, insert the logical virtual placeholder into the starting position of the smallest time slot, update the expected start time of all subsequent examinees in the queue sequence, and for other examinees who have not inserted the logical virtual placeholder, read the arrival timestamp and rearrange the service order according to the order of the timestamps using the FCFS algorithm. S65. When the target queue's type label is independent queue, read the start time and average service duration of the patients currently receiving services in the queue, add the two together to get the expected end time, and subtract the current system time from the expected end time to get the remaining service time. S66. Compare the remaining service time with the preset gap threshold. If the remaining service time is less than the gap threshold, it is determined that the current service is about to end and a service gap is generated. Calculate the service duration of the person to be inserted for physical examination based on the specific value of the pre-allocated resources. If the service duration is less than or equal to the gap corresponding to the remaining service time, a physical virtual placeholder is generated. S67. Map the physical virtual placeholder to the specific identity information of the examinee, insert it into the time node after the current service ends, update the queuing number of the subsequent examinees, and for regular examinees who are not inserted, strictly arrange the service order according to the arrival timestamp to complete the intermittent service. S68. When the target queue type label is parallel queue, read the current queue length and average service time of each parallel service window, calculate the expected waiting time of each window, select the window with the smallest expected waiting time as the target window, read the physical examination type of the examinee, and assign examinees of the same physical examination type to different parallel windows in an S-shaped order to complete the dynamic snake-shaped queue.

8. The multi-mode queue collaborative processing method based on reinforcement learning according to claim 1, characterized in that, Specifically, S7 includes: S71. After the agent executes the verified parameterized hybrid action, it reads the instant reward value calculated in step S49 and obtains the input state of the next moment from the simulation environment. It combines the instant reward value, the current input state, the verified parameterized hybrid action, and the input state of the next moment into an experience data unit and stores it in the experience playback buffer. S72. Monitor the amount of data stored in the experience playback buffer. When the amount of data stored reaches the preset batch size, randomly select a set of experience data units, read the instant reward value and status information, call the method described in step S410 to calculate the time difference error and advantage function, and update the parameters of the Actor network and Critic network to complete one proximal optimization iteration. S73. Determine whether the current training round has reached the preset convergence condition. If not, continue to collect data and execute steps S1 to S72. Otherwise, determine that the agent training is complete. After a preset online fine-tuning period, read multi-source heterogeneous data from the real environment to construct the real-time input state. Input the trained agent to obtain actions and execute them. Based on the reward value of the real feedback, fine-tune and update the network parameters according to the method of step S72. S74. The weight matrices and bias vectors of the finally updated Actor network and Critic network are stored in the control terminal of the actual scheduling system to complete the collaborative processing of agent deployment and health check cycle.

Images