Reinforcement learning-based dynamic optimal scheduling method for teleconsultation

Through the dynamic optimization scheduling method based on reinforcement learning, the problems of waste and low efficiency of traditional scheduling methods in remote consultation are solved, and dynamic matching and optimization of remote consultation needs and resources are achieved, service quality and efficiency are improved, and self-service is provided. Adaptable to learning ability.

WO2024234401A1PCT designated stage expired Publication Date: 2024-11-21THE FIRST AFFILIATED HOSPITAL OF ZHENGZHOU UNIV

Patent Information

Application Number
PCT/CN2023/095946
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-15
Filing Date
2023-05-24
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Traditional remote consultation scheduling methods are based on fixed rules or heuristic algorithms, and are difficult to adapt to dynamic changes and diversity, resulting in waste of resources, low service efficiency and quality, but high service costs.

Method used

The dynamic optimization scheduling method based on reinforcement learning is adopted, and the Q-learning model is trained to dynamically match the remote consultation needs and resources to achieve optimal utilization of resources.

Benefits of technology

It improves the efficiency and accuracy of remote consultation and scheduling, saves labor costs and management time, improves service quality, and has the ability to adapt to changes in external conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023095946_21112024_PF_FP_ABST
    Figure CN2023095946_21112024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of teleconsultation scheduling, and provides a reinforcement learning-based dynamic optimal scheduling method for teleconsultation. The problems that conventional scheduling methods are based on fixed rules or heuristic algorithms, and are thus difficult to adapt to dynamic changes and diversity of teleconsultation and easy to cause waste of resources, resulting in low service efficiency and quality but high service cost, are solved, dynamic matching and optimization of teleconsultation requirements and resources are achieved, the efficiency and accuracy of teleconsultation scheduling are improved, the labor cost and management time are reduced, and the quality of teleconsultation service is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

A dynamic optimization scheduling method for remote consultation based on reinforcement learning Technical Field

[0001] The present invention relates to the technical field of remote consultation scheduling, and in particular to a remote consultation dynamic optimization scheduling method based on reinforcement learning. Background Art

[0002] Teleconsultation is a service model that leverages information technology and network technologies to facilitate expert consultations between medical institutions in different locations. Typically, doctors and patients at primary hospitals request a teleconsultation through an online platform from experts at higher-level hospitals, who are then assigned and arranged by a management center. Teleconsultation can improve the utilization efficiency of medical resources, alleviate resource imbalances, and enhance patient satisfaction.

[0003] However, this process involves the coordination and arrangement of multiple types of resources and personnel. Scheduling is a complex multi-objective optimization problem that involves the needs and preferences of multiple parties and is subject to equipment and resource constraints. Traditional scheduling methods are often based on fixed rules or heuristic algorithms. These methods, based on fixed rules or specific training models, are difficult to adapt to the dynamic changes and diversity of remote consultations. When external conditions change, the model often needs to be modified or retrained, which can easily lead to wasted resources, low service efficiency and quality, and high service costs.

[0004] Summary of the Invention

[0005] The present invention discloses a dynamic optimization scheduling method for remote consultation based on reinforcement learning, which solves the problem that traditional scheduling methods are based on fixed rules or heuristic algorithms, are difficult to adapt to the dynamic changes and diversity of remote consultations, easily cause waste of resources, and lead to low service efficiency and quality, but high service costs. It realizes dynamic matching and optimization of remote consultation needs and resources, improves the efficiency and accuracy of remote consultation scheduling, saves labor costs and management time, and effectively improves the quality of remote consultation services.

[0006] To achieve the above object, the technical solution of the present invention is specifically implemented as follows:

[0007] The present invention discloses a remote consultation dynamic optimization scheduling method based on reinforcement learning, comprising the following steps:

[0008] S1: Construct a state space S, which is the set of all environmental states s. The environmental state s includes at least the state of the waiting queue, the current consultation venue time resource state, and the current expert resource state. The waiting array state is the set of all consultation requests, each of which includes at least the consultation department, disease type, condition description label, and whether it is urgent.

[0009] S2: Construct an action space A, which is the set of all legal actions a. The action space includes at least triage and detriage.

[0010] S3: Training the model. Training the model includes the following steps:

[0011] S31: Initialize the Q-table that stores state-action pairs;

[0012] S32: Initialize N sa , the N sa Indicates the frequency of selecting action a in state s, with an initial value of 0;

[0013] S33: Define the exploration function f(u,n), which determines whether the model is more inclined to choose the existing optimal solution u or to explore actions that are not frequently tried and have a small frequency n;

[0014] S34: Define the reward function R(s,a,s′), where R(s,a,s′) represents the reward obtained by taking action a in state s to reach state s′.

[0015] S35: Define the learning rate α and discount factor γ;

[0016] S36: If the current triage queue is not empty, increase N sa The value of [s,a], where s and a are the state and action of the previous time series, are initially set to empty. The value in the Q-table is updated according to the following formula:

[0017] in, Indicates the maximum possible value when the state value is s′ in the Q-table;

[0018] S37: Generate the subsequent state s and action a according to the following formula, where a is obtained by the exploration function defined in step S33, and s is the new state generated after executing the action a; s,a←s′,argmax a′ f(Q[s′,a′],N sa [s′,a′])

[0019] S38: Go to step S36 and continuously update Q-table and N sa , and continuously train the model.

[0020] Furthermore, the remote consultation dynamic optimization scheduling method based on reinforcement learning also includes the following steps:

[0021] S4: Manual feedback, which selects and modifies the triage plan through manual operation and assigns higher values ​​to the corresponding state-action pairs in the Q-table;

[0022] S5: Test and improve, test and evaluate its performance in a real environment, and optimize various parameters defined in the training process based on the results.

[0023] Furthermore, the Q-table is initialized with small random values.

[0024] Furthermore, we explore the function f(u,n) which increases with u and decreases with n.

[0025] Furthermore, the reward function R(s,a,s′) is negatively correlated with the waiting time of consultation application.

[0026] Furthermore, the current time resource status of the consultation site is represented by a two-dimensional table with the horizontal axis representing the consultation site and the vertical axis representing time slots in which the working time is divided according to the standard consultation duration.

[0027] Furthermore, the current expert resource status is represented by an array.

[0028] Beneficial technical effects:

[0029] 1. The present invention discloses a dynamic optimization scheduling method for remote consultation based on reinforcement learning. This method solves the problem that traditional scheduling methods are based on fixed rules or heuristic algorithms, which are difficult to adapt to the dynamic changes and diversity of remote consultations, easily lead to resource waste, and result in low service efficiency and quality, but high service costs. It realizes dynamic matching and optimization of remote consultation needs and resources, improves the efficiency and accuracy of remote consultation scheduling, saves labor costs and management time, and effectively improves the quality of remote consultation services.

[0030] 2. The present invention adopts a reinforcement learning algorithm, which can dynamically adjust the allocation of remote consultations according to the real-time needs and resource conditions of remote consultations, thereby achieving optimal matching and utilization;

[0031] 3. The present invention has the ability of adaptive learning and continuous optimization, and can continuously update and improve the scheduling strategy based on the historical data and feedback of remote consultations. When external conditions change, there is no need to rewrite rules or train models;

[0032] 4. The present invention requires more manual feedback in the early stage, but theoretically, as the running time increases, it can gradually approach the manual selection results without writing complex rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0034] FIG1 is a flowchart of the steps of a dynamic optimization scheduling method for remote consultation based on reinforcement learning according to the present invention. DETAILED DESCRIPTION

[0035] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0037] Unless otherwise specifically stated, the relative arrangement of the parts and steps, the numerical expressions and the numerical values ​​set forth in these embodiments do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The techniques, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods and equipment should be considered as part of the authorization specification. In all examples shown and discussed here, any specific values ​​should be interpreted as being merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following figures, and therefore, once an item is defined in one figure, it does not need to be further discussed in subsequent figures.

[0038] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0039] The present invention discloses a dynamic optimization scheduling method for remote consultation based on reinforcement learning, referring to FIG1 , which specifically includes the following steps:

[0040] S1: Construct a state space S, which is the set of all environmental states s. The environmental state s includes at least the state of the waiting queue, the current consultation venue time resource state, and the current expert resource state. The waiting array state is the set of all consultation applications, expressed as [application for consultation 1, application for consultation 2, application for consultation 3, etc.]. Each consultation application includes at least the consultation department, disease type, condition description label, and whether it is urgent.

[0041] Specifically, when constructing the state space S, it should include as many possible states of the environment as possible. The external environment can be abstractly represented by state s, that is, the state at each decision. s should at least have the following elements: the state of the queue to be triaged, the current time and resource status of the consultation venue, and the current expert resource status. Among them, the queue to be triaged is empty, indicating that it has reached the terminal state. New consultation applications will enter the queue to be triaged. Each consultation application must contain the following key elements: consultation department, disease type, condition description label, and whether it is urgent. After triage, the elements will be transferred from the queue to be triaged to the triaged queue.

[0042] Preferably, the current time resource status of the consultation site is represented by a two-dimensional table with the horizontal axis representing the consultation site and the vertical axis representing the time slots in which the working time is divided according to the standard consultation duration, as shown in Table 1:

[0043] Table 1

[0044] Table 1 uses time slots to represent time resources. For example, if 30 minutes is fixed as the standard consultation time, we have 6 time slots in the three hours from 9:00 am to 12:00 pm on weekdays.

[0045] Usually, the time that experts participate in consultations every week is fixed, or at least has a certain regularity. We need to know which experts are available in the corresponding departments and diseases at a certain time slot. Therefore, we can use relatively simple data structures such as arrays to represent expert resources and find out whether the experts are free at a certain time period on a certain day.

[0046] It can be understood that the state s is a tuple, which is an abstract representation of the actual scene.

[0047] S2: Construct an action space A, which is the set of all legal actions a. The action space includes at least triage and detriage.

[0048] Specifically, triage involves taking a consultation application from the waiting queue and placing it in the triaged queue. A matching decision is made based on consultation needs, time, and expert expertise. The application is then bound to a time slot in a consultation room and to an expert who is available for appointment during that time slot. This means that a specific expert is scheduled to conduct a consultation for a specific application at a specific time.

[0049] Canceling triage is the opposite of the triage process. It places a triaged application from the triaged queue back to the pending triage queue, and at the same time sets the bound venue resources and expert resources to available status.

[0050] S3: Training model:

[0051] The agent interacts with the environment by selecting actions and receiving rewards. During training, the agent learns to take actions to maximize the reward signal. The agent uses the Q-learning algorithm to update the Q-table, which updates the Q value of each state-action pair based on the observed reward and the maximum Q value of the next state. Specifically, it includes the following steps:

[0052] S31: Initialize the Q-table that stores state-action pairs;

[0053] Specifically, the Q-table stores the corresponding action in each state, that is, the expected reward of the state-action pair, and the Q-table is initialized with a small random value;

[0054] S32: Initialize N sa , the N sa Indicates the frequency of selecting action a in state s, with an initial value of 0;

[0055] S33: Define the exploration function f(u,n), which determines whether the model is more inclined to choose the existing optimal solution u or to explore actions that are not frequently tried and have a small frequency n;

[0056] Specifically, the function f(u,n) is explored to increase with respect to u and decrease with respect to n;

[0057] S34: Define the reward function R(s,a,s′), where R(s,a,s′) represents the reward obtained by taking action a in state s to reach state s′.

[0058] Specifically, the reward function R is a mathematical function that maps the agent's behavior to a scalar value that represents the quality of the behavior. R(s,a,s′) represents the reward obtained by taking action a in state s to reach state s′. The reward is negatively correlated with the waiting time for consultation applications and positively correlated with the utilization rate of site experts. Taking into account emergency needs, a higher incentive weight is given to critical and severe consultations.

[0059] S35: Define the learning rate α and discount factor γ;

[0060] Specifically, the learning rate defines the difference in utility between successive states and is also an important parameter in deep learning, determining whether the objective function can converge to a local optimum and how quickly it converges to the optimum. The existence of the discount factor makes the model tend to obtain rewards as quickly as possible. The smaller the value, the smaller the impact of long-term returns.

[0061] S36: If the current triage queue is not empty, increase N sa The value of [s,a], where s and a are the state and action of the previous time series, are initially set to empty. The value in the Q-table is updated according to the following formula:

[0062] in, Indicates the maximum possible value when the state value is s′ in the Q-table;

[0063] S37: Generate the subsequent state s and action a according to the following formula, where a is obtained by the exploration function defined in step S33, and s is the new state generated after executing the action a; s,a←s′,argmax a′ f(Q[s′,a′],N sa [s′,a′])

[0064] S38: Go to step S36 and continuously update Q-table and N sa , and continuously train the model.

[0065] S4: Manual feedback, which selects and modifies the triage plan through manual operation and assigns higher values ​​to the corresponding state-action pairs in the Q-table;

[0066] Specifically, manual selection and modification of triage plans are permitted. In this case, manually selected results and actions should be given higher value, and the corresponding state-action pairs in the Q-table should be assigned higher values, so that the agent is more likely to choose the result provided by manual feedback when faced with the same scenario in the future.

[0067] S5: Test and improve, test and evaluate its performance in a real environment, and optimize various parameters defined in the training process based on the results.

[0068] Specifically, the agent is tested in a real environment and its performance is evaluated. Based on the results, various parameters defined in the training process, such as the Q-table and reward function, are optimized.

[0069] The reinforcement learning-based dynamic optimization scheduling method for remote consultation disclosed in the present invention adopts a reinforcement learning algorithm and can dynamically adjust the allocation of remote consultation according to the real-time needs and resource conditions of the remote consultation to achieve optimal matching and utilization; and has the ability of adaptive learning and continuous optimization, and can continuously update and improve the scheduling strategy based on the historical data and feedback of the remote consultation, that is, when external conditions change, there is no need to rewrite rules or train models.

[0070] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0071] The above embodiments are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary engineers and technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A dynamic optimization scheduling method for remote consultation based on reinforcement learning, characterized in that: The following steps are involved: S1: Construct a state space S, which is a set of all environmental states s, and the environmental state s at least includes the state of the queue to be triaged, the current time resource state of the consultation venue, and the current expert resource state; the state of the array to be triaged is a set of all consultation applications, and each consultation application at least includes the consultation department, disease type, condition description label, and whether it is urgent; S2: construct an action space A, which is a set of all legal actions a, and the action space at least includes triage and cancellation of triage; S3: Training the model. The training model includes the following steps: S31: Initialize the Q-table storing state-action pairs; S32: Initialize N sa , the N sa Indicates the frequency of selecting action a in state s, with an initial value of 0; S33: Define the exploration function f(u,n), which determines whether the model is more inclined to choose the existing optimal solution u or to explore actions that are not frequently tried and have a small frequency n; S34: Define reward function R(s,a,s′), where R(s,a,s′) represents the reward obtained by taking action a in state s to reach state s′; S35: define learning rate α and discount factor γ; S36: If the current triage queue is not empty, increase N sa [s,a], where s and a are the state and action of the previous time series, which are initially set to be empty. Update the value in the Q-table according to the following formula: in, It indicates the maximum possible value when the state value is s′ in the Q-table; S37: Generate a subsequent state s and action a according to the following formula, where a is obtained by the exploration function defined in step S33, and s is a new state generated after executing the action a; s,a←s ′ ,argmax a′ f(Q[s ′ ,a ′ ],N sa [s ′ ,a ′ ]) S38: Go to step S36, and continuously update Q-table and N sa , and continuously train the model.

2. According to claim 1, a remote consultation dynamic optimization scheduling method based on reinforcement learning is characterized in that: The following steps are also included: S4: Manual feedback, select and modify the triage plan through manual operation, and assign it to the Q-table The corresponding state-action pair in is higher value; S5: Test and improve, test and evaluate its performance in a real environment, and optimize the various parameters defined in the training process based on the results.

3. According to claim 1, a remote consultation dynamic optimization scheduling method based on reinforcement learning is characterized in that: The Q-table is initialized with small random values.

4. According to the method of dynamic optimization scheduling of remote consultation based on reinforcement learning in claim 1, it is characterized in that: Explore the function f(u,n) that increases with u and decreases with n.

5. According to the reinforcement learning-based remote consultation dynamic optimization scheduling method of claim 1, it is characterized in that: The reward function R(s,a,s′) is negatively correlated with the waiting time of consultation application.

6. According to the method of dynamic optimization scheduling of remote consultation based on reinforcement learning in claim 1, it is characterized in that: The current time resource status of the consultation site is represented by a two-dimensional table with the consultation site as the horizontal axis and the time slots in which the working time is divided according to the standard consultation duration as the vertical axis.

7. The method for dynamic optimization and scheduling of remote consultation based on reinforcement learning according to claim 1 is characterized in that: The current expert resource status is represented by an array.

Citation Information

Patent Citations

  • Triage data processing method and device, equipment and medium

    CN112017789A

  • Auxiliary intelligent inquiry system based on deep reinforcement learning

    CN113470811A

  • Container memory elastic expansion and contraction method based on combination of fuzzy inference and reinforcement learning

    CN113886095A

  • Deep reinforcement learning-based inquiry method and system

    CN114613489A

  • Job-shop adaptive scheduling method based on deep reinforcement learning

    CN114707881A

Cited By

  • Deep learning-based expiration experiment triage and queuing optimization method and system

    CN120766911A

  • Large-current constant current source control method based on reinforcement learning algorithm

    CN121209644A