An order intelligent allocation system for accompanying diagnosis service based on reinforcement learning

Abstract

The present application relates to service scheduling technical field, especially to a kind of based on reinforcement learning's accompanying diagnosis business order intelligent allocation system, through event stream analysis module accompanying diagnosis order performance data is classified as controllable stage data and waiting stage data, reference determination module determines reference accompanying diagnosis order set from similar historical accompanying diagnosis order without using similar action, determines phased performance reference value based on controllable stage data of reference accompanying diagnosis order set, evaluation determination module determines action evaluation value according to the deviation of controllable stage data and corresponding stage performance reference value, strategy update module updates allocation strategy network according to action evaluation value, and the adoption proportion of collaborative allocation action is controlled according to confidence degree.The present application reduces the interference of service place queuing condition to allocation decision evaluation, improves the decision quality of order allocation.

Description

Technical Field

[0001] This invention relates to the field of service scheduling technology, and in particular to an intelligent allocation system for escort medical services orders based on reinforcement learning. Background Technology

[0002] With the continuous growth in demand for medical companionship, intelligent allocation of companionship orders has become a core aspect of the operational efficiency of companionship platforms. Existing allocation systems mostly match orders based on conditions such as location and skill tags. Some platforms introduce reinforcement learning, using order completion results, fulfillment time, and user reviews as reward signals to iteratively optimize the order allocation strategy. However, the fulfillment process of companionship orders includes waiting and examination queues, which are dominated by the queuing situation at the service location. These queuing times are uncontrollable and constitute a significant portion of the process. Using the overall fulfillment result as a reward signal means that random fluctuations in queuing are factored into the evaluation of allocation decisions, interfering with strategy learning. Furthermore, companionship orders vary significantly across departments and examination items, making it difficult to compare fulfillment results and determine the true contribution of a single allocation decision. Simultaneously, training data only includes the fulfillment results of executed actions, lacking historical data on complex allocation methods where multiple people collaborate to complete the same companionship order, making it difficult to effectively evaluate and apply the strategy. These problems result in slow strategy convergence and unstable allocation quality in existing allocation systems, particularly in terms of queuing interference removal and the evaluation and application of collaborative allocation methods.

[0003] Chinese Patent Publication No. CN110322953A discloses an intelligent escort system and its method for accompanying patients during medical visits, belonging to the field of medical technology. The intelligent escort system includes an information collection platform, an information recognition platform, a cloud storage platform, and an information processing platform. The information processing platform includes a triage module, an appointment registration module, a medical order module, a route planning module, and a query module. The medical order module informs patients of the location, time, and precautions for their medical visits, and provides multiple reminders to prevent patients from missing their appointments. Simultaneously, the route planning module guides patients through the order of their medical visits and provides location navigation for the relevant departments, reducing patient waiting time and rationally allocating medical resources. Compared with existing technologies, the intelligent escort system provided by this invention can save patients' time, improve efficiency, and offer significant convenience to patients unfamiliar with hospitals, especially those seeking medical treatment in other locations.

[0004] However, the following problems still exist in the existing technology: Existing escort order allocation systems typically use the overall fulfillment results of escort orders as a reward signal to optimize strategies. However, the fulfillment time of escort orders is mainly determined by the queuing situation at the service venue. Once the queuing situation fluctuates randomly, the fluctuation will be included in the evaluation of the allocation decision. Furthermore, there is no unified benchmark for comparison between the fulfillment results of each escort order, making it difficult to determine the true contribution of the allocation decision to the fulfillment results. Summary of the Invention

[0005] To address this, the present invention provides an intelligent allocation system for escort medical service orders based on reinforcement learning, which overcomes the problem in the prior art that, in the scenario of escort medical service order allocation, the overall performance of escort medical service orders is used as a reward signal for strategy optimization, resulting in random fluctuations in the queuing situation at the service location interfering with the evaluation of allocation decisions and making it difficult to accurately measure the true contribution of allocation decisions.

[0006] To achieve the above objectives, the present invention provides an intelligent order allocation system for escort services based on reinforcement learning, comprising: The state construction module is used to obtain the characteristics of the accompanying medical order, the personnel characteristics of the candidate accompanying medical staff, and the event process characteristics of the service location in order to construct the environmental state. The strategy decision module is used to determine the target allocation action from the extended action space based on the allocation strategy network and the environmental state. The fulfillment data collection module is used to collect the fulfillment data of the accompanying medical appointment orders. The event stream parsing module is used to divide the escort order fulfillment data into controllable stage data and waiting stage data according to the service process event progress data of the service venue and the service event and controllable attributes. The reference determination module is used to determine a set of reference escort orders that did not use the same action as the target allocation action from historical escort orders similar to the escort order, and to determine the phased performance reference value based on the controllable stage data of the reference escort order set. The evaluation and determination module is used to determine the action evaluation value based on the deviation between the controllable stage data and the corresponding performance reference value. The strategy update module is used to update the allocation strategy network based on the action evaluation value, determine the confidence level of the action evaluation value, and control the adoption ratio of collaboratively allocated actions according to the confidence level.

[0007] Furthermore, the state construction module includes: The accompanying medical information submodule is used to determine the characteristics of accompanying medical orders based on data on service demand content; The skill status submodule is used to obtain the skill parameters, real-time location, and committed service time of each candidate escort in order to determine the characteristics of the personnel; The event process submodule is used to obtain real-time progress data of each service process event within the service location, determine the expected arrival time distribution of subsequent service events for the escort order, and thus determine the event process characteristics.

[0008] Furthermore, the process by which the strategy decision module determines the target allocation action from the extended action space based on the allocation policy network and the environmental state includes: Based on the aforementioned personnel characteristics, determine the candidate assignment actions that each candidate escort can perform; The expected fulfillment time of each candidate allocation action is determined based on the event process characteristics; The value of each candidate assigned action in the extended action space is evaluated to determine the action value of each candidate assigned action. The target assignment action is determined based on the action value of each candidate assignment action and a preset random selection ratio. The extended action space includes whole-order assignment actions and collaborative allocation actions.

[0009] Furthermore, the process by which the fulfillment collection module collects the fulfillment data of the accompanying medical appointment order includes: In response to the execution of the accompanying medical appointment order, collect the accompanying medical appointment order completion result, accompanying medical appointment order interruption event, and user evaluation data; Collect service process records from each accompanying medical staff member; If the target allocation action is a collaborative allocation action completed by multiple escorts for the same escort order, a handover record is generated at the handover point between adjacent service stages.

[0010] Furthermore, the event stream parsing module includes: The phase segmentation module is used to segment the escort order fulfillment data into service phase data corresponding to each service event according to a preset event flow template; The attribute determination submodule is used to distinguish the service stage data into controllable stage data and waiting stage data based on the controllability determination criteria.

[0011] Furthermore, the reference determination module includes: The similarity retrieval submodule is used to calculate the similarity between the accompanying medical order and each historical accompanying medical order based on the accompanying medical order characteristics, and to determine the set of similar accompanying medical orders; The reference filtering submodule is used to filter historical escort orders that did not use the same type of action as the target allocation action from the similar escort order set, and to determine the reference escort order set; The reference value determination submodule is used to perform statistical calculations based on the controllable stage data of the reference escort order set to determine the phased performance reference value.

[0012] Furthermore, the process of determining the phased performance reference values ​​in the reference value determination submodule includes: Based on the waiting stage data, the accompanying medical order is aligned with each historical accompanying medical order in the reference accompanying medical order set according to the service event stage; Extract the controllable stage data of each historical escort order after alignment according to the service stage, perform the statistical calculation, and determine the staged performance reference value; The waiting stage data is used for stage alignment and is not included in the statistical calculation.

[0013] Furthermore, the process by which the evaluation and determination module determines the action evaluation value based on the deviation between the controllable stage data and the corresponding performance reference value includes: Calculate the difference between the data of each service stage in the controllable stage data and the corresponding performance reference value of the stage, and determine the stage deviation value of each service stage; If the target assignment action is a complete order assignment action performed by a single escort, then the sum of the stage deviation values ​​of each service stage is determined as the action evaluation value. If the target allocation action is a collaborative allocation action in which multiple escorts work together to complete the same escort order, then according to the service stage each escort is responsible for, the stage deviation value of the corresponding service stage is respectively assigned to each escort, and the action evaluation value is determined based on the corresponding results.

[0014] Furthermore, the policy update module includes: The network update submodule is used to update the network parameters of the allocation strategy network according to the action evaluation value; The confidence determination submodule is used to determine the confidence level of the action evaluation value based on the sample size of the reference escort order set; A proportional control submodule, which controls the adoption ratio of the collaboratively allocated actions according to the confidence level; The proportion of the collaborative allocation action adopted increases with the increase of the confidence level.

[0015] Furthermore, the confidence determination submodule, based on the sample size of the reference escort order set, determines the confidence level of the action evaluation value, including: Count the number of reference samples corresponding to each service stage in the aforementioned set of reference escort orders; The number of reference samples corresponding to each service stage is compared with the preset sample size benchmark to determine the sub-confidence of each service stage. The confidence level of the action evaluation value is determined based on each of the sub-confidence levels.

[0016] Compared with existing technologies, this invention classifies escort service order fulfillment data into controllable stage data and waiting stage data through an event flow parsing module. A reference determination module identifies a set of reference escort service orders from similar historical orders that did not employ the same actions. Based on the controllable stage data of the reference escort service order set, a staged fulfillment reference value is determined. An evaluation determination module determines the action evaluation value based on the deviation between the controllable stage data and the corresponding stage's fulfillment reference value. A strategy update module updates the allocation strategy network based on the action evaluation value, controlling the adoption ratio of collaboratively allocated actions according to confidence levels. This invention reduces the interference of queuing conditions at service locations on allocation decision evaluation and improves the quality of order allocation decisions.

[0017] In particular, this invention addresses the problem of queuing conditions interfering with allocation decision evaluation. In actual operation, the fulfillment time of escort orders is mainly occupied by waiting, examination queuing, and report issuance. These stages are dominated by the queuing conditions of the service venue, and the escort's service behavior and allocation decisions cannot change their duration. If the overall fulfillment result of escort orders is used as the basis for strategy optimization, random fluctuations in queuing conditions will be mistakenly interpreted as the quality of allocation decisions, and the order dispatch strategy will tend to select escort orders with shorter queues rather than making better allocations. The event flow parsing module of this invention divides the escort order fulfillment data into controllable stage data and waiting stage data according to service events and controllable attributes based on service process event progress data. The action evaluation value is determined only based on the controllable stage data, and the waiting stage data does not participate in the evaluation calculation. This excludes random fluctuations in queuing conditions from the evaluation of allocation decisions, ensuring the correctness of the strategy optimization direction.

[0018] In particular, addressing the lack of comparability in the fulfillment results of various escort service orders, this invention provides a unified measurement benchmark across escort service orders. Different escort service orders differ in department type, examination items, and patient conditions. Directly comparing the fulfillment results of escort service orders with varying levels of difficulty fails to reflect the true allocation situation. The reference determination module of this invention identifies a set of reference escort service orders from historical escort service orders similar to the target allocation action that did not employ such actions. Based on controllable stage data from the reference escort service order set, it determines staged fulfillment reference values. The action evaluation value only assesses the service effect that exceeds the normal level of similar escort service orders resulting from the allocation decision, reducing the impact of the difficulty level of the escort service order itself. For collaborative allocation actions, the action evaluation value can also be assigned to each executing escort service member according to the service stage, enabling the service contribution of each escort service member to be determined separately when multiple people collaborate to complete the same escort service order. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the intelligent allocation system for escort services based on reinforcement learning, as described in an embodiment of the present invention. Figure 2This is a schematic diagram illustrating how the fulfillment data of a patient escort order is divided into controllable stage data and waiting stage data according to an embodiment of the present invention; Figure 3 This is a logic diagram for determining action evaluation values ​​in an embodiment of the present invention; Figure 4 This is a logic diagram showing the proportion used in controlling the collaborative allocation action in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0021] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0022] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0023] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0024] Please see Figure 1 As shown, this is a schematic diagram of the intelligent allocation system for escort services based on reinforcement learning according to an embodiment of the present invention. The system of this embodiment includes: The state construction module is used to obtain the characteristics of the accompanying medical order, the personnel characteristics of the candidate accompanying medical staff, and the event process characteristics of the service location in order to construct the environmental state. The strategy decision module is used to determine the target allocation action from the extended action space based on the allocation strategy network and the environmental state. The fulfillment data collection module is used to collect fulfillment data for escort service orders. The event stream parsing module is used to divide the escort order fulfillment data into controllable stage data and waiting stage data according to the service process event progress data of the service venue and the service event and controllable attributes. The reference determination module is used to determine the set of reference medical companion orders that did not adopt the same type of action as the target allocation action from historical medical companion orders similar to medical companion orders, and to determine the phased performance reference value based on the controllable phase data of the reference medical companion order set. The evaluation and determination module is used to determine the action evaluation value based on the deviation between the controllable stage data and the corresponding stage performance reference value; The strategy update module is used to update the allocation strategy network based on the action evaluation value, determine the confidence level of the action evaluation value, and control the adoption ratio of collaboratively allocated actions according to the confidence level.

[0025] It should be noted that the multiple functional modules involved in this application are only logical divisions based on the functions implemented according to the present invention. The specific physical structures of the state construction module, strategy decision module, performance acquisition module, event flow parsing module, reference determination module, evaluation determination module, and strategy update module are not limited. They can be composed of logical components, including field-programmable gate arrays, application-specific integrated circuits, computers, or microprocessors in computers, which will not be elaborated here.

[0026] Specifically, the state construction module, the strategy decision-making module, the event flow parsing module, the reference determination module, the evaluation determination module, and the strategy update module are deployed on the server side of the escort platform. On the platform side, they complete environment state construction, allocation decisions, data segmentation, reference value determination, evaluation, and strategy updates. The fulfillment collection module communicates with the escort terminal and the user terminal to obtain service process records such as arrival time, start time, and end time of each service step from the escort terminal, and to obtain user evaluation data from the user terminal. The escort order completion result and escort order interruption event are provided by the escort order management system on the platform side.

[0027] Specifically, this invention constructs the environmental state at the current allocation moment through a state construction module, determines the target allocation action from the extended action space based on the allocation strategy network and executes the dispatch, collects the escort order fulfillment data during the execution of the escort order, divides the escort order fulfillment data into controllable stage data and waiting stage data according to service events and controllable attributes, determines the reference determination module to determine the reference escort order set that did not use the same action from similar historical escort orders and determines the staged fulfillment reference value, determines the action evaluation value based on the deviation between the controllable stage data and the corresponding stage's fulfillment reference value, and updates the allocation strategy network based on the action evaluation value and controls the adoption ratio of collaborative allocation actions according to confidence level. The updated allocation strategy network is used for allocation decision at the next allocation moment, forming a closed loop of strategy iteration.

[0028] It is understandable that, since the action evaluation value is determined only based on data from a controllable phase, random fluctuations in the queuing situation at service locations are not included in the evaluation of allocation decisions. The allocation strategy network learns the service effect of the allocation decision itself, and the direction of strategy optimization is not affected by the randomness of queuing.

[0029] It is understood that the reinforcement learning-based intelligent allocation system for escort services proposed in this invention is mainly applied to the allocation of escort service orders on escort service platforms, such as the scheduling and allocation of escort orders for outpatient visits in general hospitals and specialized hospitals. It is particularly suitable for escort scenarios where waiting times for appointments and examinations are long. The technical approach of this invention can also be extended to other fields with similar needs for personnel scheduling, including the scheduling of caregiver work orders, the dispatch of home care service orders, and the scheduling of guidance personnel in government service halls, etc., where the fulfillment process involves queuing at the venue. These will not be elaborated further.

[0030] Specifically, the state construction module includes: The escort information submodule is used to determine the characteristics of escort orders based on service demand data, including service location, department type, service time period, user information, and the service items included in the escort order. The Skill Status submodule is used to determine personnel characteristics in order to assess the service capabilities and status of candidate escorts, including the candidate escort's skill parameters, real-time location, and committed service time. The event process submodule is used to determine the event process characteristics based on the expected arrival time distribution determined by the real-time progress data of each service process event within the service location, so as to determine the progress of the service process events associated with the escort order, including several expected arrival time distributions corresponding to each subsequent service event, wherein the expected arrival time distribution is determined based on the real-time progress data of each service process event within the service location.

[0031] Understandably, by determining the proficiency of candidate companions in each skill item through skill parameters, and by determining the committed service time based on the service time of candidate companions already occupied by other companion orders, the personnel characteristics simultaneously reflect the service capabilities and available service time of candidate companions. Based on this, the strategy decision module can allocate companion orders to candidate companions with matching skills and available time slots, avoiding service quality degradation due to insufficient skills and order interruption due to time slot conflicts, thus improving the reliability of allocation decisions.

[0032] Specifically, the strategy decision-making module, based on the allocation strategy network, determines the target allocation action from the extended action space according to the environmental state, including the following process: Based on the aforementioned personnel characteristics, determine the candidate assignment actions that each candidate escort can perform; The expected fulfillment time of each candidate allocation action is determined based on the event process characteristics; The value of each candidate assigned action in the extended action space is evaluated to determine the action value of each candidate assigned action, so as to determine the long-term benefit expected to be obtained after the candidate assigned action is selected. The greater the action value, the higher the expected action evaluation value after the corresponding candidate assigned action is selected. The target assignment action is determined based on the action value of each candidate assignment action and a preset random selection ratio. The extended action space includes whole-order assignment actions and collaborative allocation actions.

[0033] In this embodiment, the allocation strategy network evaluates each candidate allocation action based on the environmental state and outputs the action value of each candidate allocation action. The strategy decision module determines the target allocation action based on the action value of each candidate allocation action and a preset random selection ratio. The larger the random selection ratio, the greater the probability of randomly determining the target allocation action from each candidate allocation action. The smaller the random selection ratio, the more inclined it is to determine the candidate allocation action with the highest action value as the target allocation action. The random selection ratio decreases as the number of strategy iterations increases.

[0034] It is understood that the method for estimating the value of an action is not limited to the method given in this embodiment. Any method that can evaluate each candidate action based on the environmental state and output the corresponding value can be used. The value and adjustment method of the random selection ratio are determined according to the business scale, which will not be elaborated here.

[0035] It is understandable that collaborative allocation refers to the process of multiple companions working together to allocate the same companionship order. Collaborative allocation includes relay allocation and joint execution: relay allocation involves assigning the companionship order to different companions according to the service stage for sequential execution, while joint execution involves two companions jointly executing the same companionship order.

[0036] In this embodiment, the strategy decision module is used to determine whether to apply the relay allocation action to the corresponding escort order based on the waiting stage.

[0037] It is understandable that the specific form of collaborative allocation is not limited to the above forms. Allocation methods in which multiple escorts divide the work according to the service stage or jointly complete the same escort order all fall under the category of collaborative allocation, which will not be elaborated here.

[0038] Specifically, the process by which the fulfillment collection module collects the fulfillment data of the accompanying medical appointment orders includes: In response to the execution of the accompanying medical appointment order, collect the accompanying medical appointment order completion result, accompanying medical appointment order interruption event, and user evaluation data; Collect service process records from each accompanying medical staff member; If the target allocation action is a collaborative allocation action completed by multiple escorts for the same escort order, a handover record is generated at the handover point between adjacent service stages.

[0039] Specifically, the medical escort order fulfillment data is a collection of data generated throughout the entire process of a medical escort order, from allocation to completion, reflecting the execution status of the order. This data includes the order completion results, order interruption events, user evaluation data, and service process records of each accompanying person.

[0040] Specifically, the event stream parsing module includes: The phase segmentation module is used to segment the escort order fulfillment data into service phase data corresponding to each service event according to a preset event flow template; The attribute determination submodule is used to determine the controllability determination benchmark based on the service location and department type of the escort order, so as to distinguish whether the duration of the service stage can be changed by the escort's service behavior, and to divide the service stage data into controllable stage data and waiting stage data based on the controllability determination benchmark.

[0041] Please see Figure 2 As shown, this is a schematic diagram illustrating how the fulfillment data of a patient escort order is divided into controllable stage data and waiting stage data in an embodiment of the present invention.

[0042] The event flow parsing module responds to the completion of the current escort order. According to the event flow template, the escort order fulfillment data is divided along the time axis into service stage data corresponding to each service event. Based on the controllability judgment criteria, each service stage is divided into a controllable stage and a waiting stage. The waiting stage data includes waiting periods for consultation, examination queues, and report issuance; the controllable stage data includes periods for ticket retrieval, medical accompaniment, payment retrieval, examination accompaniment, and medication retrieval.

[0043] Specifically, the event flow template is a predefined sequence of service process events used to divide the fulfillment data of a patient escort order into service stage data corresponding to each service event. The service events in the event flow template are arranged in the order of the service process.

[0044] In this embodiment, the event flow template is configured according to the service process of the service location. Taking a general hospital outpatient clinic as an example, the service events in the event flow template include, in sequence, taking a number, waiting for a consultation, seeing a doctor, paying, examination, issuing a report, and picking up medication. For companion orders at specialized hospitals or with fewer examination items, some service events in the event flow template are deleted, which will not be elaborated here.

[0045] It is understandable that the service processes differ in different service locations. The specific content and order of the event flow template are related to the type of service location and the department settings. As long as it can cover the actual service process events of the escort order, it is sufficient. The event flow template can also be generated based on the service process records of historical escort orders, which will not be elaborated here.

[0046] Specifically, the controllability assessment criteria are determined based on the dominant factor of the service phase duration, which determines the controllability attributes of the service phase.

[0047] In this embodiment, the duration of the waiting period, examination queuing period, and report issuance waiting period is primarily determined by the queuing situation at the service location and is thus classified as a waiting period. The duration and effectiveness of the number retrieval, patient accompaniment, payment retrieval, examination accompaniment, and medication retrieval stages are primarily determined by the service behavior of the accompanying staff and are thus classified as controllable stages. The attribute determination submodule determines the current service stage of the accompanying order based on the number queuing progress and the queue length in the examination department, and determines the controllable attributes of this service stage according to the controllability determination criteria.

[0048] It is understandable that the specific content of the controllability judgment criteria is related to the business type of the service venue; controllability attributes can also be determined based on the duration statistics of each service stage in historical escort orders, which will not be elaborated here.

[0049] Specifically, the reference determination module includes: The similarity retrieval submodule is used to calculate the similarity between a medical companion order and each historical medical companion order based on the characteristics of the medical companion order, and to determine the set of similar medical companion orders; The reference filtering submodule is used to filter historical escort orders that did not use the same type of action as the target assigned action from the similar escort order set, and to determine the reference escort order set; The reference value determination submodule is used to perform statistical calculations based on the controllable stage data of the reference escort order set to determine the staged performance reference value.

[0050] Specifically, the reference medical escort order set is a collection of historical medical escort orders that are similar to the original medical escort order but did not employ the same type of action as the target assigned action. This set serves as a benchmark for determining the action evaluation value. The differences in medical escort order characteristics between the historical medical escort orders in the reference medical escort order set and the original medical escort order are within a preset range, ensuring the comparability of the controllable stage data between the two.

[0051] In this embodiment, the similarity retrieval submodule performs similarity calculations on the characteristics of escort orders and historical escort orders based on service location, department type, service time period, and user information. Historical escort orders with similarity scores higher than a preset similarity threshold are identified as a set of similar escort orders. The reference filtering submodule removes historical escort orders from the similar escort order set that used the same type of action as the target assignment action, resulting in a reference escort order set. For example, when the target assignment action is a collaborative assignment action, all historical escort orders in the reference escort order set are escort orders completed using the whole-order assignment action.

[0052] Understandably, the purpose of similarity calculation is to eliminate the influence of the difficulty of the accompanying medical order itself on the comparison results. The feature dimensions and calculation methods used for similarity calculation can be adjusted according to the business data. Alternatively, a propensity score matching method can be used to determine the set of similar accompanying medical orders, which will not be elaborated here.

[0053] Specifically, the similarity threshold is used to determine whether a historical escort order is similar to another escort order. When the similarity between a historical escort order and another escort order is higher than the similarity threshold, the historical escort order is included in the set of similar escort orders. The higher the value of the similarity threshold, the fewer historical escort orders are included in the set of similar escort orders, and the higher the degree of similarity between them.

[0054] In this embodiment, the similarity threshold is determined based on the size requirement of the similar medical companion order set. Each historical medical companion order is sorted by its similarity to other medical companion orders from highest to lowest. Based on the sorting results and the preset set size requirement, a similarity threshold is determined, ensuring that the number of historical medical companion orders with similarity higher than the similarity threshold meets the accumulation requirements of the reference sample. Specifically, the process of determining the phased performance reference values ​​in the reference value determination submodule includes: Based on the waiting stage data, the accompanying medical order is aligned with each historical accompanying medical order in the reference accompanying medical order set according to the service event stage; Extract the controllable stage data of each historical escort order after alignment according to the service stage, perform the statistical calculation, and determine the staged performance reference value; The waiting stage data is used for stage alignment and is not included in the statistical calculation.

[0055] Please see Figure 3 As shown, it is a logic judgment diagram for determining the action evaluation value in an embodiment of the present invention, including: Calculate the difference between the data of each service stage in the controllable stage data and the corresponding performance reference value of the stage, and determine the stage deviation value of each service stage; If the target assignment action is the complete assignment action of a single escort to perform the entire escort order, then the sum of the stage deviation values ​​of each service stage is determined as the action evaluation value, so as to quantify the actual contribution of the target assignment action to the fulfillment result of the escort order. If the target allocation action is a collaborative allocation action in which multiple escorts work together to complete the same escort order, then according to the service stage each escort is responsible for, the stage deviation value of the corresponding service stage is respectively assigned to each escort, and the action evaluation value is determined based on the corresponding results.

[0056] Specifically, the policy update module includes: The network update submodule is used to update the network parameters of the allocation strategy network according to the action evaluation value; The confidence determination submodule is used to determine the confidence level of the action evaluation value based on the sample size of the reference escort order set; A proportional control submodule is used to control the adoption ratio of collaborative allocation actions according to the confidence level; wherein the adoption ratio of the collaborative allocation actions increases as the confidence level increases.

[0057] Please see Figure 4 As shown, this is a logic judgment diagram for controlling the proportion of collaborative allocation actions in an embodiment of the present invention. The proportion control submodule obtains the confidence level of the action evaluation value and compares the confidence level with a preset confidence threshold. If the confidence level is lower than the confidence threshold, the proportion of collaborative allocation actions is restricted to a preset exploration proportion. If the confidence level is not lower than the confidence threshold, the restriction on the proportion of actions is lifted.

[0058] Specifically, the confidence threshold is a preset threshold used to determine whether the credibility of the action evaluation value meets the requirements for scaling up. The exploration ratio is the upper limit of the adoption ratio of collaboratively assigned actions when the confidence level is lower than the confidence threshold. When the confidence level is lower than the confidence threshold, it indicates that the sample size of the reference escort order set is insufficient to support the credible application of the action evaluation value, and the collaboratively assigned actions are only allocated for confirmatory purposes based on the order volume corresponding to the exploration ratio. When the confidence level is not lower than the confidence threshold, the restriction on the adoption ratio is lifted.

[0059] In this embodiment, the strategy update module also includes a threshold confirmation submodule, which is used to determine a confidence threshold based on the tolerance of the escort platform for allocation risk. The lower the tolerance of the escort platform for allocation risk, the larger the value of the confidence threshold.

[0060] Specifically, the confidence determination submodule determines the confidence level of the action evaluation value based on the sample size of the reference escort order set, including: Count the number of reference samples corresponding to each service stage in the aforementioned set of reference escort orders; The number of reference samples corresponding to each service stage is compared with the preset sample size benchmark to determine the sub-confidence of each service stage, so as to determine the statistical sufficiency of the reference samples of a single service stage. The larger the number of reference samples, the smaller the statistical fluctuation of the stage deviation value. The confidence level of the action evaluation value is determined based on each of the sub-confidence levels.

[0061] Specifically, sub-confidence is a parameter that reflects the statistical sufficiency of the reference sample for a single service stage. Sub-confidence is determined separately for each service stage and is used to determine the confidence level of the action evaluation value.

[0062] It is understood that the mapping relationship between the sub-confidence and the number of reference samples, as described above, can also be determined using piecewise mapping or a smoothing function, which will not be elaborated further here.

[0063] In this embodiment, the confidence determination submodule determines the minimum value among the sub-confidences of each service stage as the confidence of the action evaluation value; when the target assigned action corresponds to only a single service stage, the sub-confidence of that service stage is determined as the confidence of the action evaluation value.

[0064] Specifically, the sample size benchmark is used to measure the statistical sufficiency of the reference order set. A larger sample size benchmark value requires a larger number of reference samples to determine the sub-confidence level. The sample size benchmark is determined during the system trial operation phase: based on the variance of the performance data collected during the trial operation phase for each service phase, the sample size benchmark corresponding to each service phase is determined; the larger the variance, the larger the sample size benchmark value. The sample size benchmark is configured separately according to service location and department type, and is updated periodically as order data accumulates after the system is officially launched.

[0065] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A smart order allocation system for escort services based on reinforcement learning, characterized in that, include: The state construction module is used to obtain the characteristics of the accompanying medical order, the personnel characteristics of the candidate accompanying medical staff, and the event process characteristics of the service location in order to construct the environmental state. The strategy decision module is used to determine the target allocation action from the extended action space based on the allocation strategy network and the environmental state. The fulfillment data collection module is used to collect the fulfillment data of the accompanying medical appointment orders. The event stream parsing module is used to divide the escort order fulfillment data into controllable stage data and waiting stage data according to the service process event progress data of the service venue and the service event and controllable attributes. The reference determination module is used to determine a set of reference escort orders that did not use the same action as the target allocation action from historical escort orders similar to the escort order, and to determine a phased performance reference value based on the controllable stage data of the reference escort order set. The evaluation and determination module is used to determine the action evaluation value based on the deviation between the controllable stage data and the corresponding performance reference value. The strategy update module is used to update the allocation strategy network based on the action evaluation value, determine the confidence level of the action evaluation value, and control the adoption ratio of collaboratively allocated actions according to the confidence level.

2. The intelligent order allocation system for escort services based on reinforcement learning according to claim 1, characterized in that, The state construction module includes: The accompanying medical information submodule is used to determine the characteristics of accompanying medical orders based on data on service demand content; The skill status submodule is used to obtain the skill parameters, real-time location, and committed service time of each candidate escort in order to determine the characteristics of the personnel; The event process submodule is used to obtain real-time progress data of each service process event within the service location, determine the expected arrival time distribution of subsequent service events for the escort order, and thus determine the event process characteristics.

3. The intelligent order allocation system for escort services based on reinforcement learning according to claim 1, characterized in that, The strategy decision-making module, based on the allocation strategy network, determines the target allocation action from the extended action space according to the environmental state, including the following process: Based on the aforementioned personnel characteristics, determine the candidate assignment actions that each candidate escort can perform; The expected fulfillment time of each candidate allocation action is determined based on the event process characteristics; The value of each candidate assigned action in the extended action space is evaluated to determine the action value of each candidate assigned action. The target assignment action is determined based on the action value of each candidate assignment action and a preset random selection ratio. The extended action space includes whole-order assignment actions and collaborative allocation actions.

4. The intelligent order allocation system for escort services based on reinforcement learning according to claim 1, characterized in that, The process by which the fulfillment collection module collects the fulfillment data of the accompanying medical appointment order includes: In response to the execution of the accompanying medical appointment order, collect the accompanying medical appointment order completion result, accompanying medical appointment order interruption event, and user evaluation data; Collect service process records from each accompanying medical staff member; If the target allocation action is a collaborative allocation action completed by multiple escorts for the same escort order, a handover record is generated at the handover point between adjacent service stages.

5. The intelligent order allocation system for escort services based on reinforcement learning according to claim 1, characterized in that, The event stream parsing module includes: The phase segmentation module is used to segment the escort order fulfillment data into service phase data corresponding to each service event according to a preset event flow template; The attribute determination submodule is used to distinguish the service stage data into controllable stage data and waiting stage data based on the controllability determination criteria.

6. The intelligent order allocation system for escort services based on reinforcement learning according to claim 1, characterized in that, The reference determination module includes: The similarity retrieval submodule is used to calculate the similarity between the accompanying medical order and each historical accompanying medical order based on the accompanying medical order characteristics, and to determine the set of similar accompanying medical orders; The reference filtering submodule is used to filter historical escort orders that did not use the same type of action as the target allocation action from the similar escort order set, and to determine the reference escort order set; The reference value determination submodule is used to perform statistical calculations based on the controllable stage data of the reference escort order set to determine the phased performance reference value.

7. The intelligent order allocation system for escort services based on reinforcement learning according to claim 6, characterized in that, The process by which the reference value determination submodule determines the phased performance reference value includes: Based on the waiting stage data, the accompanying medical order is aligned with each historical accompanying medical order in the reference accompanying medical order set according to the service event stage; Extract the controllable stage data of each historical escort order after alignment according to the service stage, perform the statistical calculation, and determine the staged performance reference value; The waiting stage data is used for stage alignment and is not included in the statistical calculation.

8. The intelligent order allocation system for escort services based on reinforcement learning according to claim 1, characterized in that, The process by which the evaluation determination module determines the action evaluation value based on the deviation between the controllable stage data and the corresponding performance reference value includes: Calculate the difference between the data of each service stage in the controllable stage data and the corresponding performance reference value of the stage, and determine the stage deviation value of each service stage; If the target assignment action is a complete order assignment action performed by a single escort, then the sum of the stage deviation values ​​of each service stage is determined as the action evaluation value. If the target allocation action is a collaborative allocation action in which multiple escorts work together to complete the same escort order, then according to the service stage each escort is responsible for, the stage deviation value of the corresponding service stage is respectively assigned to each escort, and the action evaluation value is determined based on the corresponding results.

9. The intelligent order allocation system for escort services based on reinforcement learning according to claim 1, characterized in that, The policy update module includes: The network update submodule is used to update the network parameters of the allocation strategy network according to the action evaluation value; The confidence determination submodule is used to determine the confidence level of the action evaluation value based on the sample size of the reference escort order set; A proportional control submodule, which controls the adoption ratio of the collaboratively allocated actions according to the confidence level; The proportion of the collaborative allocation action adopted increases with the increase of the confidence level.

10. The intelligent order allocation system for escort services based on reinforcement learning according to claim 9, characterized in that, The confidence determination submodule determines the confidence level of the action evaluation value based on the sample size of the reference escort order set, including the following process: Count the number of reference samples corresponding to each service stage in the aforementioned set of reference escort orders; The number of reference samples corresponding to each service stage is compared with the preset sample size benchmark to determine the sub-confidence of each service stage. The confidence level of the action evaluation value is determined based on each of the sub-confidence levels.

Images