A GSP medicine operation whole-process virtual simulation system based on human-computer interaction

Abstract

The application belongs to the intersection of medical education technology and GSP medicine management informationization, and provides a GSP medicine management whole-process virtual simulation system based on human-computer interaction, which comprises a virtual simulation engine end, an interactive device, a voice input device, an analysis module and a guiding module; the virtual simulation engine end is used for realistically presenting a medical simulation working scene; the interactive device is used for capturing the current action of a practical training personnel and acquiring the operation content of the practical training personnel in reality according to the current practical training process in which the practical training personnel is located; the voice input device is used for receiving the voice input of the practical training personnel in the current practical training process; the analysis module is used for performing interactive logic judgment on the operation content and the voice input and performing behavior feedback, determining the behavior correct guidance of the practical training personnel in the current practical training process, and generating the behavior guidance strategy of the current practical training process; and the guiding module is used for continuously guiding the practical training personnel according to the behavior guidance strategy. The immersion learning improves the practical training effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical education and GSP medicine management informationization, and proposes a GSP medicine management whole-process virtual simulation system based on human-computer interaction. BACKGROUND

[0002] At present, in the field of GSP simulation training systems, the main product form is a GSP medicine management virtual simulation training system. This product mainly constructs a medicine circulation enterprise scene in a virtual environment through 3D virtual simulation technology, and completes related enterprise cognition, enterprise work process training, work task simulation examination, etc. on a computer. However, this system relies on keyboard, mouse or simple touch screen operation, that is, two-dimensional operation, and cannot simulate three-dimensional space operation in a real medicine management scene. The existing VR system only supports handle operation, which is essentially different from the actual medicine handling, document transfer and other delicate actions. Moreover, it lacks simulation of key communication scenes in medicine management (such as prescription medicine sales dialogue and medicine procurement contract negotiation), has the defects of insufficient immersion and single interaction mode, makes it difficult for users to enter a learning state, and reduces the learning training effect. It is difficult to support the work training of posts that require communication and operation.

[0003] Therefore, the present application provides a GSP medicine management whole-process virtual simulation system based on human-computer interaction. SUMMARY

[0004] The present application provides a GSP medicine management whole-process virtual simulation system based on human-computer interaction, which is used to realize three-dimensional scene display through a virtual simulation engine, and then capture and analyze gesture interaction and voice interaction in the scene to train the GSP whole process, so as to improve the training effect through immersive learning.

[0005] The present application provides a GSP medicine management whole-process virtual simulation system based on human-computer interaction, which is used to realize three-dimensional scene display through a virtual simulation engine, and then capture and analyze gesture interaction and voice interaction in the scene to train the GSP whole process, so as to improve the training effect through immersive learning.

[0006] A virtual simulation engine end is used to realize real display of a medical simulation work scene, wherein the medical simulation work scene is related to each training process in a medical training whole process;

[0007] An interaction device is used to capture the current action of a training personnel in the real displayed medical simulation work scene and acquire the operation content of the training personnel in reality according to the current training process in which the training personnel is located;

[0008] A voice input device is used to receive voice input of the training personnel in the current training process;

[0009] The analysis module is configured to determine the behavior of the real training personnel in the current real training process and to generate a behavior guidance strategy for the current real training process.

[0010] The guidance module is configured to guide the real training personnel according to the behavior guidance strategy until the behavior of the real training personnel in each real training process is standardized.

[0011] Preferably, the interactive device comprises:

[0012] The acquisition unit is configured to acquire real-time skeletal key point data of the current action of the human body, wherein the skeletal key point data comprises coordinate information of 27 skeletal key points of the human hand in a three-dimensional space.

[0013] The model establishment unit is configured to convert the rotation information of each skeletal key point into quaternion representation according to the acquired skeletal key point data, and to establish a skeletal key point quaternion motion model of the current action by combining the translation coordinates, so as to describe the posture and position of each skeletal key point in the three-dimensional space.

[0014] The feature extraction unit is configured to continuously acquire the quaternion motion model of a plurality of action frames to form action sequence data, and to input the action sequence data into a pre-trained LSTM network to extract time-series action features in the action sequence, including the order, duration and rhythm change time-dependent information of the action.

[0015] The space mapping unit is configured to map the skeletal key point information of the human body action in the real world to a virtual character in a virtual space according to the extracted time-series action features and the quaternion motion model.

[0016] Preferably, the interactive device further comprises:

[0017] The locking unit is configured to sequentially lock the position points of each skeletal key point in the continuous action frames of the virtual character, and to construct a movement vector Yx={w t1i ,i=1,2,3,...,Tn} of the corresponding skeletal key point and an action gesture Ds={r t2j ,j=1,2,3,...,27} of each action frame, wherein w t1i represents the position coordinates of the corresponding skeletal key point at time t1i in the i-th action frame; and r t2j represents the position coordinates of the j-th skeletal key point at time t2j in the corresponding action frame.

[0018] The point set construction unit is configured to determine a standard vector of each key point of the skeleton and a training trigger point set according to a standard operation behavior of a simulation work scene constructed based on a simulation training task of a current training process, wherein the training trigger point set comprises a trigger position point, a trigger duration and a trigger sequence of a training trigger point;

[0019] The matching unit is configured to match each movement vector with a corresponding standard vector once, and sequentially match each movement vector and each action frame with the training trigger point set twice.

[0020] The content determination unit is configured to determine an operation content of the training personnel according to a first matching point of each action frame filtered according to the matching result once and a second matching point of each action frame filtered according to the matching result twice.

[0021] Preferably, the analysis module comprises:

[0022] The first comparison unit is configured to determine a gesture interaction feedback result of the virtual character to the corresponding simulation work scene according to the operation content, and compare and analyze the gesture interaction feedback result with a gesture theoretical feedback result of the corresponding training simulation task to determine whether a gesture feedback difference between the two satisfies a first set standard.

[0023] If the first set standard is satisfied, a satisfaction coefficient of 1 is assigned to the corresponding operation content.

[0024] Otherwise, a1 is assigned to the corresponding operation content according to a close relationship between the gesture feedback difference and the first set standard, wherein a1 is a satisfaction coefficient based on gestures.

[0025] The second comparison unit is configured to determine a voice interaction feedback result to the corresponding simulation work scene according to the voice input, and compare and analyze the voice interaction feedback result with a voice theoretical feedback result of the corresponding training simulation task to determine whether a voice feedback difference between the two satisfies a second set standard.

[0026] If the second set standard is satisfied, a satisfaction coefficient of 1 is assigned to the corresponding voice input.

[0027] Otherwise, a2 is assigned to the corresponding voice input according to a close relationship between the voice feedback difference and the second set standard, wherein a2 is a satisfaction coefficient based on voice.

[0028] The coefficient comparison unit is configured to compare the two satisfaction coefficients, and take the interaction mode with the larger satisfaction coefficient as the main interaction logic judgment basis, and take the interaction mode with the smaller satisfaction coefficient as the auxiliary interaction logic judgment basis.

[0029] If the two satisfaction coefficients are equal, both interaction modes are taken as the main interaction logic judgment basis.

[0030] Preferably, the analysis module further comprises:

[0031] a weight determination unit configured to assign a first weight to the gesture interaction and a second weight to the voice interaction according to the interaction logic determination result;

[0032] a gesture guiding unit configured to determine a behavior error starting point according to the gesture interaction feedback difference and the behavior difference determined based on the first matching point and the second matching point, and determine gesture guidance with the behavior error starting point as a starting point in dependence on the training trigger point set;

[0033] a voice guiding unit configured to determine correct voice from a voice scene interaction database under a corresponding practical training process according to the voice interaction feedback difference, and perform voice guidance;

[0034] a combination unit configured to obtain behavior correct guidance of the practical training personnel in the current practical training process according to the gesture guidance and the voice guidance, and in combination with the first weight and the second weight.

[0035] Preferably, the analysis module further comprises:

[0036] a delay determination unit configured to collect an initial collection time of a test voice signal collected by each array unit, and obtain a propagation delay time of a corresponding array unit according to a distance between the specified position and each array unit and in combination with an initial sound emission time of the test voice signal;

[0037] an acoustic determination unit configured to determine acoustic characteristics according to sound information of the test voice signal collected by each array unit;

[0038] a vector analysis unit configured to mark the propagation delay time and the acoustic characteristics on the corresponding array unit of the array structure to obtain an analysis group vector, and input the analysis group vector into a vector analysis model to determine a priority of each array unit, wherein the priority comprises a first priority and a second priority, and the first priority is higher than the second priority;

[0039] a screening unit configured to perform N1 times of voice test on the voice input device, count a first number of array units set with the first priority, and regard a unit with a ratio of the first number to N1 greater than a preset threshold as a priority unit;

[0040] wherein, when collecting voice input, voice interaction analysis is performed based on voice signals collected by the priority unit.

[0041] Preferably, the analysis module further comprises:

[0042] The probability determination unit is configured to synchronize and split the gesture guidance and the voice guidance in the correct behavior guidance according to a test point list under a corresponding practical training process, and predict a guidance success probability of each synchronized split block in combination with historical guidance behavior characteristics of the practical training personnel.

[0043] The score determination unit is configured to determine a first score of each synchronized split block according to the guidance success probability of each synchronized split block.

[0044] The strategy determination unit is configured to determine a recommended guidance frequency of the corresponding synchronized split block from a score-frequency database according to the first score of each synchronized split block, so as to obtain the behavior guidance strategy.

[0045] Preferably, the matching unit comprises:

[0046] The overlapping segment determination subunit is configured to determine a relative overlapping segment of a moving track of the corresponding moving vector and a standard track of the standard vector.

[0047] The judgment subunit is configured to judge whether the relative overlapping segment contains a key interaction point in the standard vector.

[0048] If the relative overlapping segment contains the key interaction point, the moving vector is reserved based on a partial vector under the relative overlapping segment.

[0049] If the relative overlapping segment does not contain the key interaction point, it is determined that the first matching point does not exist in the corresponding matching result.

[0050] The partial vector is reserved as the matching result.

[0051] Compared with the prior art, the application has the following beneficial effects:

[0052] The three-dimensional scene is presented through the virtual simulation engine, and the GSP whole process is trained through the capture and analysis of the gesture interaction and the voice interaction in the scene, so that the immersion learning improves the training effect.

[0053] Other features and advantages of the present application will be further described in the following description, and some will become apparent from the description, or will be learned by practice of the present application. The purpose and other advantages of the present application can be achieved and obtained by the structure specifically pointed out in the written description and the accompanying drawings.

[0054] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, and are used to explain the present application together with embodiments of the present application, and do not constitute a limitation to the present application. In the drawings:

[0056] Figure 1 This is a structural diagram of a virtual simulation system for the entire GSP (Good Supply Practice) drug management process based on human-computer interaction, as described in an embodiment of the present invention.

[0057] Figure 2 This is a domain knowledge graph in an embodiment of the present invention;

[0058] Figure 3 This is a code diagram for dialogue state tracking in an embodiment of the present invention;

[0059] Figure 4 This is a flowchart of the virtual simulation training for GSP drug management in an embodiment of the present invention. Detailed Implementation

[0060] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0061] This invention proposes a virtual simulation system for the entire GSP (Good Supply Practice) drug management process based on human-computer interaction, such as... Figure 1 As shown, it includes:

[0062] The virtual simulation engine is used to realistically represent medical simulation work scenarios, which are related to each training process in the entire medical training process.

[0063] An interactive device is used to capture the trainee's current actions and acquire the trainee's actual operation content based on the current training process when the trainee is in a realistically presented medical simulation work scenario.

[0064] A voice input device is used to receive voice input from the trainees during the current training process.

[0065] The analysis module is used to perform interactive logic judgment on the operation content and voice input and provide behavioral feedback to determine the correct behavioral guidance for the trainee in the current training process and generate behavioral guidance strategy for the current training process.

[0066] The guidance module is used to continuously guide the trainees based on the behavior guidance strategy until the trainees have mastered the behavioral operation specifications for each training simulation task.

[0067] In this embodiment, 3D modeling technology is used to create virtual pharmacies, warehouses, distribution centers, and other scenarios, which highly restore the real environment and simulate various equipment and systems required by GSP, such as temperature and humidity monitoring systems, electronic regulatory code scanning equipment, and refrigeration equipment. Standardized operating procedures are designed according to GSP terms, and each procedure corresponds to a specific training scenario.

[0068] The virtual drug management and operation scenario created based on the GSP specification includes simulation scenarios of drug procurement, acceptance, storage, maintenance, sales, warehouse-out review, transportation and the like.

[0069] In this embodiment, the training process is a specific operation link in the GSP process, such as a drug warehouse-in acceptance process, a temperature and humidity monitoring process, an unqualified drug handling process and the like.

[0070] In this embodiment, the next action is a specific operation action of the training personnel in the virtual GSP scenario, such as scanning a code, opening a box, checking a drug appearance, recording data and the like.

[0071] The operation content is an operation behavior in accordance with the GSP specification parsed from the action, such as checking a drug expiration date, checking a drug quantity, recording temperature and humidity data and the like, and the voice input is an instruction, inquiry or report in accordance with the GSP specification issued by the training personnel in the operation process, such as a drug batch number, whether the temperature and humidity are up to the standard, whether this batch of drugs needs to be rejected and the like.

[0072] The interaction logic judgment is to analyze which of the operation content and the voice input of the student is more important according to the GSP specification, and the behavior feedback is to provide instant compliance feedback for the operation of the student, such as correct operation, please check the drug expiration date and the like.

[0073] The behavior correct guidance is to provide specific correction suggestions and operation guidelines for the operation that does not conform to the GSP specification, and the behavior guidance strategy is to develop a personalized training plan and guidance mode according to the operation performance of the student.

[0074] The training simulation task is a specific operation task designed based on the GSP process, such as completing the warehouse-in acceptance of a batch of cold storage drugs, handling a piece of unqualified drug return and the like, and the behavior operation specification is a standard operation process and method in accordance with the requirements of the GSP clauses.

[0075] In this embodiment, the domain knowledge graph is constructed as shown in Figure 2 .

[0076] In this embodiment, the BERT+CRF model is used to realize the dialogue state tracking, to grab the keywords in the dialogue, to complete the dialogue voice analysis and feedback, as shown in Figure 3 .

[0077] In this embodiment, for the typical work flow:

[0078] Drug warehouse-in receipt:

[0079] (1) Gesture interaction:

[0080] Single-handed forward stretch simulation receives the action of the accompanying information; the right-hand index finger clicks the virtual acceptance button; the five fingers are clenched to simulate the action of taking the medicine box; the hand rotates and slides to simulate the action of rotating the medicine box for inspection, etc.

[0081] (2) Voice interaction:

[0082] In the goods receiving link, necessary verbal communication is simulated with the drug transporter and the drug acceptance person, such as:

[0083] Transporter: Hello, this is the accompanying information of this batch of drugs, please check.

[0084] User: OK, I will handle it immediately.

[0085] Drug sales:

[0086] (1) Voice interaction

[0087] In the three-dimensional virtual simulation scenario, an aunt enters the drugstore, and the system simulates the reception service of the clerk.

[0088] The user asks: "Aunt, do you need any help?"

[0089] The virtual character answers: "Hello, I have a cold and want to buy some medicine";

[0090] The user asks: "Please describe your symptoms in detail";

[0091] The virtual character answers: "I have a cough and runny nose since yesterday, and today I feel a little worse, and I also have a headache, but no fever".

[0092] (2) Gesture interaction

[0093] When the prescription medicine needs to be taken, the arm is stretched forward to receive the prescription, and the hand gesture is used to complete the prescription signature.

[0094] In this embodiment, the above-mentioned scheme can also be implemented in the manner as shown in Figure 4 .

[0095] The beneficial effects of the above technical scheme are: through the three-dimensional scene presentation of the virtual simulation engine end, and then through the capture and analysis of the gesture interaction and voice interaction in the scene, the GSP full process is trained, and the immersion learning improves the training effect.

[0096] The application provides a GSP full process virtual-real fusion simulation system based on multi-modal human-computer interaction, and the interactive device comprises:

[0097] The collection unit is used for collecting the bone key point data of the current action of the human body in real time, and the bone key point data includes coordinate information of 27 bone key points of a hand of the human body in a three-dimensional space;

[0098] The model establishing unit is used for converting rotation information of each bone key point into quaternion representation according to the collected bone key point data, combining translation coordinates, establishing a quaternion motion model of the bone key points of the current action, and describing the posture and position of each bone key point in the three-dimensional space.

[0099] The feature extraction unit is used for continuously acquiring the quaternion motion model of a plurality of action frames, forming action sequence data, inputting the action sequence data into a pre-trained LSTM network, and extracting time sequence action features in the action sequence, including the sequence, duration and rhythm change of the action.

[0100] The space mapping unit is used for mapping the bone key point information of the human body action in the real world to a virtual character in a virtual space according to the extracted time sequence action features and the quaternion motion model.

[0101] In this embodiment, the code for mapping the virtual and real spaces is as follows:

[0102] python

[0103] def coordinate_mapping(real_pos):

[0104] # nonlinear calibration formula

[0105] virtual_x = a * real_x^3 + b * real_x^2 + c * real_x

[0106] virtual_y = d * log(real_y) + e

[0107] return(virtual_x, virtual_y).

[0108] In this embodiment, the 27 bone key points of the hand include accurate coordinate points of the wrist, palm, finger joints and other parts, which are used to completely describe the hand action, and the three-dimensional space coordinate information is the accurate position of each key point on the X, Y and Z axes, usually in millimeters, which is measured by an accelerometer and a gyroscope. For example, in the medicine acceptance scenario, the system collects the hand action of the student in real time, identifies the fine action of picking up the medicine box and rotating the wrist to check the label, and captures the hand posture and movement trajectory of the student when operating the electronic supervision code scanning device in the GSP training.

[0109] In this embodiment, the rotation information is the orientation and angular change of the skeletal key points in three-dimensional space, the quaternion representation represents a three-dimensional rotation with four numbers (w, x, y, z), avoiding the gimbal lock problem of Euler angles, the translation coordinates are the position movements of the skeletal key points in space, and the quaternion motion model is a mathematical model that integrates rotation and position information, accurately describing skeletal motion. For example, in the drug storage operation, the system converts the rotation action of the student's hand (such as twisting the bottle cap) into a quaternion representation, ensuring that the virtual character's actions are natural and smooth. When simulating cold storage equipment operation, the force and angle of the fingers pressing the buttons are accurately captured and reproduced in the virtual environment through the quaternion model.

[0110] In this embodiment, the action sequence data is a set of quaternion motion models for multiple consecutive time frames, and the time sequence action features are the time-dependent relationships of actions, such as order, duration, rhythm change, etc. In the drug maintenance operation, the system analyzes the action sequence of the student checking the appearance of the drug through the LSTM network to determine whether it is performed in the order of "one look, two smell, three shake" as specified by GSP. In the cold chain drug transportation scenario, it is identified whether the time interval of the student operating the temperature recorder meets the requirement of recording once every 30 minutes.

[0111] In this embodiment, the virtual space is a computer-generated three-dimensional environment, such as a virtual drugstore or warehouse, and the virtual character is a digital avatar representing the student in the virtual space. The mapping is achieved through:

[0112] Action reorientation technology: adapting captured human actions to the skeletal structure of the virtual character.

[0113] IK (Inverse Kinematics) algorithm: calculating the pose of the entire skeletal chain based on the position of the end joints (such as fingers).

[0114] Physical simulation: adding factors such as gravity and inertia to make virtual actions more realistic.

[0115] For example, in the virtual warehouse scenario, the student's actual moving actions are mapped to the virtual character, which synchronously completes the operations of moving drug boxes and placing them on shelves.

[0116] In the simulation of the drug dispensing process in the pharmacy, the student's hand actions (such as picking up the medicine bottle, pouring out the pills, and filling out the label) are accurately mapped to the virtual character, ensuring operation standardization and accuracy.

[0117] The beneficial effects of the above technical solutions are: quaternion representation and LSTM time sequence analysis ensure that the virtual character's actions are natural and coherent, through time sequence feature analysis, it is determined whether the student's operation meets the GSP process and time requirements, the virtual character is synchronized with the student's actions, enhancing the realism and participation of training, and facilitating the analysis of abnormal behaviors to provide a basis for subsequent guidance.

[0118] The application provides a GSP medicine operation whole-process virtual simulation system based on human-computer interaction, and the interaction device further comprises:

[0119] A locking unit is configured to sequentially lock position points of each skeletal key point in a continuous action frame of a virtual role, and construct a moving vector Yx={wi,ti} of the corresponding skeletal key point, wherein i=1, 2, 3,..., Tn and Ds={rj,tj} of each action frame, wherein j=1, 2, 3,..., 27. t1i t2j wi represents a position coordinate of the corresponding skeletal key point at a time t1i in the i-th action frame; and rj represents a position coordinate of the j-th skeletal key point at a time t2j in the corresponding action frame. t1i t2j

[0120] A point set construction unit is configured to determine a standard vector of each skeletal key point and a training trigger point set according to a standard operation behavior of a simulation work scene constructed based on a simulation training task of a current real training process, wherein the training trigger point set comprises a trigger position point, a trigger time length and a trigger sequence of a training trigger point.

[0121] A matching unit is configured to perform a first matching of each moving vector with a corresponding standard vector, and perform a second matching of each moving vector and each action gesture of each action frame with the training trigger point set.

[0122] A content determination unit is configured to determine an operation content of a real training personnel according to a first matching point in each action frame screened according to a first matching result and a second matching point of each action frame screened according to a second matching result.

[0123] The matching unit comprises:

[0124] A superposition segment determination subunit is configured to determine a relative superposition segment of a moving track of a corresponding moving vector and a standard track of a standard vector.

[0125] A judgment subunit is configured to judge whether the relative superposition segment contains a key interaction point in the standard vector.

[0126] If the relative superposition segment contains the key interaction point, the moving vector is reserved based on a partial vector in the relative superposition segment.

[0127] If the relative superposition segment does not contain the key interaction point, it is determined that there is no first matching point in the corresponding first matching result.

[0128] The reserved partial vector is taken as the first matching result.

[0129] ​​​In this embodiment, the action frame is a single frame of action data in a continuous time sequence, such as one frame in 30 frames per second, the skeleton key point position point is the coordinates (x, y, z) of the wrist joint in three-dimensional space, and the movement vector is the displacement vector of the skeleton key point between adjacent action frames, for example, the vector from (x1, y1, z1) to (x2, y2, z2). For example, in the medicine code scanning scenario, the locking unit tracks the continuous action frame of the index finger pressing the code scanning gun trigger, and calculates the movement vector of the finger joint.

[0130] In this embodiment, the action gesture is a posture formed by a combination of multiple skeleton key points, such as a pinch gesture determined by the relative positions of the thumb, index finger, and middle finger.

[0131] In this embodiment, the standard operation behavior is a sequence of actions required by the GSP specification, such as checking the appearance of the medicine for 5 seconds during acceptance, and the standard vector is a skeleton movement path that meets the specification, for example, the code scanning gun should move at a uniform speed along the medicine barcode.

[0132] The training trigger point set is a key node in the operation process, for example:

[0133] Trigger position point: the contact point of the code scanning gun and the medicine.

[0134] Trigger duration: the code scanning action should last more than 0.5 seconds.

[0135] Trigger sequence: first scan the code, then check the information, and finally record.

[0136] In this embodiment, the relative overlap section is the overlapping part of the student's action trajectory and the standard trajectory, for example, the percentage of the overlap of the student's actual code scanning path and the standard path, and the relative refers to the allowed deviation range of each trajectory point within ±5mm, as long as it is within the range, it is considered that the points coincide.

[0137] The key interaction point is a key position in the operation, such as the starting position of the code scanning gun and the center point of the medicine label.

[0138] In this embodiment, the first matching point is the matching position of the action trajectory and the standard trajectory, such as the starting point of the code scanning, and the second matching point is the matching result of the action and the trigger point set, such as completing the code scanning within the specified time, and the operation content is the specific behavior analyzed from the comprehensive matching result, such as having completed the code scanning operation of medicine A. For example, according to the first matching point (code scanning trajectory) and the second matching point (triggering the trigger action of the code scanning gun), it is determined that the student has performed the medicine code scanning operation. In the unqualified medicine handling scenario, by matching the gesture of filling out the return order and the trajectory of putting the medicine into the return area, it is confirmed that the operation is completed.

[0139] In this embodiment, the process of secondary matching is as follows:

[0140] Mobile Vector Standardization: Convert raw mobile vectors into unit vectors and normalize their lengths to eliminate force differences.

[0141] Gesture Feature Quantization: Convert motion gestures into feature vectors, such as the relative angle between fingers, the distance ratio between key points, and the orientation and rotation angle of the palm.

[0142] Trigger Point Set Preprocessing: Define the spatial range for each trigger point (e.g., a 5mm sphere centered on the standard position), set the time window (e.g., the trigger point must be activated within 0.5 seconds), and mark the trigger priority (e.g., mandatory trigger points and optional trigger points).

[0143] For example, in the medicine scanning code scenario, the movement trajectory of the scanning code gun is converted into a standardized vector sequence, and the features of the gesture holding the scanning code gun are extracted (e.g., the angle between the thumb and index finger is 45°±10°).

[0144] Spatial Dimension Matching:

[0145] Trigger Area Detection: Determine whether the endpoint of the mobile vector falls within the spatial range of the trigger point.

[0146] Trajectory Overlap Calculation: Use the Dynamic Time Warping (DTW) algorithm to calculate the similarity between the actual trajectory and the standard trajectory, and calculate the Hausdorff distance between the trajectories to evaluate the degree of deviation.

[0147] Spatial Relationship Verification: Check the relative position relationship between multiple key points, such as whether the vertical distance between the scanning code gun and the medicine label is within the range of 5-10mm, and whether the fingers are correctly placed on the trigger position of the scanning code gun.

[0148] Temporal Dimension Matching:

[0149] Temporal Alignment: Align the action sequence of the student with the standard temporal graph to solve the speed difference problem.

[0150] Duration Verification: Check whether the action stays in the trigger point area for the required time, such as:

[0151] The scanning code action needs to last for 0.5-1.0 seconds, and the medicine inspection needs to be maintained for more than 3 seconds.

[0152] Temporal Logic Check: Verify whether the activation order of the trigger points is correct, such as first scanning the code, then checking the information, and for cold medicine, first detecting the temperature, then entering the warehouse.

[0153] For example, the system records the time sequence of the student from picking up the scanning code gun to completing the scanning code, and judges whether all necessary operations are completed within the specified time.

[0154] Gesture and Action Type Matching:

[0155] Gesture Template Matching: Matching the current gesture with a predefined template library, e.g.:

[0156] Grabbing Gesture: Distance between thumb and index finger is less than 2 cm;

[0157] Rotating Gesture: Change in palm rotation angle exceeds 90°;

[0158] Action Semantic Analysis: Inferring action type in context, e.g.:

[0159] In an acceptance scenario, picking up a medicine followed by a flipping action is interpreted as checking the medicine's appearance;

[0160] Confidence Score Calculation: Assigning a confidence score to each matching result, e.g.:

[0161] Gesture Matching: 85%, Trajectory Similarity: 92%, Temporal Alignment: 98%;

[0162] For example, the system recognizes the learner's pressing gesture and combines it with the movement trajectory of the code scanner to determine that the learner has performed a code scanning operation.

[0163] Multi-Dimensional Matching Result Fusion:

[0164] Weighted Scoring: Assigning weights to different dimensions of matching results, e.g.:

[0165] Spatial Matching: 40%;

[0166] Temporal Matching: 30%;

[0167] Gesture Matching: 30%;

[0168] Fuzzy Logic Judgment: Handling partial matching situations, e.g.:

[0169] Trajectory deviation of 10% but correct gesture → judged as basically consistent;

[0170] Rule Engine Decision: Judging the final matching result based on predefined rules, e.g.:

[0171] When spatial matching degree > 80% and temporal matching degree > 70%, trigger a code scanning success event;

[0172] If the key trigger point is not activated, it is determined that the operation is incomplete;

[0173] Matching Result Visualization and Feedback:

[0174] Real-time Visualization: Highlighting the matching area and trigger points in the virtual scenario.

[0175] Operation Tips: Correcting non-standard actions through voice or text prompts, e.g.

[0176] Please keep the scanner perpendicular to the barcode.

[0177] Please slow down the operation speed.

[0178] Error marking: record error types and locations, and generate detailed operation reports.

[0179] The beneficial effects of the above technical solutions are: real-time comparison of student actions and GSP standards to ensure compliance with the operation process, provide targeted guidance based on matching results, accumulate operation records for analyzing student weaknesses, and optimize training content.

[0180] The present application provides a GSP medicine operation whole process virtual simulation system based on human-computer interaction, the analysis module comprises:

[0181] The first comparison unit is used for determining the gesture interaction feedback result of the virtual character to the corresponding simulation work scene according to the operation content, and comparing and analyzing the gesture theoretical feedback result of the corresponding training simulation task to judge whether the gesture feedback difference meets the first set standard;

[0182] If the first set standard is met, a satisfaction coefficient of 1 is assigned to the corresponding operation content;

[0183] Otherwise, according to the close relationship between the gesture feedback difference and the first set standard, a1 is assigned to the corresponding operation content, wherein a1 is a satisfaction coefficient based on gestures;

[0184] The second comparison unit is used for determining the voice interaction feedback result to the corresponding simulation work scene according to the voice input, and comparing and analyzing the voice theoretical feedback result of the corresponding training simulation task to judge whether the voice feedback difference meets the second set standard;

[0185] If the second set standard is met, a satisfaction coefficient of 1 is assigned to the corresponding voice input;

[0186] Otherwise, according to the close relationship between the voice feedback difference and the second set standard, a2 is assigned to the corresponding voice input, wherein a2 is a satisfaction coefficient based on voice;

[0187] The coefficient comparison unit is used for comparing the two satisfaction coefficients, and taking the interaction mode with the larger satisfaction coefficient as the main interaction logic judgment basis, and taking the interaction mode with the smaller satisfaction coefficient as the auxiliary interaction logic judgment basis;

[0188] If the two satisfaction coefficients are equal, both interaction modes are regarded as the main interaction logic judgment basis.

[0189] In this embodiment, the gesture interaction feedback result is taken as an input control instruction, and the feedback result of the system to the instruction is obtained, for example, after scanning the code, part of the information of the medicine should be displayed, and the gesture theoretical feedback result is a standard operation feedback in line with the GSP specification, such as displaying all information of the medicine after scanning the code;

[0190] At this time, the information not displayed is determined by comparing the part of the information of the medicine displayed with the all information of the medicine displayed, and the control instruction under the gesture interaction corresponding to the information not displayed, that is, the gesture action difference, is input into the gesture standard analysis model to determine whether the first set standard is met, and the model is obtained by training the neural network model based on different gesture interaction differences and whether the first set standard is met, so that the judgment result can be directly obtained.

[0191] In this embodiment, a1=1-sim(F1(sf),G(sb)), wherein F1(sf) represents a representation based on the gesture feedback difference sf; and G(sb) represents a representation based on the first set standard sb.

[0192] And the above-mentioned representation is matched from a representation database, and the database contains different differences and standards.

[0193] The voice interaction feedback result is the recognition and response of the system to the voice instruction of the student, such as confirming the batch number of the medicine and recording the temperature and humidity, sim(g1,g0) represents the voice feedback difference (a value) based on the voice interaction feedback result g1 and the voice theoretical feedback result g0, and whether it is less than the threshold value of the second set standard is determined, if it is less than the threshold value of the second set standard, a2 is 1, otherwise, Wherein, d2 is the threshold value of the second set standard, and the value is generally 0.1.

[0194] In this embodiment, the main interaction logic judgment basis is the information source that dominates the decision, and the auxiliary interaction logic judgment basis is the information used to supplement or verify the main basis.

[0195] For example, the student displays the scanning action specification of the code (a1=0.9) by gestures, but says that this is a qualified medicine (the actual medicine is unqualified) (a2=0.3) by voice, and the system determines that the operation is incomplete, and prompts to check the validity period of the medicine.

[0196] The beneficial effects of the above technical solutions are that the dual-mode fusion decision mechanism significantly improves the intelligence and adaptability of the training system, and single-mode errors do not lead to overall denial, and the system can still make correct judgments according to another mode, automatically adjusts the evaluation strategy according to the advantage interaction mode of the student, and improves the learning experience.

[0197] The present application provides a GSP medicine operation whole-process virtual simulation system based on human-computer interaction, and the analysis module further comprises:

[0198] a weight determination unit configured to determine a first weight for the gesture interaction and a second weight for the voice interaction according to the interaction logic determination result;

[0199] a gesture guiding unit configured to determine a behavior error start point according to a gesture interaction feedback difference and a behavior difference determined based on the first matching point and the second matching point, and determine a gesture guide with the behavior error start point as a starting point according to the training trigger point set;

[0200] a voice guiding unit configured to determine a correct voice from a voice scene interaction database corresponding to the real training process according to the voice interaction feedback difference, and perform voice guiding according to the correct voice;

[0201] a combination unit configured to obtain a behavior correct guide for the real training personnel in the real training process according to the gesture guide and the voice guide, and in combination with the first weight and the second weight.

[0202] In this embodiment, the value of a1 / (a1+a2) is regarded as the first weight, and the value of a2 / (a1+a2) is regarded as the second weight.

[0203] In this embodiment, the behavior error start point is a time point at which a first deviation occurs in the operation, for example, a motion deformation starts from the third frame.

[0204] In this embodiment, a first motion frame that does not meet the standard is located by searching forward from the current frame, and an index such as a key point position error and an angle deviation is calculated, a predefined correction template is called according to an error type, for example:

[0205] a trajectory error is displayed with a standard path, a timing error is prompted with a correct operation sequence, and a force error is provided with pressure feedback, for example, in a medicine storage scene, a student's moving motion causes a virtual medicine box to tilt at an angle exceeding 15° (the standard is ≤10°), the system analyzes that the medicine box angle is abnormal from the fifth frame, and generates a guide to keep the medicine box horizontal during moving.

[0206] In the refrigerated medicine acceptance, the student skips the temperature detection step and directly stores in the warehouse, the system identifies that the corresponding trigger point is missing, and prompts to detect the medicine temperature first.

[0207] In this embodiment, the voice scene interaction database includes guide content corresponding to error types of different voice differences, and the error types are classified as: missing keywords, supplementing key information, syntax errors, providing correct sentence patterns, semantic deviations, clarifying operation intentions, and the like.

[0208] The beneficial effects of the above technical solutions are: accurate positioning of the error starting point, providing targeted correction suggestions, dynamically adjusting the guidance mode according to the operation characteristics of the students, adapting to different learning styles, and ensuring that the GSP operation specification of the students meets the standard rate and satisfaction through real-time multi-modal feedback.

[0209] The application provides a GSP medicine operation whole-process virtual simulation system based on human-computer interaction, and further comprises:

[0210] The delay determination time is used for collecting an initial collection time of the test voice signal collected by each array unit, and obtaining a propagation delay time of the corresponding array unit according to the distance between the specified position and each array unit and in combination with an initial sound emission time of the test voice signal;

[0211] The acoustic determination unit is used for determining acoustic characteristics according to sound information of the test voice signal collected by each array unit;

[0212] The vector analysis unit is used for marking the propagation delay time and the acoustic characteristics on the corresponding array unit of the array structure to obtain an analysis group vector, and inputting the analysis group vector into a vector analysis model to determine a priority of each array unit, wherein the priority comprises a first priority and a second priority, and the first priority is higher than the second priority;

[0213] The screening unit is used for performing N1 times of voice tests on the voice input device, counting a first number of array units setting the first priority, and regarding the units with a ratio of the first number to N1 greater than a preset threshold as priority units;

[0214] In the voice input collection, the voice signal collected by the priority unit is used as a basis for voice interaction analysis.

[0215] In the embodiment, the array unit is a single microphone or a sound pickup module in the voice input device, for example, 8 microphones in a ring-shaped microphone array, the test voice signal is a standard audio used for calibrating the system, for example, a calibration voice with a fixed frequency is played, the initial collection time is a time when the array unit starts to receive the voice signal, for example, the microphone A captures the sound at 0.5 seconds, and the specified position is an actual spatial coordinate of voice emission, for example, a training staff emits the voice at a distance of 1 meter from the array center.

[0216] The propagation delay time refers to a difference between an actual transmission time and a theoretical transmission time of the same array unit, and the propagation delay time is Tc1-Tc0-L / vs, wherein Tc1 is the initial collection time, Tc0 is the initial sound emission time, L represents the corresponding distance, and vs represents a sound propagation speed.

[0217] In this embodiment, the vector analysis model is trained by using a sample of an analysis group vector and priority analysis results for each element in the vector, so that the priority of each array unit can be directly set, and the analysis group vector is {propagation delay time and acoustic characteristics of each array unit}, and it should be noted that each array unit in the vector is sequentially numbered for easy analysis and search.

[0218] In this embodiment, the preset threshold value is 2N1 / 3.

[0219] The beneficial effects of the above technical solutions are: starting from the aspects of propagation delay time and acoustic characteristics, the priority of the array unit is given by combining vector analysis, and the reliability of subsequent voice interaction is ensured.

[0220] The application provides a GSP medicine operation whole-process virtual simulation system based on human-computer interaction.

[0221] The probability determination unit is configured to synchronize and split the gesture guidance and the voice guidance in the correct behavior guidance according to a test point list under a corresponding practical training process, and predict a guidance success probability of each synchronized split block in combination with historical guidance behavior characteristics of the practical training personnel.

[0222] The test score determination unit is configured to determine a first test score of each synchronized split block according to the guidance success probability of each synchronized split block.

[0223] The strategy determination unit is configured to determine a recommended guidance frequency of the corresponding synchronized split block from a test score-frequency database according to the first test score of each synchronized split block, and obtain a behavior guidance strategy.

[0224] In this embodiment, the test point list is a set of test points specified in the GSP practical training process, for example, checking the consistency of the drug expiration date, batch number and invoice under the drug acceptance process.

[0225] The correct behavior guidance is guidance content generated by the system to correct the error operation of the student, for example, please keep the code scanning gun vertical (gesture guidance), please say the drug batch number completely (voice guidance).

[0226] The synchronized splitting is to segment the gesture guidance and the voice guidance according to the test points, because there are three cases of only gesture, only voice, and both gesture and voice for each test point.

[0227] In this embodiment, the historical guidance behavior characteristics are response data of different guidance in the past learning of the student, for example, the acceptance success rate of a student to the operation speed too fast type guidance is 70%, and the acceptance success rate of the student to the position deviation type guidance is 85%.

[0228] The specific acquisition method of the success probability is as follows:

[0229] The interaction type and the type weight of the interaction type of the corresponding test point of the synchronous split block are obtained from the test point-type database, and the interaction type is combined with the historical guidance behavior characteristics and the biased behavior type of the split knowledge points involved under each interaction type to determine the guidance feature set under each interaction type, wherein the guidance feature set includes: the historical guidance success probability of each split knowledge point in the biased behavior type under the corresponding interaction type.

[0230] In this embodiment, the test point-type database contains the interaction type and the type weight for different test points, which are pre-set, and each test point contains a plurality of split knowledge points, which are pre-stored.

[0231] The guidance success probability is predicted according to the guidance feature set;

[0232]

[0233] wherein, is the weight of the case where only gestures exist; is the weight of the case where only speech exists; ε s , ε y is the gesture weight and speech weight under the case where both gestures and speech exist; z1 i1 , p1 i1 are the difficulty coefficient and the historical guidance success probability of the i1th split knowledge point under the case where only gestures exist; m1, m2, and m3 represent the total number of split knowledge points under the three cases (only gestures exist, only speech exists, and both gestures and speech exist), and m1=m2=m3; s1 k1 , p3 k1 respectively represent the difficulty coefficient and the historical guidance success probability of the k1th split knowledge point based on the gesture direction under the case where both gestures and speech exist; s2 k1 , p4 k1 respectively represent the difficulty coefficient and the historical guidance success probability of the k1th split knowledge point based on the speech direction under the case where both gestures and speech exist; ε s , ε y respectively represent the weights based on the gesture direction and the speech direction under the case where both gestures and speech exist; A1 is a modal adaptation function; A2 is a forgetting function; E is a decay period; ΔT u1 represents the forgetting time related to the difficulty coefficient of the u1th split knowledge point in the h1th biased behavior type; d h1 represents the number of split knowledge points in the h1th biased behavior type in the synchronous split block; p h1represents the historical guidance success probability of the h1th biased behavior type in the synchronous split block; m4 represents the number of biased behavior types in the synchronous split block.

[0234] In this embodiment, the pure gesture, pure voice, and mixed modal classification processing are performed through (single modal weight) and epsilon s , epsilon y (sub-weight under mixed modal), which accurately depict the characteristic differences of different interaction types. The difficulty coefficient reflects the complexity of the knowledge point itself (for example, "cold chain temperature calibration" is more difficult than "code scanning", and the corresponding difficulty coefficient is larger), the historical probability reflects the individualized ability of the student (for example, student A has a high success rate of "gesture type" guidance, which is closer to 1), and delta T u1 is a "difficulty-forgetting" correlation term (the more difficult the knowledge point, the faster the forgetting).

[0235] In this embodiment, the difficulty coefficient, the historical guidance success probability, and the type weight correspond to a value range of 0 to 1.

[0236] Through multi-modal subdivision, dynamic historical correction, and forgetting rule modeling, the complexity of the practical training interaction is met, and the core defects of traditional prediction are solved, which is a key mathematical basis for supporting practical training guidance.

[0237] In this embodiment, the first test score = guidance success probability x 100 x coefficient of complex test points.

[0238] In this embodiment, the test score-frequency database is a pre-established mapping table that records the recommended guidance frequency corresponding to different test score intervals, for example, test scores 80-90 correspond to 1 prompt every 3 operations, and test scores 90-100 correspond to 1 prompt every 5 operations.

[0239] The recommended guidance frequency is the frequency at which the system repeatedly displays certain guidance content to the student, which is used to balance learning effectiveness and interference level.

[0240] The behavior guidance strategy is a personalized guidance scheme formed by integrating the recommended guidance frequencies of all split blocks, for example, in the medicine code scanning process, the code scanning angle guidance is prompted once every 2 operations, and the batch number checking guidance is prompted once every 4 operations.

[0241] The beneficial effects of the above technical solutions are: customizing guidance according to the student's historical behavior and test point difficulty, making the guidance content more in line with actual needs, avoiding excessive prompts that cause interference, reducing ineffective repeated guidance, dynamically adjusting the guidance frequency, the system can quickly adapt to the learning pace of different students, continuously accumulate student behavior data, optimize the test score-frequency mapping rules and prediction model, improve training quality, and significantly enhance the mastery of key operation links.

[0242] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A GSP drug product operation whole-process virtual simulation system based on human-computer interaction, characterized in that, The method comprises the following steps: a virtual simulation engine end is used to present a medical simulation work scene, wherein the medical simulation work scene is related to each training process in a medical training whole process; an interactive device is used to capture the current action of a training personnel in the presented medical simulation work scene and acquire the operation content of the training personnel in the real world according to the current training process in which the training personnel is located; a voice input device is used to receive the voice input of the training personnel in the current training process; an analysis module is used to make an interactive logic judgment on the operation content and the voice input, make a behavior feedback, determine the correct behavior guidance of the training personnel in the current training process, and generate a behavior guidance strategy of the current training process; a guidance module is used to continuously guide the training personnel according to the behavior guidance strategy until the behavior operation of the training personnel in each training process is standardized. The interactive device comprises: a collection unit is used to collect the skeletal key point data of the current action of a human body in real time, wherein the skeletal key point data comprises the coordinate information of 27 skeletal key points of a human hand in a three-dimensional space; a model establishing unit is used to convert the rotation information of each skeletal key point into a quaternion representation according to the collected skeletal key point data, combine the translation coordinates, establish a skeletal key point quaternion motion model of the current action, and describe the posture and position of each skeletal key point in the three-dimensional space; a feature extraction unit is used to continuously acquire the quaternion motion model of a plurality of action frames, form action sequence data, input the action sequence data into a pre-trained LSTM network, extract the time sequence action features in the action sequence, and the time sequence action features comprise the sequence, duration and rhythm change time dependence information of the action; a space mapping unit is used to map the skeletal key point information of the human body action in the real world to a virtual character in a virtual space according to the extracted time sequence action features and the quaternion motion model. The interactive device further comprises: A locking unit is configured to sequentially lock position points of each bone key point in a continuous action frame of a virtual role, and construct a movement vector of the corresponding bone key point and an action gesture of each action frame , wherein represents the position coordinates of the corresponding bone key point at time in the i-th action frame; represents the position coordinates of the j-th bone key point at time t2j in the corresponding action frame. a point set constructing unit is used to determine the standard vector of each skeletal key point and a training trigger point set according to the standard operation behavior of the simulation work scene constructed based on the simulation training task of the current training process, wherein the training trigger point set comprises the trigger position point, trigger duration and trigger sequence of the training trigger point; a matching unit is used to match each movement vector with the corresponding standard vector once, and match each movement vector and the action gesture of each action frame with the training trigger point set in turn twice; a content determining unit is used to filter the first matching point under each action frame according to the first matching result and filter the second matching point of each action frame according to the second matching result, and determine the operation content of the training personnel.

2. The GSP pharmaceutical product operation whole-process virtual simulation system based on human-computer interaction according to claim 1, characterized in that, The analysis module comprises: a first comparison unit is used to determine the gesture interaction feedback result of the virtual character to the corresponding simulation work scene according to the operation content, compare and analyze the gesture interaction feedback result with the gesture theoretical feedback result of the corresponding training simulation task, and judge whether the gesture feedback difference meets a first set standard. If the first set standard is met, a satisfaction coefficient of 1 is assigned to the corresponding operation content; Otherwise, according to the close relationship between the gesture feedback difference and the first set standard, a1 is assigned to the corresponding operation content, where a1 is a satisfaction coefficient based on gestures; The second comparison unit is configured to determine a voice interaction feedback result for the corresponding simulation working scenario according to the voice input, and compare the voice interaction feedback result with a voice theoretical feedback result of the corresponding training simulation task to determine whether a voice feedback difference meets a second set standard; If the second set standard is met, a satisfaction coefficient of 1 is assigned to the corresponding voice input; Otherwise, according to the close relationship between the voice feedback difference and the second set standard, a2 is assigned to the corresponding voice input, where a2 is a satisfaction coefficient based on voice; The coefficient comparison unit is configured to compare the two satisfaction coefficients, and take the interaction mode with the larger satisfaction coefficient as the main interaction logic judgment basis, and take the interaction mode with the smaller satisfaction coefficient as the auxiliary interaction logic judgment basis. If the two satisfaction coefficients are equal, both of the two interaction modes are taken as the main interaction logic judgment basis. 3.The GSP pharmaceutical product operation whole-process virtual simulation system based on human-computer interaction according to claim 2, characterized in that, The analysis module further includes: The weight determination unit is configured to assign a first weight to the gesture interaction and a second weight to the voice interaction according to the interaction logic judgment result; The gesture guidance unit is configured to determine a behavior error start point according to the gesture interaction feedback difference and a behavior difference determined based on the first matching point and the second matching point, and determine gesture guidance with the behavior error start point as a starting point according to the training trigger point set; The voice guidance unit is configured to determine correct voice from a voice scene interaction database under a corresponding practical training process according to the voice interaction feedback difference, and perform voice guidance; The combination unit is configured to obtain a behavior correct guidance of the practical training personnel in the current practical training process according to the gesture guidance, the voice guidance, and in combination with the first weight and the second weight.

4. The GSP pharmaceutical product operation whole-process virtual simulation system based on human-computer interaction according to claim 1, characterized in that, Further includes: The delay determination unit is configured to collect an initial collection time of a test voice signal collected by each array unit, and determine a propagation delay time of the corresponding array unit according to a distance between a specified position and each array unit and in combination with an initial sound emission time of the test voice signal; The acoustic determination unit is configured to determine acoustic characteristics according to sound information of the test voice signal collected by each array unit; The vector analysis unit is configured to label the propagation delay time and the acoustic characteristics on the corresponding array unit of the array structure to obtain an analysis group vector, and input the analysis group vector into a vector analysis model to determine a priority of each array unit, where the priority includes a first priority and a second priority, and the first priority is higher than the second priority; The screening unit is configured to perform N1 times of voice test on the voice input device, count a first number of array units with the first priority, and take units with a ratio of the first number to N1 greater than a preset threshold as priority units. When collecting the voice input, the voice signal collected by the priority unit is used as a basis for voice interaction analysis.

5. The GSP pharmaceutical product operation whole-process virtual simulation system based on human-computer interaction according to claim 3, characterized in that, The analysis module further includes: The probability determination unit is configured to synchronize and split the gesture guidance and the voice guidance in the correct behavior guidance according to a test point list under a corresponding practical training process, and to predict a guidance success probability of each synchronized split block in combination with historical guidance behavior characteristics of the practical training personnel. The test score determination unit is configured to determine a first test score of each synchronized split block according to the guidance success probability of each synchronized split block. The strategy determination unit is configured to determine a recommended guidance frequency of the corresponding synchronized split block from a test score-frequency database according to the first test score of each synchronized split block, so as to obtain a behavior guidance strategy. 6.The GSP drug business whole-process virtual simulation system based on human-computer interaction according to claim 1, characterized in that, The matching unit comprises: The overlapping segment determination subunit is configured to determine a relative overlapping segment of a moving track of the corresponding moving vector and a standard track of the standard vector. The judgment subunit is configured to judge whether the relative overlapping segment contains a key interaction point in the standard vector. If yes, the moving vector is reserved based on a partial vector under the relative overlapping segment. If no, it is determined that the first matching point does not exist in the corresponding one-time matching result. The reserved partial vector is taken as the one-time matching result.

Images