GSP drug management full-process virtual simulation system based on human-computer interaction

Abstract

The invention belongs to the crossing of a medicine education technology and GSP medicine operation management informatization, and provides a human-computer interaction-based GSP medicine operation full-process virtual simulation system, which comprises a virtual simulation engine end, an interaction device, a human-computer interaction platform, a human-computer interaction platform, a human-computer interaction platform and a human-computer interaction platform, and is characterized in that the virtual simulation engine end is used for displaying a medicine simulation working scene in reality; the action capturing module is used for capturing the current action of the practical training personnel and acquiring the real operation content of the practical training personnel according to the current practical training process of the practical training personnel; the voice input equipment is used for receiving voice input of practical training personnel in the current practical training process; the analysis module is used for carrying out interactive logic judgment on the operation content and the voice input and carrying out behavior feedback, determining correct behavior guidance of the practical training personnel in the current practical training process and generating a behavior guidance strategy of the current practical training process; and the guiding module is used for continuously guiding the training personnel depending on the behavior guiding strategy. And the practical training effect is improved through immersive learning.

Description

Technical Field

[0001] The present invention relates to the cross-technical field of medical education technology and GSP drug operation and management informatization, and proposes a GSP drug operation full-process virtual simulation system based on human-computer interaction. Background Art

[0002] At present, in the field of GSP simulation training systems, the main product form is the GSP pharmaceutical business virtual simulation training system. This product mainly uses 3D virtual simulation technology to build pharmaceutical distribution enterprise scenarios in a virtual environment, and complete relevant enterprise cognition, enterprise workflow training, work task simulation assessment, etc. on the computer. However, the system relies on keyboard, mouse or simple touch screen operation, that is, it uses two-dimensional operation, and cannot simulate the three-dimensional space operation in the real pharmaceutical business scenario. The existing VR system only supports handle operation, which is essentially different from the actual fine movements such as drug handling and document handover. It also lacks simulation of key communication scenarios in pharmaceutical business (such as prescription drug sales dialogues, drug procurement contract negotiations, etc.). It has the defects of insufficient immersion and a single interactive method, which makes it difficult for users to enter the learning state, reduces the learning and training effect, and lacks support for job training that requires communication and operation.

[0003] Therefore, the present invention provides a GSP drug management full-process virtual simulation system based on human-computer interaction. Summary of the Invention

[0004] The present invention provides a virtual simulation system for the entire GSP drug management process based on human-computer interaction, which is used to display three-dimensional scenarios through a virtual simulation engine, and then to conduct practical training on the entire GSP process by capturing and analyzing gesture interactions and voice interactions in the scenarios, thereby improving the training effect through immersive learning.

[0005] The present invention proposes a virtual simulation system for the entire process of GSP drug management based on human-computer interaction, including:

[0006] A virtual simulation engine is used to realistically present pharmaceutical simulation work scenarios, wherein the pharmaceutical simulation work scenarios are related to each practical training process in the entire pharmaceutical training process;

[0007] Interactive equipment, used to capture the trainee's current actions when the trainee is in a realistic medical simulation work scenario and obtain the trainee's actual operation content based on the trainee's current training process;

[0008] A voice input device, used to receive voice input from the trainee during the current training process;

[0009] An analysis module is used to perform interactive logical judgment on the operation content and voice input and provide behavioral feedback, determine the correct behavioral guidance for the trainee in the current training process, and generate a behavioral guidance strategy for the current training process;

[0010] The guidance module is used to continuously guide the trainee based on the behavior guidance strategy until the trainee's behavior operation is standardized for the training simulation task of each training process.

[0011] Preferably, the interactive device includes:

[0012] An acquisition unit is used to collect the skeleton key point data of the human body's current movement in real time. The skeleton key point data includes the coordinate information of 27 skeleton key points of the human hand in three-dimensional space;

[0013] The model building unit is used to convert the rotation information of each skeletal key point into a quaternion representation based on the collected skeletal key point data, and combine it with the translation coordinates to build a quaternion motion model of the skeletal key point of the current action to describe the posture and position of each skeletal key point in three-dimensional space;

[0014] A feature extraction unit is used to continuously obtain quaternion motion models of multiple action frames to form action sequence data, input the action sequence data into a pre-trained LSTM network, and extract temporal action features in the action sequence, including the order of actions, duration, and time-dependent information of rhythm changes;

[0015] The spatial mapping unit is used to map the skeleton key point information of human body movements in the real world to the virtual character in the virtual space based on the extracted temporal motion features and quaternion motion model.

[0016] Preferably, the interactive device further includes:

[0017] The locking unit is used to sequentially lock the position of each skeleton key point in the continuous action frames of the virtual character, and construct the movement vector Yx={w t1i , i=1,2,3,...,Tn} and the action gesture Ds of each action frame={r t2j ,j=1,2,3,...,27}, where w t1i Represents the position coordinates of the corresponding skeleton key point at time t1i in the i-th action frame; r t2j Represents the position coordinates of the jth skeleton key point at time t2j in the corresponding action frame;

[0018] A point set construction unit is used to determine a standard vector of each skeletal key point and a training trigger point set according to the standard operating behavior of the simulation work scenario constructed based on the simulation training task of the current training process, wherein the training trigger point set includes the trigger position point, trigger duration and trigger sequence of the training trigger point;

[0019] A matching unit is used to match each motion vector with the corresponding standard vector once, and at the same time, to match each motion vector and the action gesture of each action frame with the training trigger point set twice in sequence;

[0020] The content determination unit is used to screen the first matching point under each action frame according to the primary matching result and screen the second matching point under each action frame according to the secondary matching result to determine the operation content of the trainee.

[0021] Preferably, the analysis module includes:

[0022] The first comparison unit is used to determine the gesture interaction feedback result of the virtual character for the corresponding simulated work scenario according to the operation content, and compare and analyze the gesture interaction feedback result of the corresponding training simulation task to determine whether the difference between the gesture feedback results meets the first set standard;

[0023] If the first set standard is met, a satisfaction coefficient of 1 is assigned to the corresponding operation content;

[0024] Otherwise, according to the close relationship between the gesture feedback difference and the first set standard, a1 is assigned to the corresponding operation content, where a1 is a satisfaction coefficient based on the gesture;

[0025] The second comparison unit is used to determine the voice interaction feedback result of the corresponding simulation work scenario based on the voice input, and compare and analyze the voice interaction feedback result of the corresponding training simulation task to determine whether the difference between the two voice feedbacks meets the second set standard;

[0026] If the second set criterion is met, a satisfaction coefficient of 1 is assigned to the corresponding voice input;

[0027] Otherwise, according to the close relationship between the speech feedback difference and the second set standard, assign a2 to the corresponding speech input, where a2 is a speech-based satisfaction coefficient;

[0028] A coefficient comparison unit is used to compare the sizes of two satisfaction coefficients, and regard the interaction mode with the larger satisfaction coefficient as the main interaction logic judgment basis, and regard the interaction mode with the smaller satisfaction coefficient as the auxiliary interaction logic judgment basis;

[0029] Among them, if the two satisfaction coefficients are equal, both interaction modes are regarded as the main interaction logic judgment basis.

[0030] Preferably, the analysis module further includes:

[0031] a weight determination unit, configured to assign a first weight to the gesture interaction and a second weight to the voice interaction according to a result of the interaction logic determination;

[0032] a gesture guidance unit, configured to determine a behavioral error starting point based on a difference in gesture interaction feedback and a behavioral difference determined based on the first matching point and the second matching point, and to determine a gesture guidance with the behavioral error starting point as a starting point based on the training trigger point set;

[0033] A voice guidance unit is used to retrieve the correct voice from the voice scenario interaction database under the corresponding training process and provide voice guidance based on the difference in the voice interaction feedback;

[0034] The combination unit is used to obtain correct guidance for the trainee's behavior in the current training process based on the gesture guidance, voice guidance, and the first weight and the second weight.

[0035] Preferably, it also includes:

[0036] Delay determination time, for collecting the initial acquisition time of the test voice signal by each array unit, and obtaining the propagation delay time of the corresponding array unit based on the distance between the designated position and each array unit and the initial utterance time of the test voice signal;

[0037] an acoustic determination unit, configured to determine acoustic features based on sound information of a test speech signal collected by each array unit;

[0038] a vector analysis unit, configured to mark the propagation delay time and the acoustic characteristics on corresponding array elements of the array structure to obtain an analysis group vector, and input the vector into a vector analysis model to determine a priority of each array element, wherein the priority includes a first priority and a second priority, and the first priority is superior to the second priority;

[0039] a screening unit, configured to perform N1 voice tests on the voice input device, count a first number of array elements for which a first priority is set, and regard elements for which a ratio of the first number to N1 is greater than a preset threshold as priority elements;

[0040] Among them, when collecting voice input, voice interaction analysis is performed based on the voice signal collected by the priority unit.

[0041] Preferably, the analysis module further includes:

[0042] a probability determination unit, configured to synchronously split the gesture guidance and voice guidance in the correct behavior guidance according to the test point list under the corresponding training process, and predict the guidance success probability of each synchronous split block based on the historical guidance behavior characteristics of the trainee;

[0043] a test score determining unit, configured to determine a first test score for each synchronous split block according to a guidance success probability of each synchronous split block;

[0044] The strategy determination unit is used to determine the recommended guidance frequency of the corresponding synchronous split block from the test score-frequency database according to the first test score of each synchronous split block, and obtain a behavior guidance strategy.

[0045] Preferably, the matching unit includes:

[0046] an overlapping segment determining subunit, configured to determine a relative overlapping segment between a moving trajectory of a corresponding moving vector and a standard trajectory of a standard vector;

[0047] a judging subunit, configured to judge whether the relatively overlapping segment contains a key interaction point in the standard vector;

[0048] If included, retaining the motion vector based on the partial vector under the relatively overlapping segment;

[0049] If not included, it is determined that the first matching point does not exist in the corresponding matching result;

[0050] Among them, the retained part of the vector is used as a matching result.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] The three-dimensional scene is displayed through the virtual simulation engine, and then the whole GSP process is trained by capturing and analyzing the gesture interaction and voice interaction in the scene, and the training effect is improved through immersive learning.

[0053] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0056] Figure 1 This is a structural diagram of a GSP drug management full-process virtual simulation system based on human-computer interaction in an embodiment of the present invention;

[0057] Figure 2 This is the domain knowledge graph in the embodiment of the present invention;

[0058] Figure 3 A code diagram for tracking the conversation state in an embodiment of the present invention;

[0059] Figure 4 This is a flow chart of the virtual simulation training of GSP drug management in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0061] The present invention proposes a virtual simulation system for the entire process of GSP drug management based on human-computer interaction. Figure 1 Shown, including:

[0062] A virtual simulation engine is used to realistically present pharmaceutical simulation work scenarios, wherein the pharmaceutical simulation work scenarios are related to each practical training process in the entire pharmaceutical training process;

[0063] Interactive equipment, used to capture the trainee's current actions when the trainee is in a realistic medical simulation work scenario and obtain the trainee's actual operation content based on the trainee's current training process;

[0064] A voice input device, used to receive voice input from the trainee during the current training process;

[0065] An analysis module is used to perform interactive logical judgment on the operation content and voice input and provide behavioral feedback, determine the correct behavioral guidance for the trainee in the current training process, and generate a behavioral guidance strategy for the current training process;

[0066] The guidance module is used to continuously guide the trainee based on the behavior guidance strategy until the trainee's behavior operation is standardized for the training simulation task of each training process.

[0067] In this embodiment, 3D modeling technology is used to create virtual pharmacies, warehouses, distribution centers and other scenarios, highly restoring the real environment and simulating various equipment and systems required by GSP, such as temperature and humidity monitoring systems, electronic supervision code scanning equipment, refrigeration equipment, etc. Standardized operating procedures are designed according to GSP terms, and each process corresponds to a specific training scenario.

[0068] Among them, the virtual drug management scenario created based on the GSP specifications includes simulation scenarios of drug procurement, acceptance, storage, maintenance, sales, warehouse review, transportation and other links.

[0069] In this embodiment, the training process is a specific operational link in the GSP process, such as the drug warehousing acceptance process, the temperature and humidity monitoring process, the unqualified drug handling process, etc.

[0070] In this embodiment, the current action is the specific operation action of the trainee in the virtual GSP scenario, such as scanning the code, unpacking, checking the appearance of the medicine, recording data, etc.

[0071] The operation content is the operational behavior that complies with GSP standards analyzed from the action, such as checking the expiration date of the drug, verifying the quantity of the drug, recording the temperature and humidity data, etc. The voice input is the instructions, inquiries or reports issued by the trainees during the operation that comply with GSP standards, such as what is the batch number of the drug, whether the temperature and humidity meet the standards, and whether this batch of drugs needs to be rejected.

[0072] Interactive logic judgment is based on GSP specifications to analyze whether the trainee's operation content or voice input is more important. Behavioral feedback provides immediate compliance feedback on the trainee's operation, such as correct operation, please check the expiration date of the drug, etc.

[0073] Correct behavioral guidance provides specific corrective suggestions and operational guidelines for operations that do not comply with GSP standards. Behavioral guidance strategies develop personalized training plans and guidance methods based on the trainees' operational performance.

[0074] The training simulation tasks are specific operational tasks designed based on the GSP process, such as completing the warehousing acceptance of a batch of refrigerated medicines, handling the return of unqualified medicines, etc. The behavioral operating specifications are standard operating procedures and methods that comply with the requirements of the GSP terms.

[0075] In this embodiment, the domain knowledge graph is constructed as follows Figure 2 shown.

[0076] In this embodiment, the BERT+CRF model is used to implement conversation state tracking, capture keywords in the conversation, and complete conversation speech analysis and feedback. Figure 3 shown.

[0077] In this embodiment, for a typical workflow:

[0078] Receiving medicines:

[0079] (1) Gesture interaction:

[0080] Extend one hand forward to simulate the action of receiving the accompanying information; click the virtual acceptance button with the right index finger; put the five fingers together and make a fist to simulate the action of picking up the medicine box; rotate and slide the hand to simulate the action of rotating the medicine box for inspection, etc.

[0081] (2) Voice interaction:

[0082] During the receiving process, simulate necessary conversations with drug transporters and drug inspectors, such as:

[0083] Transporter: Hello, this is the accompanying documentation for this batch of medicines, please check.

[0084] User: OK, I’ll take care of it right away.

[0085] Pharmaceutical sales:

[0086] (1) Voice interaction

[0087] In the three-dimensional virtual simulation scenario, an aunt is simulated entering a pharmacy, and the system simulates the reception service of the clerk.

[0088] User asks: "Hello, auntie, how can I help you?"

[0089] The virtual character replied: "Hello, I have a cold and want to buy some medicine";

[0090] User asks: "Please describe your symptoms to me in detail";

[0091] The virtual character replied: "I have had a cough and runny nose since yesterday. Today it feels a little worse. I also have a slight headache, but no fever."

[0092] (2) Gesture interaction

[0093] When you need to pick up prescription medicine, you need to extend your arm forward to receive the prescription, and use the pen-holding gesture to sign the prescription.

[0094] In this embodiment, the content of the above solution can also be as follows Figure 4 The method shown is implemented.

[0095] The beneficial effects of the above technical solution are: three-dimensional scene display is carried out through the virtual simulation engine, and then the whole GSP process is trained by capturing and analyzing the gesture interaction and voice interaction in the scene, thereby improving the training effect through immersive learning.

[0096] The present invention proposes a GSP full-process virtual-reality fusion simulation system based on multimodal human-computer interaction, wherein the interaction device comprises:

[0097] An acquisition unit is used to collect the skeleton key point data of the human body's current movement in real time. The skeleton key point data includes the coordinate information of 27 skeleton key points of the human hand in three-dimensional space;

[0098] The model building unit is used to convert the rotation information of each skeletal key point into a quaternion representation based on the collected skeletal key point data, and combine it with the translation coordinates to build a quaternion motion model of the skeletal key point of the current action to describe the posture and position of each skeletal key point in three-dimensional space;

[0099] A feature extraction unit is used to continuously obtain quaternion motion models of multiple action frames to form action sequence data, input the action sequence data into a pre-trained LSTM network, and extract temporal action features in the action sequence, including the order of actions, duration, and time-dependent information of rhythm changes;

[0100] The spatial mapping unit is used to map the skeleton key point information of human body movements in the real world to the virtual character in the virtual space based on the extracted temporal motion features and quaternion motion model.

[0101] In this embodiment, the code for virtual-real space mapping is as follows:

[0102] Python

[0103] defcoordinate_mapping(real_pos):

[0104] #Non-linear calibration formula

[0105] virtual_x=a*real_x^3+b*real_x^2+c*real_x

[0106] virtual_y=d*log(real_y)+e

[0107] return(virtual_x,virtual_y).

[0108] In this embodiment, 27 key skeletal points of the hand—including precise coordinates of the wrist, base of the palm, and finger joints—are used to fully describe hand movements. The three-dimensional spatial coordinate information represents the precise position of each key point on the X, Y, and Z axes, typically measured in millimeters. Accelerometers and gyroscopes are used to measure joint angles and motion. For example, in a drug acceptance scenario, the system collects trainees' hand movements in real time, identifying fine movements like picking up a medicine box and turning the wrist to read a label. In GSP training, it captures trainees' hand posture and movement trajectory when operating electronic surveillance code scanning equipment.

[0109] In this embodiment, the rotation information is the orientation and angle change of the skeleton key points in three-dimensional space. The quaternion representation uses four numbers (w, x, y, z) to represent the three-dimensional rotation to avoid the universal joint lock problem of Euler angles. The translation coordinates are the position movement of the skeleton key points in space. The quaternion motion model is a mathematical model that integrates rotation and position information to accurately describe skeleton movement. For example, in the operation of drug warehousing, the system converts the student's hand rotation movement (such as unscrewing the bottle cap) into a quaternion representation to ensure that the virtual character's movement is natural and smooth. When simulating the operation of refrigeration equipment, the strength and angle of the finger pressing the button are accurately captured and reproduced in the virtual environment through the quaternion model.

[0110] In this embodiment, action sequence data is a collection of quaternion motion models spanning multiple consecutive time frames. Sequential action features are temporal dependencies between actions, such as sequence, duration, and rhythmic changes. In drug maintenance operations, the system uses an LSTM network to analyze the trainee's action sequence while inspecting the drug's appearance, determining whether the sequence follows the GSP's "look, smell, shake" sequence. In cold-chain drug transportation scenarios, the system identifies whether the trainee's operation of the temperature recorder meets the 30-minute recording requirement.

[0111] In this embodiment, the virtual space is a computer-generated three-dimensional environment, such as a virtual pharmacy or warehouse, and the virtual character is a digital image representing the student in the virtual space. The specific implementation means of the mapping are:

[0112] Based on motion redirection technology: the captured human body motion is adapted to the skeletal structure of the virtual character.

[0113] IK (Inverse Kinematics) algorithm: Calculates the pose of the entire skeleton chain based on the position of the end joints (such as fingers).

[0114] Physics simulation: Add physical factors such as gravity and inertia to make virtual actions more realistic.

[0115] For example, in a virtual warehouse scenario, the trainees’ actual carrying movements are mapped onto a virtual character, which then simultaneously completes operations such as carrying medicine boxes and placing them on shelves.

[0116] During the simulated pharmacy dispensing process, the trainees' hand movements (such as picking up medicine bottles, pouring out pills, and filling out labels) are accurately mapped to the virtual characters to ensure standardized and accurate operations.

[0117] The beneficial effects of the above technical solution are: quaternion representation and LSTM timing analysis ensure the natural and coherent movements of the virtual character. Through timing feature analysis, it is determined whether the trainee's operation complies with the GSP process and time requirements. The virtual character and the trainee's movements are synchronized, which enhances the realism and participation of the training, facilitates the analysis of existing abnormal behaviors, and provides a basis for subsequent guidance.

[0118] The present invention proposes a virtual simulation system for the entire process of GSP drug management based on human-computer interaction, wherein the interactive device further comprises:

[0119] The locking unit is used to sequentially lock the position of each skeleton key point in the continuous action frames of the virtual character, and construct the movement vector Yx={w t1i , i=1,2,3,...,Tn} and the action gesture Ds of each action frame={r t2j ,j=1,2,3,...,27}, where w t1i Represents the position coordinates of the corresponding skeleton key point at time t1i in the i-th action frame; r t2j Represents the position coordinates of the jth skeleton key point at time t2j in the corresponding action frame;

[0120] A point set construction unit is used to determine a standard vector of each skeletal key point and a training trigger point set according to the standard operating behavior of the simulation work scenario constructed based on the simulation training task of the current training process, wherein the training trigger point set includes the trigger position point, trigger duration and trigger sequence of the training trigger point;

[0121] A matching unit is used to match each motion vector with the corresponding standard vector once, and at the same time, to match each motion vector and the action gesture of each action frame with the training trigger point set twice in sequence;

[0122] The content determination unit is used to screen the first matching point under each action frame according to the primary matching result and screen the second matching point under each action frame according to the secondary matching result to determine the operation content of the trainee.

[0123] Wherein, the matching unit includes:

[0124] an overlapping segment determining subunit, configured to determine a relative overlapping segment between a moving trajectory of a corresponding moving vector and a standard trajectory of a standard vector;

[0125] a judging subunit, configured to judge whether the relatively overlapping segment contains a key interaction point in the standard vector;

[0126] If included, retaining the motion vector based on the partial vector under the relatively overlapping segment;

[0127] If not included, it is determined that the first matching point does not exist in the corresponding matching result;

[0128] Among them, the retained part of the vector is used as a matching result.

[0129] In this embodiment, the action frame is a single frame of action data in a continuous time sequence, such as one frame in 30 frames per second. The position point of the skeleton key point is the coordinates (x, y, z) of the wrist joint in three-dimensional space. The movement vector is the displacement vector of the skeleton key point between adjacent action frames, such as the vector from (x1, y1, z1) to (x2, y2, z2). For example, in the scenario of drug code scanning, the locking unit tracks the continuous action frames of the index finger pressing the trigger of the scanner gun and calculates the movement vector of the finger joint.

[0130] In this embodiment, the action gesture is a posture formed by combining multiple skeletal key points, such as a pinch gesture determined by the relative positions of the thumb, index finger, and middle finger.

[0131] In this embodiment, the standard operating behavior is the action sequence required by the GSP specification, such as checking the appearance of the drug for 5 seconds during acceptance. The standard vector is a skeletal movement path that complies with the specification, for example, the barcode scanner should move at a constant speed along the drug barcode.

[0132] The training trigger points are key nodes in the operation process, such as:

[0133] Trigger point: the point where the barcode scanner contacts the medicine.

[0134] Trigger duration: The scanning action should last for more than 0.5 seconds.

[0135] Trigger sequence: scan the code first, then check the information, and finally record.

[0136] In this embodiment, the relative overlapping segment is the overlapping part of the student's action trajectory and the standard trajectory, such as the overlapping percentage of the student's actual scanning path and the standard path, and relative means that the allowable deviation range of each trajectory point is within ±5mm. As long as it is within this range, the point is considered to overlap.

[0137] Key interaction points are critical locations in an operation, such as the starting position of a barcode scanner and the center point of a medicine label.

[0138] In this embodiment, the first matching point is the matching position of the action trajectory and the standard trajectory, such as the starting point of the code scanning, and the second matching point is the matching result of the action and the trigger point set, such as completing the code scanning within the specified time. The operation content is the specific behavior analyzed from the comprehensive matching results, such as the completion of the code scanning operation of drug A. For example, based on the first matching point (code scanning trajectory) and the second matching point (triggering the trigger action of the code scanning gun), it is determined that the student has performed the drug scanning operation. In the scenario of handling unqualified drugs, the operation is confirmed to be completed by matching the gesture of filling out the return form and the trajectory of placing the drug in the return area.

[0139] In this embodiment, the secondary matching process is as follows:

[0140] Normalize motion vectors: Convert the original motion vectors to unit vectors and normalize their lengths to eliminate the effects of force differences.

[0141] Gesture feature quantization: Converting motion gestures into feature vectors, such as the relative angles between fingers, the distance ratios between key points, and the orientation and rotation angle of the palm;

[0142] Trigger point set preprocessing: define the spatial range for each trigger point (such as a 5mm sphere centered on a standard position), set a time window (such as the trigger point must be activated within 0.5 seconds), and mark the trigger priority (such as required trigger points and optional trigger points);

[0143] For example, in the scenario of drug code scanning, the movement trajectory of the barcode scanner is converted into a standardized vector sequence, and the features of the gesture of holding the barcode scanner are extracted (such as the angle between the thumb and index finger is 45°±10°).

[0144] Spatial dimension matching:

[0145] Trigger area detection: Determine whether the end point of the motion vector falls within the spatial range of the trigger point.

[0146] Trajectory overlap calculation: Use the Dynamic Time Warping (DTW) algorithm to calculate the similarity between the actual trajectory and the standard trajectory, and calculate the Hausdorff distance between the trajectories to evaluate the degree of deviation

[0147] Spatial relationship verification: Check the relative position relationship between multiple key points, for example: whether the vertical distance between the scanner and the drug label is within the range of 5-10mm, and whether the finger is correctly placed on the trigger position of the scanner;

[0148] Time dimension matching:

[0149] Timing alignment: Align the trainee’s movement sequence with the standard timing diagram to resolve speed differences.

[0150] Duration verification: Check whether the duration of the action in the trigger point area meets the requirements, for example:

[0151] The scanning action should last for 0.5-1.0 seconds, and the drug inspection action should last for more than 3 seconds;

[0152] Sequential logic check: Verify that the activation order of the trigger points is correct. For example, the code must be scanned before the information is verified, and refrigerated medicines must be temperature-tested before being put into storage.

[0153] For example, the system records the time sequence from when the trainee picks up the barcode scanner to when he completes the scanning, and determines whether all necessary operations are completed within the specified time.

[0154] Gestures match action types:

[0155] Gesture template comparison: Match the current gesture with a predefined template library, for example:

[0156] Grasping gesture: The distance between thumb and index finger is less than 2cm;

[0157] Rotation gesture: The rotation angle of the palm changes by more than 90°;

[0158] Action semantic parsing: Infer the action type based on the context, for example:

[0159] In the acceptance scenario, picking up the medicine is followed by a flipping action, which is interpreted as checking the appearance of the medicine;

[0160] Confidence calculation: Assign a confidence score to each matching result, for example:

[0161] Gesture matching: 85%, trajectory similarity: 92%, timing consistency: 98%;

[0162] For example, the system recognizes the student's pressing gesture and, combined with the movement trajectory of the barcode scanner, determines that the student has performed a scanning operation.

[0163] Multi-dimensional matching result fusion:

[0164] Weighted scoring: Assign weights to matching results in different dimensions, for example:

[0165] Spatial matching: 40%;

[0166] Time matching: 30%;

[0167] Gesture matching: 30%;

[0168] Fuzzy logic judgment: handle partial matching situations, such as:

[0169] Trajectory deviation is 10%, but the gesture is correct → judged as basically compliant;

[0170] Rule engine decision: Determines the final matching result based on predefined rules, for example:

[0171] When the spatial matching degree is greater than 80% and the temporal matching degree is greater than 70%, a successful scan event is triggered;

[0172] If the key trigger point is not activated, the operation is considered incomplete;

[0173] Matching result visualization and feedback:

[0174] Real-time visualization: Matching areas and trigger points are highlighted in the virtual scene.

[0175] Operation prompts: Correct irregular actions through voice or text prompts, for example:

[0176] Please keep the scanner perpendicular to the barcode;

[0177] The operation speed is too fast, please slow down;

[0178] Error Marking: Records the error type and location, and generates detailed operation reports.

[0179] The beneficial effects of the above technical solution are: real-time comparison of trainee movements with GSP standards to ensure compliance with operational procedures, provision of targeted guidance based on matching results, accumulation of operational records for analysis of trainee weaknesses, and optimization of training content.

[0180] The present invention proposes a virtual simulation system for the entire process of GSP drug management based on human-computer interaction, wherein the analysis module includes:

[0181] The first comparison unit is used to determine the gesture interaction feedback result of the virtual character for the corresponding simulated work scenario according to the operation content, and compare and analyze the gesture interaction feedback result of the corresponding training simulation task to determine whether the difference between the gesture feedback results meets the first set standard;

[0182] If the first set standard is met, a satisfaction coefficient of 1 is assigned to the corresponding operation content;

[0183] Otherwise, according to the close relationship between the gesture feedback difference and the first set standard, a1 is assigned to the corresponding operation content, where a1 is a satisfaction coefficient based on the gesture;

[0184] The second comparison unit is used to determine the voice interaction feedback result of the corresponding simulation work scenario based on the voice input, and compare and analyze the voice interaction feedback result of the corresponding training simulation task to determine whether the difference between the two voice feedbacks meets the second set standard;

[0185] If the second set criterion is met, a satisfaction coefficient of 1 is assigned to the corresponding voice input;

[0186] Otherwise, according to the close relationship between the speech feedback difference and the second set standard, assign a2 to the corresponding speech input, where a2 is a speech-based satisfaction coefficient;

[0187] A coefficient comparison unit is used to compare the sizes of two satisfaction coefficients, and regard the interaction mode with the larger satisfaction coefficient as the main interaction logic judgment basis, and regard the interaction mode with the smaller satisfaction coefficient as the auxiliary interaction logic judgment basis;

[0188] Among them, if the two satisfaction coefficients are equal, both interaction modes are regarded as the main interaction logic judgment basis.

[0189] In this embodiment, the gesture interaction feedback result is the system's feedback result for the control command input by the gesture interaction. For example, after scanning the code, partial information of the drug should be displayed. The gesture theory feedback result: standard operation feedback that complies with the GSP specification, such as after scanning the code, all information of the drug should be displayed;

[0190] At this time, partial information of the displayed medicine is compared with all information of the displayed medicine to determine the control instructions under the gesture interaction corresponding to the undisplayed information, that is, the gesture action difference, and the gesture action difference is input into the gesture standard analysis model to determine whether the first set standard is met. The model is obtained by training the neural network model based on samples of different gesture interaction differences and whether the first set standard is met. Therefore, the judgment result can be obtained directly.

[0191] In this embodiment, a1=1-sim(F1(sf), G(sb)), where F1(sf) represents the representation based on the gesture feedback difference sf; and G(sb) represents the representation based on the first set standard sb.

[0192] The above representations are matched from a representation database, which contains different differences and standard correspondences.

[0193] The voice interaction feedback result is the system's recognition and response to the trainee's voice command, such as confirming the drug batch number and recording temperature and humidity. sim(g1,g0) represents the voice feedback difference (a value) based on the voice interaction feedback result g1 and the voice theory feedback result g0, and determines whether it is less than the threshold of the second set standard. If it is less than the threshold of the second set standard, a2 is 1, otherwise, Wherein, d2 is the threshold of the second setting standard, and its value is generally 0.1.

[0194] In this embodiment, the primary interaction logic judgment basis is the dominant information source when making a decision, and the auxiliary interaction logic judgment basis is the information used to supplement or verify the primary basis.

[0195] For example, the student's gesture shows that the scanning action is standard (a1=0.9), but the voice says that this is a qualified drug (the actual drug is unqualified) (a2=0.3). The system determines that the operation is incomplete and prompts the student to check the expiration date of the drug.

[0196] The beneficial effects of the above technical solution are: the dual-modal fusion decision-making mechanism significantly improves the intelligence and adaptability of the training system. Single-modal errors will not lead to total negation. The system can still make correct judgments based on the other modality and automatically adjust the evaluation strategy according to the students' advantageous interaction methods to improve the learning experience.

[0197] The present invention proposes a virtual simulation system for the entire process of GSP drug management based on human-computer interaction, wherein the analysis module further includes:

[0198] a weight determination unit, configured to assign a first weight to the gesture interaction and a second weight to the voice interaction according to a result of the interaction logic determination;

[0199] a gesture guidance unit, configured to determine a behavioral error starting point based on a difference in gesture interaction feedback and a behavioral difference determined based on the first matching point and the second matching point, and to determine a gesture guidance with the behavioral error starting point as a starting point based on the training trigger point set;

[0200] A voice guidance unit is used to retrieve the correct voice from the voice scenario interaction database under the corresponding training process and provide voice guidance based on the difference in the voice interaction feedback;

[0201] The combination unit is used to obtain correct guidance for the trainee's behavior in the current training process based on the gesture guidance, voice guidance, and the first weight and the second weight.

[0202] In this embodiment, the value of a1 / (a1+a2) is regarded as the first weight, and the value of a2 / (a1+a2) is regarded as the second weight.

[0203] In this embodiment, the behavioral error starting point is the time point when deviation first occurs in the operation, such as the action deformation starting from the third frame.

[0204] In this embodiment, we search forward from the current frame to locate the first action frame that does not meet the criteria, calculate key point position error, angle deviation and other indicators, and call a predefined correction template based on the error type, for example:

[0205] Trajectory error → display standard path, timing error → prompt correct operation sequence, force error → provide pressure feedback. For example, in the medicine warehousing scenario, the student's carrying action caused the virtual medicine box to tilt at an angle of more than 15° (the standard is ≤10°). The system analysis found that the angle of the medicine box was abnormal starting from the 5th frame, and generated instructions to keep the medicine box horizontal when carrying it.

[0206] During the acceptance inspection of refrigerated medicines, the trainee skipped the temperature detection step and directly put the medicines into storage. The system recognized that the corresponding trigger point was missing and prompted the trainee to check the temperature of the medicines first.

[0207] In this embodiment, the voice scenario interaction database includes guidance content for error types corresponding to different voice differences, and the error types are classified as follows: missing keywords → supplement key information, grammatical errors → provide correct sentence patterns, semantic deviations → clarify operation intentions, etc.

[0208] The beneficial effects of the above technical solution are: by accurately locating the starting point of the error, providing targeted correction suggestions, dynamically adjusting the guidance method according to the trainees' operating characteristics, adapting to different learning styles, and ensuring the trainees' GSP operating standard compliance rate and satisfaction through real-time multimodal feedback.

[0209] The present invention proposes a virtual simulation system for the entire process of GSP drug management based on human-computer interaction, which also includes:

[0210] Delay determination time, for collecting the initial acquisition time of the test voice signal by each array unit, and obtaining the propagation delay time of the corresponding array unit based on the distance between the designated position and each array unit and the initial utterance time of the test voice signal;

[0211] an acoustic determination unit, configured to determine acoustic features based on sound information of a test speech signal collected by each array unit;

[0212] a vector analysis unit, configured to mark the propagation delay time and the acoustic characteristics on corresponding array elements of the array structure to obtain an analysis group vector, and input the vector into a vector analysis model to determine a priority of each array element, wherein the priority includes a first priority and a second priority, and the first priority is superior to the second priority;

[0213] a screening unit, configured to perform N1 voice tests on the voice input device, count a first number of array elements for which a first priority is set, and regard elements for which a ratio of the first number to N1 is greater than a preset threshold as priority elements;

[0214] Among them, when collecting voice input, voice interaction analysis is performed based on the voice signal collected by the priority unit.

[0215] In this embodiment, the array unit is a single microphone or pickup module in the voice input device, such as 8 microphones in a ring microphone array. The test voice signal is a standard audio used to calibrate the system, such as playing a calibration voice with a fixed frequency. The initial acquisition time is the moment when the array unit starts to receive the voice signal, for example, microphone A captures the sound at 0.5 seconds. The designated position is the actual spatial coordinate of the voice, such as the trainee speaking at 1 meter away from the center of the array.

[0216] Propagation delay time refers to the difference between the actual transmission time and the theoretical transmission time of the same array unit. The propagation delay time is: Tc1-Tc0-L / vs, where Tc1 is the initial acquisition time, Tc0 is the initial sound emission time, L is the corresponding distance, and vs is the sound propagation speed.

[0217] In this embodiment, the vector analysis model is obtained by training the neural network model using the analysis group vector and the priority analysis results for each element in the vector as samples. Therefore, priority can be set directly for each array unit. The analysis group vector = {propagation delay time and acoustic characteristics of each array unit}. It should be noted that each array unit in the vector is numbered and placed in sequence to facilitate analysis and search.

[0218] In this embodiment, the preset threshold value is 2N1 / 3.

[0219] The beneficial effect of the above technical solution is: starting from the two aspects of propagation delay time and acoustic characteristics, it combines vector analysis to assign priorities to array units to ensure the reliability of subsequent voice interaction.

[0220] The present invention proposes a virtual simulation system for the entire process of GSP drug management based on human-computer interaction, wherein the analysis module further includes:

[0221] a probability determination unit, configured to synchronously split the gesture guidance and voice guidance in the correct behavior guidance according to the test point list under the corresponding training process, and predict the guidance success probability of each synchronous split block based on the historical guidance behavior characteristics of the trainee;

[0222] a test score determining unit, configured to determine a first test score for each synchronous split block according to a guidance success probability of each synchronous split block;

[0223] The strategy determination unit is used to determine the recommended guidance frequency of the corresponding synchronous split block from the test score-frequency database according to the first test score of each synchronous split block, and obtain a behavior guidance strategy.

[0224] In this embodiment, the examination point list is a set of examination points specified in the GSP training process, such as checking the expiration date of the drug and verifying the consistency of the batch number with the document under the drug acceptance process.

[0225] Correct behavioral guidance is system-generated guidance to correct students' incorrect operations, such as please keep the barcode scanner vertically (gesture guidance) and please say the drug batch number in full (voice guidance).

[0226] Synchronous splitting is to divide the gesture guidance and voice guidance into corresponding segments according to the test points, because each test point has three situations: only gestures, only voice, and both gestures and voice.

[0227] In this embodiment, the historical guidance behavior characteristics are the student's response data to different guidance in past learning, such as a student's acceptance success rate for guidance on excessive operation speed is 70%, and the acceptance success rate for guidance on position deviation is 85%.

[0228] The specific methods for obtaining the guidance success probability are as follows:

[0229] Obtain the interaction type and type weight of the test point corresponding to the synchronized split block from the test point-type database. Combine the historical guidance behavior characteristics and the biased behavior type of the split knowledge point involved in each interaction type to determine the guidance feature set for each interaction type. The guidance feature set includes: the historical guidance success probability for the biased behavior type of each split knowledge point under the corresponding interaction type;

[0230] In this embodiment, the test point-type database contains interaction types and type weights for different test points, which are pre-set, and each test point contains several split knowledge points, which are all pre-stored.

[0231] predicting a guidance success probability based on the guidance feature set;

[0232]

[0233] in, is the weight of the case where only gesture exists; is the weight of the case where only speech exists; ε s , ε y is the gesture weight and speech weight when both gesture and speech exist; z1 i1 、p1 i1 are the difficulty coefficient and the success probability of historical guidance of the i1th split knowledge point in the case of only gestures; m1, m2, and m3 respectively represent the total number of split knowledge points corresponding to the three cases (only gestures, only voice, and both gestures and voice), and m1 = m2 = m3; s1 k1 、p3 k1 They represent the difficulty coefficient and the success probability of historical guidance of the k1th split knowledge point based on the gesture direction when both gesture and speech exist; s2 k1 、p4 k1 They represent the difficulty coefficient and the success probability of historical guidance of the k1th split knowledge point based on the voice direction in the case of both gesture and voice; ε s , ε y They represent the weights based on gesture direction and speech direction in the case of both gesture and speech; A1 is the modal adaptation function; A2 is the forgetting function; E is the decay period; ΔT u1 represents the forgetting time related to the difficulty coefficient of the u1-th split knowledge point in the h1-th biased behavior type; d h1 represents the number of split knowledge points under the h1th biased behavior type in the synchronous split block; p h1represents the historical guidance success probability of the h1th biased behavior type in the synchronous split block; m4 represents the number of biased behavior types in the synchronous split block.

[0234] In this embodiment, pure gesture, pure voice, and mixed modality classification processing are performed by (unimodal weight) and ε s , ε y (Sub-weights under mixed modalities) accurately depict the differences in characteristics of different interaction types. The difficulty coefficient reflects the complexity of the knowledge point itself (e.g., "cold chain temperature calibration" is more difficult than "scanning code", and the corresponding difficulty coefficient is greater). The historical probability reflects the student's personalized ability (e.g., the higher the success rate of student A for "gesture guidance", the closer it is to 1). ΔT u1 It is a "difficulty-forgetting" correlation item (the more difficult the knowledge point, the faster it is forgotten).

[0235] In this embodiment, the corresponding value ranges of the difficulty coefficient, the historical guidance success probability, and the type weight are all between 0 and 1.

[0236] Through multimodal segmentation, dynamic history correction, and forgetting law modeling, it not only meets the complexity of practical training interaction, but also solves the core defects of traditional prediction. It is the key mathematical foundation supporting practical training guidance.

[0237] In this embodiment, the first test score = guidance success probability × 100 × coefficient of complex test point.

[0238] In this embodiment, the test score-frequency database is a pre-established mapping table that records the recommended guidance frequencies corresponding to different test score ranges. For example, a test score of 80-90 corresponds to a prompt once every three operations, and a test score of 90-100 corresponds to a prompt once every five operations.

[0239] The recommended guidance frequency is the frequency at which the system repeatedly displays certain guidance content to students, which is used to balance learning effect and interference level.

[0240] The behavioral guidance strategy is a personalized guidance plan formed by integrating the recommended guidance frequencies of all split blocks. For example, in the drug scanning process, the scanning angle guidance is prompted once every two operations, and the batch number verification guidance is prompted once every four operations.

[0241] The beneficial effects of the above technical solution are: customized guidance based on the students' historical behavior and test difficulty, making the guidance content more in line with actual needs, avoiding interference caused by excessive prompts, reducing ineffective repetitive guidance, and dynamically adjusting the guidance frequency. The system can quickly adapt to the learning rhythm of different students, continuously accumulate student behavior data, optimize the test score-frequency mapping rules and prediction models, improve training quality, and significantly enhance the mastery of key operational links.

[0242] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A virtual simulation system for the entire process of GSP drug management based on human-computer interaction, characterized by: include: A virtual simulation engine is used to realistically present pharmaceutical simulation work scenarios, wherein the pharmaceutical simulation work scenarios are related to each practical training process in the entire pharmaceutical training process; Interactive equipment, used to capture the trainee's current actions when the trainee is in a realistic medical simulation work scenario and obtain the trainee's actual operation content based on the trainee's current training process; A voice input device, used to receive voice input from the trainee during the current training process; An analysis module is used to perform interactive logical judgment on the operation content and voice input and provide behavioral feedback, determine the correct behavioral guidance for the trainee in the current training process, and generate a behavioral guidance strategy for the current training process; The guidance module is used to continuously guide the trainee based on the behavior guidance strategy until the trainee's behavior operation is standardized for the training simulation task of each training process.

2. The GSP drug management full process virtual simulation system based on human-computer interaction according to claim 1 is characterized in that: The interactive device includes: An acquisition unit is used to collect the skeleton key point data of the human body's current movement in real time. The skeleton key point data includes the coordinate information of 27 skeleton key points of the human hand in three-dimensional space; The model building unit is used to convert the rotation information of each skeletal key point into a quaternion representation based on the collected skeletal key point data, and combine it with the translation coordinates to build a quaternion motion model of the skeletal key point of the current action to describe the posture and position of each skeletal key point in three-dimensional space; A feature extraction unit is used to continuously obtain quaternion motion models of multiple action frames to form action sequence data, input the action sequence data into a pre-trained LSTM network, and extract temporal action features in the action sequence, including the order of actions, duration, and time-dependent information of rhythm changes; The spatial mapping unit is used to map the skeleton key point information of human body movements in the real world to the virtual character in the virtual space based on the extracted temporal motion features and quaternion motion model.

3. The GSP drug management full process virtual simulation system based on human-computer interaction according to claim 2 is characterized in that: The interactive device further includes: The locking unit is used to sequentially lock the position of each skeleton key point in the continuous action frames of the virtual character, and construct the movement vector Yx={w t1i , i=1,2,3,...,Tn} and the action gesture Ds of each action frame={r t2j ,j=1,2,3,...,27}, where w t1i Represents the position coordinates of the corresponding skeleton key point at time t1i in the i-th action frame; r t2j Represents the position coordinates of the jth skeleton key point at time t2j in the corresponding action frame; A point set construction unit is used to determine a standard vector of each skeletal key point and a training trigger point set according to the standard operating behavior of the simulation work scenario constructed based on the simulation training task of the current training process, wherein the training trigger point set includes the trigger position point, trigger duration and trigger sequence of the training trigger point; A matching unit is used to match each motion vector with the corresponding standard vector once, and at the same time, to match each motion vector and the action gesture of each action frame with the training trigger point set twice in sequence; The content determination unit is used to screen the first matching point under each action frame according to the primary matching result and screen the second matching point under each action frame according to the secondary matching result to determine the operation content of the trainee.

4. The GSP drug management full process virtual simulation system based on human-computer interaction according to claim 1 is characterized in that: The analysis module includes: The first comparison unit is used to determine the gesture interaction feedback result of the virtual character for the corresponding simulated work scenario according to the operation content, and compare and analyze the gesture interaction feedback result of the corresponding training simulation task to determine whether the difference between the gesture feedback results meets the first set standard; If the first set standard is met, a satisfaction coefficient of 1 is assigned to the corresponding operation content; Otherwise, according to the close relationship between the gesture feedback difference and the first set standard, a1 is assigned to the corresponding operation content, where a1 is a satisfaction coefficient based on the gesture; The second comparison unit is used to determine the voice interaction feedback result of the corresponding simulation work scenario based on the voice input, and compare and analyze the voice interaction feedback result of the corresponding training simulation task to determine whether the difference between the two voice feedbacks meets the second set standard; If the second set criterion is met, a satisfaction coefficient of 1 is assigned to the corresponding voice input; Otherwise, according to the close relationship between the speech feedback difference and the second set standard, assign a2 to the corresponding speech input, where a2 is a speech-based satisfaction coefficient; A coefficient comparison unit is used to compare the sizes of two satisfaction coefficients, and regard the interaction mode with the larger satisfaction coefficient as the main interaction logic judgment basis, and regard the interaction mode with the smaller satisfaction coefficient as the auxiliary interaction logic judgment basis; Among them, if the two satisfaction coefficients are equal, both interaction modes are regarded as the main interaction logic judgment basis.

5. The GSP drug management full process virtual simulation system based on human-computer interaction according to claim 4 is characterized in that: The analysis module further includes: a weight determination unit, configured to assign a first weight to the gesture interaction and a second weight to the voice interaction according to a result of the interaction logic determination; a gesture guidance unit, configured to determine a behavioral error starting point based on a difference in gesture interaction feedback and a behavioral difference determined based on the first matching point and the second matching point, and to determine a gesture guidance with the behavioral error starting point as a starting point based on the training trigger point set; A voice guidance unit is used to retrieve the correct voice from the voice scenario interaction database under the corresponding training process and provide voice guidance based on the difference in the voice interaction feedback; The combination unit is used to obtain correct guidance for the trainee's behavior in the current training process based on the gesture guidance, voice guidance, and the first weight and the second weight.

6. The GSP drug management full process virtual simulation system based on human-computer interaction according to claim 1 is characterized in that: Also includes: Delay determination time, for collecting the initial acquisition time of the test voice signal by each array unit, and obtaining the propagation delay time of the corresponding array unit based on the distance between the designated position and each array unit and the initial utterance time of the test voice signal; an acoustic determination unit, configured to determine acoustic features based on sound information of a test speech signal collected by each array unit; a vector analysis unit, configured to mark the propagation delay time and the acoustic characteristics on corresponding array elements of the array structure to obtain an analysis group vector, and input the vector into a vector analysis model to determine a priority of each array element, wherein the priority includes a first priority and a second priority, and the first priority is superior to the second priority; a screening unit, configured to perform N1 voice tests on the voice input device, count a first number of array elements for which a first priority is set, and regard elements for which a ratio of the first number to N1 is greater than a preset threshold as priority elements; Among them, when collecting voice input, voice interaction analysis is performed based on the voice signal collected by the priority unit.

7. The GSP drug management full process virtual simulation system based on human-computer interaction according to claim 5 is characterized in that: The analysis module further includes: a probability determination unit, configured to synchronously split the gesture guidance and voice guidance in the correct behavior guidance according to the test point list under the corresponding training process, and predict the guidance success probability of each synchronous split block based on the historical guidance behavior characteristics of the trainee; a test score determining unit, configured to determine a first test score for each synchronous split block according to a guidance success probability of each synchronous split block; The strategy determination unit is used to determine the recommended guidance frequency of the corresponding synchronous split block from the test score-frequency database according to the first test score of each synchronous split block, and obtain a behavior guidance strategy.

8. The GSP drug management full process virtual simulation system based on human-computer interaction according to claim 3 is characterized in that: The matching unit includes: an overlapping segment determining subunit, configured to determine a relative overlapping segment between a moving trajectory of a corresponding moving vector and a standard trajectory of a standard vector; a judging subunit, configured to judge whether the relatively overlapping segment contains a key interaction point in the standard vector; If included, retaining the motion vector based on the partial vector under the relatively overlapping segment; If not included, it is determined that the first matching point does not exist in the corresponding matching result; Among them, the retained part of the vector is used as a matching result.

Images