A virtual reality-based medicine taking training method and system, a terminal and a medium

Abstract

This invention provides a virtual reality-based drug retrieval training method, system, terminal, and medium, specifically relating to the fields of medical information technology and biomedical engineering technology. The solution includes: constructing several initial drug packaging models based on preset multimodal drug characteristics; constructing a drug database based on the initial drug packaging models and key feature information of several drugs, including target drugs and interfering drugs; rendering a virtual pharmacy environment using virtual reality equipment based on the drug database; and, in response to receiving a retrieval instruction for at least one target drug, performing human-computer interaction using the virtual reality equipment based on the retrieval instruction and the virtual pharmacy environment to obtain the retrieval result. This solution, by constructing a drug database with diverse packaging types and using virtual reality technology to create a realistic virtual pharmacy scene, enables medical personnel to perform immersive drug retrieval operations and training in a virtual environment, thereby improving the efficiency of drug retrieval for medical personnel.

Description

Technical Field

[0001] This invention relates to the fields of medical information technology and biomedical engineering technology, and in particular to a method, system, terminal and medium for drug dispensing training based on virtual reality. Background Technology

[0002] Currently, many traditional drug identification training programs rely heavily on physical simulations or paper-based textbooks. While this approach is helpful for imparting basic knowledge, it has several limitations in practical application. First, traditional training is often conducted in static classroom or laboratory environments, failing to simulate the complex and ever-changing work scenarios of a real pharmacy. Second, physical simulation training requires a large number of physical drugs, equipment, and classroom setups, resulting in high training costs.

[0003] In recent years, some studies have employed traditional medical visual visualization techniques, using graphics and images to present data and information to help people understand medical content. However, these visualization techniques primarily rely on a two-dimensional plane, that is, using text lists and graphics to present medical information, resulting in low efficiency in practical applications and difficulty in adapting to high-pressure situations. It is evident that existing technologies cannot simulate the complex and ever-changing work scenarios in real pharmacies, leading to low efficiency in drug search training. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method, system, terminal and medium for drug retrieval training based on virtual reality, which aims to solve the problem that the efficiency of drug retrieval training is low because it cannot simulate the complex and ever-changing working scenarios in a real pharmacy.

[0005] To achieve the above objectives, a first aspect of the present invention provides a drug retrieval training method based on virtual reality, comprising: Based on the pre-defined multimodal drug characteristics, several initial drug packaging models are constructed. Based on the initial drug packaging model and key feature information of several drugs, a drug database is constructed, including target drugs and interfering drugs. Based on the aforementioned drug database, a virtual pharmacy environment is rendered using virtual reality equipment; In response to receiving a medication retrieval instruction for at least one target drug, the system uses the virtual reality device to perform human-computer interaction based on the retrieval instruction and the virtual pharmacy environment to obtain the medication retrieval result.

[0006] Optionally, based on preset multimodal drug characteristics, several initial drug packaging models are constructed, including: The text information of the medicine is processed based on a preset text enhancement algorithm to obtain several medicine font styles; Based on the font style of the drug, the preset packaging appearance and / or the preset color coding format, several initial drug packaging models are constructed.

[0007] Optionally, the step of constructing a drug database based on the initial drug packaging model and key feature information of several drugs includes: The initial drug packaging model is fused with the key feature information of each drug to obtain several styles of drug packaging models corresponding to each drug. A drug database is constructed based on all the aforementioned drug packaging models.

[0008] Optionally, the step of rendering a virtual pharmacy environment using a virtual reality device based on the drug database includes: An initial virtual pharmacy environment was rendered using virtual reality equipment; According to the preset deployment principles, the drug packaging models in the drug database are deployed in the initial virtual pharmacy environment to render the virtual pharmacy environment.

[0009] Optionally, the preset deployment principles specifically include: The interfering drugs in the drug database include at least one similar drug with a similar name and the same drug packaging model style as the target drug, and include multiple dissimilar drugs with dissimilar names and random drug packaging model styles as the target drug. The packaging model of the target drug is randomly selected, and the packaging model of the target drug is the same as the packaging model of at least one similar drug. The packaging models of several dissimilar drugs are randomly selected and randomly arranged on the shelves in the initial virtual pharmacy environment.

[0010] Optionally, the step of obtaining the medication retrieval result by using the virtual reality device for human-computer interaction based on the medication retrieval instruction and the virtual pharmacy environment includes: In response to receiving an operation action performed by the operator corresponding to the virtual reality device in accordance with the medicine retrieval instruction and preset dynamic interference factors; Based on the operation, motion capture data and drug placement location data are generated in the virtual pharmacy environment; Based on the motion capture data and the drug placement location data, the drug retrieval result is obtained.

[0011] Optionally, obtaining the medication retrieval result based on the motion capture data and the medication placement location data includes: Based on the target position of the target drug on the target shelf and the motion capture data, the drug to be grasped is determined; Based on the designated placement location corresponding to the target drug and the drug placement location data, the correctness of the drug placement location is determined; The result of drug retrieval is obtained based on the correctness of the grasped drug and its placement position.

[0012] A second aspect of the present invention provides a virtual reality-based drug dispensing training system, the system comprising: The initial drug packaging model construction module is used to construct several initial drug packaging models based on preset multimodal drug characteristics. A drug database construction module is used to construct a drug database based on the initial drug packaging model and key feature information of several drugs, wherein the drugs include target drugs and interfering drugs; The virtual pharmacy environment rendering module is used to render a virtual pharmacy environment based on the drug database using virtual reality equipment. The medication dispensing module is used to respond to receiving a medication dispensing instruction for at least one target drug, and based on the medication dispensing instruction and the virtual pharmacy environment, to perform human-computer interaction using the virtual reality device to obtain the medication dispensing result.

[0013] A third aspect of the present invention provides a smart terminal, the smart terminal including a memory for storing executable instructions; and a processor for calling and running the executable instructions in the memory to perform any of the steps of the above-described virtual reality-based drug retrieval training method.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing program instructions that, when executed by a processor, implement any of the steps of the above-described virtual reality-based drug retrieval training method.

[0015] Compared with existing technologies, the beneficial effects of this solution are as follows: This invention constructs several initial drug packaging models based on preset multimodal drug characteristics, and integrates these initial drug packaging models with key drug feature information to build a drug database with a rich variety of drug packaging types. Based on this drug database, it uses virtual reality technology to create realistic virtual pharmacy scenes, enabling medical personnel to perform immersive drug dispensing operations and training in a virtual environment, which helps improve the efficiency of drug dispensing for medical personnel. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the virtual reality-based drug dispensing training method of the present invention; Figure 2 These are schematic diagrams of different pharmaceutical packaging models of the present invention; Figure 3 Schematic diagrams of drug packaging models corresponding to different text enhancement methods of the present invention; Figure 4 This is a schematic diagram of a three-dimensional pharmacy model according to the present invention; Figure 5 This is a schematic diagram of the target drug list of the present invention; Figure 6 This is a schematic diagram of the countdown interference display of the present invention; Figure 7 This is a schematic diagram illustrating the interruption interference display of the present invention; Figure 8 This is a schematic diagram of the drug dispensing training system module based on virtual reality of the present invention; Figure 9 This is a schematic diagram of the intelligent terminal structure of the present invention. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0019] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0025] This invention addresses the problem of low efficiency in drug retrieval training due to the inability to simulate the complex and ever-changing work scenarios of a real pharmacy in existing technologies. It proposes a virtual reality-based drug retrieval training method, primarily employing an immersive virtual environment design combined with dynamic interference factors (such as time interference and interruption interference) to simulate real-world work situations. This enhances the adaptability of medical personnel under high pressure and complex environments, improving the operator's experience. Simultaneously, this invention introduces multimodal drug feature display, including packaging form, text enhancement features, and color coding, to improve drug identification accuracy. Furthermore, it combines motion capture with drug recognition logic to accurately capture drug retrieval actions and efficiently verify drug identification. Therefore, this invention, by integrating advanced virtual reality technology and interactive design concepts, provides medical personnel with a simulated training environment aimed at improving the accuracy and efficiency of drug retrieval, thereby reducing medication errors and improving patient safety and treatment quality.

[0026] This invention provides a virtual reality-based medication dispensing training method, deployed on head-mounted display devices such as VR glasses and AR / MR headsets, and applied to scenarios where operators undergo medication dispensing training in a three-dimensional virtual pharmacy by wearing the headset. Specifically, such as... Figure 1 As shown, the steps of the method in this embodiment include: Step S100: Based on the preset multimodal drug characteristics, construct several initial drug packaging models; Specifically, in order to construct a drug packaging that closely resembles a realistic visual effect, this embodiment selects several visual features related to real drug packaging as preset multimodal drug features. Using image processing and 3D modeling methods, these multimodal drug features are combined in different arrangements to construct several initial drug packaging models, ensuring that each initial drug packaging model possesses a high degree of realism and recognizability. These multimodal drug features include, but are not limited to, the shape, color, pattern, and packaging material (such as plastic, glass, aluminum foil, etc.) of the drug packaging.

[0027] Step S200: Based on the initial drug packaging model and key feature information of several drugs, construct a drug database, wherein the drugs include target drugs and interfering drugs; Specifically, this embodiment uses information such as drug name, drug specifications (e.g., dosage, unit), manufacturer, drug use, indication, or expiration date as key feature information for each drug. This key feature information is then combined with an initial drug packaging model to obtain several drug packaging models corresponding to each drug. The drug database is constructed using the drug packaging models corresponding to all target drugs and interfering drugs used to build the database. Target drugs refer to one or more drugs included in the current drug retrieval task; these are the drugs to be identified and retrieved. Interfering drugs refer to drugs in the drug database other than the target drugs, used to simulate potential confusion factors in a real pharmacy environment, thereby enhancing the system's robustness and the operator's identification ability.

[0028] Step S300: Based on the drug database, render a virtual pharmacy environment using virtual reality equipment; Specifically, to construct a pharmacy environment close to a real-world scenario, this embodiment includes a medication list module, a shelf-based medication retrieval module, and a window-based medication delivery module within the virtual reality device. The medication list module renders a medication retrieval task list for the operator, including information such as the doctor's workstation, computer monitor, and reminder indicator lights, helping the operator clearly understand the medication retrieval task. The shelf-based medication retrieval module constructs a virtual pharmacy environment, including at least one pharmacy, medications from a drug database, shelves, and a workstation, allowing the operator to search for and interact with medications within this virtual pharmacy environment. The window-based medication delivery module includes a computer monitor, reminder indicator lights, a pharmacy window, and medication containers, triggering the medication retrieval task and providing a placement location for the operator, facilitating medication retrieval. Therefore, the virtual pharmacy environment rendered using virtual reality (VR) technology includes physical elements such as the pharmacy layout, shelves, medication placement, workstation, computer monitor, reminder indicator lights, pharmacy window, and medication containers. In this virtual pharmacy, the operator can freely browse, search, and select medications, providing an immersive medication retrieval experience.

[0029] Step S400: In response to receiving a drug retrieval instruction for at least one target drug, based on the drug retrieval instruction and the virtual pharmacy environment, human-computer interaction is performed using the virtual reality device to obtain the drug retrieval result.

[0030] Specifically, when the operator receives a medication retrieval instruction corresponding to the current medication retrieval task displayed on the computer monitor placed on the workbench via VR device, the system immediately responds and initiates the human-computer interaction process. The current medication retrieval task includes one or more target medications. The human-computer interaction process mainly includes: First, the system plans an optimal medication retrieval path for the operator based on the medication retrieval instruction and the real-time status of the virtual pharmacy environment. Then, through visual and auditory feedback from the VR device, the operator is guided along the optimal retrieval path and locates the target medication on the shelf. Next, the target medication is delivered to the pharmacy window, and the operator completes the current medication retrieval task by scanning the medication's barcode or QR code, confirming the medication information, and placing the medication into the designated medication container. Finally, based on the operator's operation results, the system updates the virtual pharmacy environment and medication retrieval status in real time and ultimately generates the medication retrieval result.

[0031] Furthermore, to ensure the accuracy and safety of medication dispensing, the system performs multiple verifications and confirmations during the process. For example, when the user selects a medication, the system will double-check the medication information to ensure it is correct; when the user places the medication into the dispensing basket, the system will check whether the quantity and type of medication meet the requirements.

[0032] In this embodiment, several initial drug packaging models are constructed based on preset multimodal drug characteristics. These initial drug packaging models are then fused with key drug feature information to create a drug database with a rich variety of packaging types. Based on this database, virtual reality technology is used to create a realistic virtual pharmacy scene, enabling medical personnel to perform immersive medication dispensing operations and training in a virtual environment, thus improving their medication dispensing efficiency. Furthermore, the system can quickly adjust and switch to different medication dispensing instructions and virtual pharmacy environments according to varying training needs and conditions, which helps improve the skill level and clinical adaptability of medical personnel.

[0033] In a preferred embodiment, step S100 involves constructing several initial drug packaging models based on preset multimodal drug characteristics, including: Step S110: Process the text information of the medicine based on the preset text enhancement algorithm to obtain several medicine font styles; Step S120: Based on the font style of the medicine, the preset packaging appearance and / or the preset color coding format, construct several initial medicine packaging models.

[0034] Specifically, this embodiment first employs a text enhancement algorithm to perform character transformations, font style adjustments, and layout optimizations on the text information used on drug packaging, resulting in several drug font styles. For example, character transformations such as bolding the drug name or specifications, increasing the font size by 1.25 times, and highlighting the font in red are used to enhance the readability and attractiveness of the text by emphasizing key information. Then, based on the type and purpose of the medicine, select an appropriate packaging appearance (such as bottled, boxed, or bagged packaging), and design the size, shape, and material of the packaging to meet the storage, transportation, and usage requirements of the medicine. Based on the importance of the text, assign different color codes to different textual information to highlight important information.

[0035] Finally, the designed drug font style was applied to the text information on the packaging using 3D modeling software to ensure that the information was clear and easy to identify. The layout and arrangement of the text on the packaging were further adjusted to coordinate with the packaging appearance and color coding format, generating several initial drug packaging models.

[0036] In this embodiment, the text on the drug packaging is enhanced based on the drug font style and color coding format, and the packaging appearance is designed according to the type and purpose of the drug to construct a variety of initial drug packaging models, providing a more realistic drug packaging effect for the drugs on the shelves in the virtual pharmacy environment.

[0037] In a preferred embodiment, step S200, which involves constructing a drug database based on the initial drug packaging model and key feature information of several drugs, includes: Step S210: The initial drug packaging model is fused with the key feature information of each drug to obtain several styles of drug packaging models corresponding to each drug. Step S2220: Construct a drug database based on all the described drug packaging models.

[0038] Specifically, for each drug, based on its key characteristic information, such as drug name, specifications (e.g., dosage, unit), manufacturer, use, indications, or expiration date, this key characteristic information is applied to each of the initial drug packaging models constructed above. This results in a corresponding drug packaging model for each initial drug packaging model. In other words, each drug has a corresponding number of drug packaging models of the same type as the initial drug packaging models, resulting in a variety of different styles of drug packaging models. This enhances the diversity of drug packaging models and increases the difficulty of identifying the target drug. Then, based on the drug packaging model information corresponding to all types of drugs, a drug database layout structure is designed to completely store and display the drug packaging model information for each drug, ensuring the accuracy and completeness of the data. It should be noted that the drug packaging model in this embodiment refers broadly to various types of packaging appearance forms used to package various types of drugs.

[0039] For example, taking berberine hydrochloride tablets as the target drug, the design is as follows: Figure 2 The schematic diagrams of the four different drug packaging models are described; based on a similar drug name paired with berberine hydrochloride tablets (such as berberine hydrochloride tablets), the following design is proposed: Figure 3 The diagram shows the drug packaging models corresponding to the three different text enhancement methods. Figure 3 (a) in the text represents a form of text enhancement using similar drug names, with the target drug name on the left and the confusing drug name on the right; Figure 3 (b) in the text indicates the form of text enhancement using text encoding. The left side is the target drug name and the right side is the obfuscated drug name. The text encoding uses the hexadecimal number #FF0000 (i.e., red text). Figure 3 (c) in the text indicates a form of text enhancement using enlarged fonts, with the target drug name on the left and the confusing drug name on the right.

[0040] In this embodiment, by fusing the initial drug packaging model with the key feature information of each drug, multiple styles of drug packaging models corresponding to each drug are obtained, thereby increasing the difficulty of identifying the target drug. Furthermore, a complete drug database is constructed using all drug packaging models of all drugs, enabling operators to interact with drug packaging models in a virtual pharmacy environment, thereby increasing the intuitiveness and realism of reading drug packaging text.

[0041] In a preferred embodiment, step S300, which involves rendering a virtual pharmacy environment using a virtual reality device based on the drug database, includes: Step S310: Render the initial virtual pharmacy environment using a virtual reality device; Step S320: According to the preset deployment principle, the drug packaging models in the drug database are deployed in the initial virtual pharmacy environment to render the virtual pharmacy environment.

[0042] Specifically, by setting up a drug list module, a drug shelf retrieval module, and a drug delivery window module in a virtual reality device, an initial virtual pharmacy environment is rendered using virtual reality (VR) technology. This initial virtual pharmacy environment includes physical elements such as the pharmacy layout, drug shelves, workbenches, computer monitors, indicator lights, pharmacy windows, and drug containers, but does not include drugs from the drug database. Then, based on the drug placement requirements, a placement principle is designed. This principle includes that the drugs placed on the shelves in the same pharmacy should include all target drugs, at least one similar interfering drug, and multiple dissimilar interfering drugs. Similar drugs refer to drugs with names similar to the target drug and the same packaging model, while dissimilar drugs refer to drugs with names dissimilar to the target drug and random packaging models. The packaging model of the target drug is randomly selected, and the packaging model of the target drug is the same as the packaging model of at least one similar drug. The packaging models of several dissimilar drugs are randomly selected. The packaging models in the drug database are deployed in the initial virtual pharmacy environment to render the virtual pharmacy environment. The environment is then displayed through an immersive VR data display module set in the virtual reality device, so that the operator wearing the virtual reality device can immerse themselves in the training process of picking up medicine in the three-dimensional virtual pharmacy environment.

[0043] In this embodiment, the medicines arranged on the shelves of the same pharmacy in the virtual pharmacy environment include both the target medicine and different types of interfering medicines. The deceptiveness is increased by arranging medicine packaging models corresponding to similar medicines, and the visual interference is increased by arranging medicine packaging models of different styles corresponding to various dissimilar medicines. This improves the realism of the rendered virtual pharmacy environment and increases the difficulty of finding the target medicine, which is conducive to improving the effectiveness of medicine retrieval training for medical staff.

[0044] In a preferred embodiment, step S400, based on the medication dispensing instruction and the virtual pharmacy environment, utilizes the virtual reality device for human-computer interaction to obtain the medication dispensing result, including: Step S410: In response to receiving the operation action performed by the operator corresponding to the virtual reality device according to the medicine retrieval instruction and preset dynamic interference factors; Specifically, by setting up an external factor interference module in the virtual reality device, dynamic interference factors in the real work environment are simulated. For example, a countdown timer can be added to limit the time for executing a medication retrieval task, and / or task interruption interference (such as introducing calculation problems) can be added to interfere with the operator's memory accuracy in finding the target medication, thereby increasing the complexity and realism of the training. During the execution of a medication retrieval task, the system responds to the operator's actions according to the medication retrieval instructions and preset dynamic interference factors received from the virtual reality device. Image recognition technology is used to analyze the appearance features of the medication packaging model (including packaging shape, color, text style, etc.). Under the constraints of the dynamic interference factors, human-computer interaction occurs with the medication packaging model, such as grasping, moving, or placing the medication. This embodiment increases the training difficulty by designing various types of interference factors, enabling the system to simulate more complex situations in a realistic work environment, which is beneficial for improving the adaptability of medical personnel.

[0045] It is easy to understand that this embodiment only uses time interference and interruption interference as examples of interference types. In practical applications, other types of interference factors can be set according to the actual drug retrieval environment to simulate drug retrieval scenarios under different interference factors, so as to ensure the effectiveness and flexibility of the system in simulation training.

[0046] Step S420: Based on the operation, generate motion capture data and drug placement location data in the virtual pharmacy environment; Specifically, by setting up a drug retrieval action execution and recognition module in a virtual reality device, this module utilizes virtual reality motion capture technology to perform drug retrieval-related actions along a preferred retrieval path on a pharmacy shelf. These actions include grasping, moving, and placing the drug. Motion capture data and drug placement location data are generated based on these actions. The motion capture data represents the actions of moving and grasping the drug packaging model along the preferred retrieval path; the drug placement location data represents the location where the grasped drug packaging model is placed. Therefore, this embodiment uses virtual reality motion capture technology and image recognition technology to assist in the logical judgment of the entire drug retrieval process, which is beneficial for accurately determining the completion status of the drug retrieval task.

[0047] Step S430: Based on the motion capture data and the drug placement location data, obtain the drug retrieval result.

[0048] Specifically, based on the target drug's location on the target shelf and motion capture data, the system determines the key characteristics of the drug to be grasped, such as drug name, specifications (e.g., dosage, unit), manufacturer, intended use, indication, or expiration date. Based on the designated placement location of the target drug and the drug's placement location data, the system verifies the correctness of the placement location to determine if it meets the requirements of the current task. Based on the correctness of the grasped drug and its placement location, the system obtains the drug retrieval result and determines the task completion status. If the grasped drug is the target drug and its placement location meets the requirements, the system's identification of the target drug is correct, and the task is marked as complete. If the grasped drug is not the target drug or its location is incorrect, the system's identification of the target drug is incorrect, and a drug retrieval error message is sent to prompt the operator to correct the error and re-execute the drug retrieval task.

[0049] Furthermore, detailed training reports can be generated based on the system's accuracy in identifying target drugs and the speed of drug dispensing. These reports include the completion time for each task, drug identification accuracy, error type, and frequency, serving as statistical results for training assessment and providing a basis for subsequent training effectiveness evaluation and personalized feedback. Video tutorials can also be generated and online learning platforms can be built to guide medical personnel in drug identification through video instruction.

[0050] It should be noted that the above embodiments of the present invention are merely exemplary embodiments and are not intended to limit the scope of protection of the present invention. Any display method of a pharmaceutical packaging model sample employing the method of the present invention, or containing similar text enhancement methods, or containing packaging design concepts similar to pharmaceutical packaging design, or the interactive logic of the present invention, are all within the scope of protection of the present invention.

[0051] In summary, the interactive logic of the medication dispensing training method of the present invention is mainly divided into the following three stages: (1) Teaching scenario: The operator interacts with the objects on the table. The purpose of this scenario is to familiarize the operator with the basic operations of the VR device (such as grabbing objects, clicking buttons, moving the body, etc.). After completing the teaching scenario, the operator will enter the test scenario.

[0052] (2) Test scenario: This scenario is a simplified version of the formal experimental scenario. The operator needs to complete 3 tasks. First, the operator enters the prescription scenario, stands in front of the monitor workbench, and presses the red button placed on the workbench. Then, the computer screen next to him will display the name of a household item as the search object. The operator needs to enter the corresponding pharmacy to search for the item. After finding the target item, return to the workbench and place it in the designated medicine basket; the first task is complete. Next, the operator presses the red button on the workbench again, and the computer screen will display the second task: finding everyday items. When the operator enters the pharmacy and opens the door, a countdown timer will appear directly above their field of vision, starting the countdown. After finding the target item, return to the workbench and place it in the designated medicine basket; the second task is complete. Finally, the operator presses the red button on the workbench again, and the computer screen will display the third task: finding everyday items. After the operator enters the pharmacy and searches for 5 seconds, two two-digit addition and subtraction problems will appear on the pharmacy's wall-mounted TV to interrupt the search. The operator must click "Yes" or "No" to answer the questions before continuing. After finding the target item, return to the workbench and place it in the designated medicine basket; the third task is complete. The test scenario ends.

[0053] (3) Formal Experimental Scenario: Each operator will randomly enter an experimental scenario with interference conditions, namely a time interference experimental scenario or a calculation problem interference experimental scenario. The task of retrieving medicine will be repeated 72 times in the test scenario, and all daily necessities in the scenario will be replaced with medical medicines.

[0054] As an exemplary example, in a formal experimental scenario, in order to ensure the randomness of the appearance of the target drug packaging and the uniformity of the drug packaging enhancement forms, so that no other interference factors appear during the entire task execution process except for the two interference modes of time interference and interruption interference, the following randomness conditions and uniformity conditions are set, specifically as follows.

[0055] Randomization conditions: Assume there are 12 target pillboxes designed for each target drug. There are also 12 similar drug boxes (B) with names identical to each of the 12 target pillboxes. When the first target pillbox A appears, its paired similar pillbox B will also appear on the same pharmacy shelf. Each person performs one set of experiments, each set containing 72 tasks: 12 target drug packaging models A * 3 text enhancement methods * 2 interference modes = 72 tasks. In other words, the operator performs 72 tasks in each set. In each task, one target pillbox A, one similar pillbox B with the same name and packaging as the target drug, and 38 randomized pillboxes (with names dissimilar to the target drug) will appear on the shelf. These 38 randomized pillboxes are randomly arranged in each task.

[0056] The identical text enhancement conditions are set as follows: The text enhancement style for all medications in each of the three pharmacies is the same, including 38 misleading drug names. For example, all medications in pharmacy 1 have black text enhancement, all medications in pharmacy 2 have red text enhancement, and all medications in pharmacy 3 have 1.25x black text enhancement. Secondly, target drug A and its corresponding misleading drug B appear in the same packaging. Furthermore, when entering a pharmacy to find medication, the target drug name will uniformly disappear from the display screen.

[0057] In the verification of this invention, a highly realistic virtual scene is used, incorporating interference factors to allow operators to perform tasks in a lifelike environment, creating a strong sense of immersion. Experimental scenes can be quickly switched according to training requirements (e.g., switching between different text enhancement methods) to test the operator's speed and accuracy in finding medicine boxes packaged with different text enhancement methods. This invention uses virtual reality technology to build a virtual pharmacy environment, providing a highly realistic setting that allows medical personnel to select medicines in a virtual environment, thereby reducing the risk of errors in a real environment and improving the safety and efficiency of medication training.

[0058] The following experiment demonstrates the operation flow of the method of this invention by constructing a virtual pharmacy environment and several drug packaging models to perform a drug retrieval task: Step 1: System Initialization and Setup This experiment uses the Oculus Quest2 VR headset. Participants will wear the headset and use two controllers for the VR experience. This experiment provides a VR-based visual experience system, mainly including: a medicine list module, a medicine shelf retrieval module, a window medicine delivery module, an immersive VR data display module, and an external factor interference module. A 3D pharmacy model is built in a virtual reality environment, including a pharmacy window model, a pharmacy waiting hall model, a pharmacy model, pharmacy medicine and pharmacy display workbench models, medicine shelves, computers, tables, signs, ceiling lights, doors, walls, etc. Figure 4 The image shown is a schematic diagram of a three-dimensional pharmacy model. The operator will begin the formal experiment after understanding the task requirements and entering their operator number.

[0059] Three scenarios for the medicine retrieval task are set up: a scenario without external interference, a scenario with a countdown timer, and a scenario with interruption interference involving a calculation problem.

[0060] Step 2: Configure the drug list module A computer monitor is placed on the doctor's workbench, loading a list of medications to be picked up. The monitor displays the doctor's medication pickup task list, and indicator lights are used to notify the participant when the medication pickup task begins. Figure 5 The diagram shows a list of target drugs. It should be noted that... Figure 5 The only part that needs to be focused on is the text displayed on the computer monitor. The text content in other areas is not the research target and is only shown here for illustration purposes.

[0061] Step 3: Configure the medicine shelf dispensing module The virtual pharmacy environment features three separate pharmacies with a total of 240 types of medications, randomly placed on shelves. Each medication comes in three different packaging formats. Users must move to their designated pharmacy and retrieve their medication from the shelves.

[0062] Step 4: Configure the window-based medicine delivery module The system includes a pharmacy window and medicine baskets, allowing users to place their medications in the designated baskets on the workbench after completing their medication collection task.

[0063] Step 5: External Factor Interference Module Both Test Environment 1 and Test Environment 2 will incorporate time-related interference and work interruption interference. During the period when the tester performs steps 2 to 4, the system will invoke algorithms such as countdown timers or random calculation problems to simulate interference factors in a real work environment. Figure 6 The image shown is a schematic diagram of the countdown interference display. Figure 7 The image shown is a schematic diagram illustrating the interruption display in the form of a calculation problem. It should be noted that... Figure 6The only part that needs attention is the countdown timer. The text on the medicine packaging models on the shelf can be clearly seen when the operator zooms in, and is only shown here as an illustration.

[0064] Step 6: Training Data Recording: Immersive VR Data Display Module The system records each user's medication retrieval process data in real time, including user ID, room number, start time, end time, target medication, misleading medication, enhanced font mode, medication packaging type, stress mode, time taken per retrieval, submitted medication name, answer to a calculation problem, remaining time on the timer, and whether the countdown task failed. This data is transmitted in real time via the Oculus Quest2 VR device and recorded in the computer system's backend.

[0065] Through the above implementation examples, the present invention can efficiently and safely train medical personnel in medication operation in a virtual reality environment, improve their accuracy and response capabilities in a real environment, thereby reducing the probability of medication errors and improving medical safety.

[0066] like Figure 8 As shown, corresponding to the above-mentioned virtual reality-based drug retrieval training method, this embodiment of the invention also provides a virtual reality-based drug retrieval training system, which includes: The initial drug packaging model construction module 810 is used to construct several initial drug packaging models based on preset multimodal drug features; The drug database construction module 820 is used to construct a drug database based on the initial drug packaging model and key feature information of several drugs, wherein the drugs include target drugs and interfering drugs; The virtual pharmacy environment rendering module 830 is used to render a virtual pharmacy environment based on the drug database using a virtual reality device. The medication dispensing module 840 is used to respond to receiving a medication dispensing instruction for at least one target drug, and based on the medication dispensing instruction and the virtual pharmacy environment, to perform human-computer interaction using the virtual reality device to obtain the medication dispensing result.

[0067] Specifically, in this embodiment, the specific functions of the above-mentioned virtual reality-based drug retrieval training system can also be referred to the corresponding description in the above-mentioned virtual reality-based drug retrieval training method, which will not be repeated here.

[0068] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 9As shown. This smart terminal can be used to execute the virtual reality-based drug retrieval training method provided in the above embodiments, which will not be described in detail here for the sake of brevity. The smart terminal includes: a processor, coupled to a memory, the memory for storing computer programs or instructions, and the processor for executing the computer programs or instructions stored in the memory, so that the method in the above method embodiments is executed.

[0069] The present invention also provides a computer-readable storage medium having stored thereon computer instructions for implementing the methods in the above-described method embodiments.

[0070] For example, when the computer program is executed by a computer, it enables the computer to implement the methods described in the above method embodiments.

[0071] This application also provides a computer program product containing instructions that, when executed by a computer, cause the computer to implement the methods described in the above method embodiments.

[0072] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0073] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0077] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A drug retrieval training method based on virtual reality, characterized in that, Includes the following steps: Based on the pre-defined multimodal drug characteristics, several initial drug packaging models are constructed. Based on the initial drug packaging model and key feature information of several drugs, a drug database is constructed, including target drugs and interfering drugs. Based on the aforementioned drug database, a virtual pharmacy environment is rendered using virtual reality equipment; In response to receiving a medication retrieval instruction for at least one target drug, the system uses the virtual reality device to perform human-computer interaction based on the retrieval instruction and the virtual pharmacy environment to obtain the medication retrieval result.

2. The drug retrieval training method based on virtual reality according to claim 1, characterized in that, Based on preset multimodal drug characteristics, several initial drug packaging models are constructed, including: The text information of the medicine is processed based on a preset text enhancement algorithm to obtain several medicine font styles; Based on the font style of the drug, the preset packaging appearance and / or the preset color coding format, several initial drug packaging models are constructed.

3. The drug retrieval training method based on virtual reality according to claim 1, characterized in that, The process of constructing a drug database based on the initial drug packaging model and key feature information of several drugs includes: The initial drug packaging model is fused with the key feature information of each drug to obtain several styles of drug packaging models corresponding to each drug. A drug database is constructed based on all the aforementioned drug packaging models.

4. The drug retrieval training method based on virtual reality according to any one of claims 1-3, characterized in that, The step of rendering a virtual pharmacy environment using virtual reality equipment based on the drug database includes: An initial virtual pharmacy environment was rendered using virtual reality equipment; According to the preset deployment principles, the drug packaging models in the drug database are deployed in the initial virtual pharmacy environment to render the virtual pharmacy environment.

5. The drug retrieval training method based on virtual reality according to claim 4, characterized in that, The preset deployment principles specifically include: The interfering drugs in the drug database include at least one similar drug with a similar name and the same drug packaging model style as the target drug, and include multiple dissimilar drugs with dissimilar names and random drug packaging model styles as the target drug. The packaging model of the target drug is randomly selected, and the packaging model of the target drug is the same as the packaging model of at least one similar drug. The packaging models of several dissimilar drugs are randomly selected and randomly arranged on the shelves in the initial virtual pharmacy environment.

6. The drug retrieval training method based on virtual reality according to claim 4, characterized in that, The process of obtaining a medication retrieval result by using the virtual reality device for human-computer interaction based on the medication retrieval instruction and the virtual pharmacy environment includes: In response to receiving an operation action performed by the operator corresponding to the virtual reality device in accordance with the medicine retrieval instruction and preset dynamic interference factors; Based on the operation, motion capture data and drug placement location data are generated in the virtual pharmacy environment; Based on the motion capture data and the drug placement location data, the drug retrieval result is obtained.

7. The drug retrieval training method based on virtual reality according to claim 6, characterized in that, The process of obtaining the medication retrieval result based on the motion capture data and the medication placement location data includes: Based on the target position of the target drug on the target shelf and the motion capture data, the drug to be grasped is determined; Based on the designated placement location corresponding to the target drug and the drug placement location data, the correctness of the drug placement location is determined; The result of drug retrieval is obtained based on the correctness of the grasped drug and its placement position.

8. A medicine dispensing training system based on virtual reality, characterized in that, The system includes: The initial drug packaging model construction module is used to construct several initial drug packaging models based on preset multimodal drug characteristics. A drug database construction module is used to construct a drug database based on the initial drug packaging model and key feature information of several drugs, wherein the drugs include target drugs and interfering drugs; The virtual pharmacy environment rendering module is used to render a virtual pharmacy environment based on the drug database using virtual reality equipment. The medication dispensing module is used to respond to receiving a medication dispensing instruction for at least one target drug, and based on the medication dispensing instruction and the virtual pharmacy environment, to perform human-computer interaction using the virtual reality device to obtain the medication dispensing result.

9. A smart terminal, characterized in that, include: Memory, used to store executable instructions; A processor for calling and running the executable instructions in the memory to perform the steps of the virtual reality-based drug retrieval training method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed by a processor, implement the virtual reality-based drug retrieval training method as described in any one of claims 1-7.

Images