Artificial intelligence language model-based virtual reality communication system including error prevention function

Abstract

Provided are an artificial intelligence language model-based virtual reality communication method and system including an error prevention function. The artificial intelligence language model-based virtual reality communication method including an error prevention function, proposed by the present invention, includes the steps of: providing, to a user through a VR device, a virtual hospital environment for virtual reality-based nurse SBAR communication training and receiving an input of speech from the user; converting the received input of speech into text through a speech-text conversion unit; determining SBAR from the text and extracting patient and situation-related data through a data extraction unit; merging the extracted patient and situation-related data with previously stored conversation analysis data through a data merging unit; analyzing the merged data to identify missing information and generating a scenario-based doctor response conversation as text through a conversation generation unit; requesting speech conversion for the generated text of the doctor response conversation through the speech-text conversion unit and extracting speech converted from the text; and playing the doctor response conversation as speech to a user through the VR device.

Description

A virtual reality communication system with error prevention features based on an artificial intelligence language model.

[0001] The present invention relates to a virtual reality communication method and system including an error prevention function based on an artificial intelligence language model, and more particularly, to a method and system for virtual reality-based SBAR (Situation, Background, Assessment, Recommendation) communication training for nurses.

[0002] The domestic and international software (SW) industry is experiencing steady growth in sales and employment. However, the industry's technical workforce shortage rate is currently at 4.3%, exacerbating the labor shortage. Despite the explosive growth in demand, the supply of skilled workers is limited.

[0003] Furthermore, various factors, such as remote work and digital transformation, are driving the demand for work collaboration tools, leading to the adoption of multiple collaboration tools like Slack and Notion. Furthermore, traditional work environments are disappearing, driving the need for new forms of collaboration. Furthermore, there's a growing trend of close collaboration not only among internal employees but also with external stakeholders.

[0004] Software development imposes a significant burden on acquiring work-related knowledge, and the complex and repetitive nature of communication processes makes it difficult to maintain work efficiency and productivity. Furthermore, the development process doesn't improve with additional staffing, and it takes time for newly hired staff to become productive.

[0005] These problems occur not only in software development but also in various other tasks, and research is needed to find solutions to perform tasks more efficiently.

[0006] The technical task of the present invention is to provide a virtual reality communication method and system that includes an error prevention function based on an artificial intelligence language model, which facilitates effective communication training among medical professionals through SBAR (Situation, Background, Assessment, Recommendation) communication training to improve the communication skills of nurses. According to an embodiment of the present invention, the virtual reality communication system is replaced with a software program based on an artificial intelligence language model to automate the process.

[0007] In one aspect, the virtual reality communication method including an error prevention function based on an artificial intelligence language model proposed in the present invention comprises the steps of providing a user with a virtual hospital environment for virtual reality-based SBAR communication training of nurses through a VR device, receiving a voice input from the user, converting the input voice into text through a voice-to-text conversion unit, recognizing SBAR from the converted text through a data extraction unit and extracting patient and situation-related data, merging the extracted patient and situation-related data with previously stored conversation analysis data through a data merging unit, analyzing the merged data through a conversation generation unit to check for missing information and generating a scenario-based doctor-response dialogue as text, requesting voice conversion for the generated doctor-response dialogue text through a voice-to-text conversion unit and extracting the converted voice, and playing the doctor-response dialogue as voice to the user through the VR device.

[0008] The step of recognizing SBAR in the converted text through the data extraction unit and extracting patient and situation-related data uses the converted text input from the speech-to-text conversion unit and previously stored previous context information to extract and classify information contained in the text and the intent of the user's voice.

[0009] The step of recognizing SBAR from the converted text through the data extraction unit and extracting patient and situation-related data receives the identification number of the VR device and scenario information from the voice-to-text conversion unit, and learns according to the corresponding scenario based on an interactive artificial intelligence language model.

[0010] The step of merging the extracted patient and situation-related data with the previously stored previous conversation analysis data through the above data merging unit classifies the user voice intent information and slot information regarding the patient and situation-related data extracted from the text into data according to a format determined in advance based on the corresponding scenario, and stores the data by merging it with the previously stored previous conversation analysis data extracted from the user voice.

[0011] The step of analyzing the merged data through the above-mentioned dialogue generation unit to check for missing information and generating a scenario-based doctor response dialogue as text compares the accumulated data from the starting point of virtual reality communication with the patient and situation-related data required in the scenario, analyzes the slot information required at the current point in time among the missing information, and generates a text to request from the user.

[0012] The step of analyzing the merged data through the above-mentioned dialogue generation unit to check for missing information and generating a scenario-based doctor-response dialogue as text involves inputting scenario information into a pre-trained conversational artificial intelligence language model and requesting a situational drama from the user, generating a doctor-response dialogue as text to convey the patient and situation-related data required in the scenario, and analyzing user input phrases input in real time according to the request through the conversational artificial intelligence language model.

[0013] The step of analyzing the merged data through the above-mentioned dialogue generation unit to check for missing information and generating a scenario-based doctor-response dialogue as text includes: obtaining previously stored previous dialogue analysis data from the data merging unit to generate the scenario-based doctor-response dialogue as text, using the previously trained conversational artificial intelligence language model to utilize the previously stored previous dialogue analysis data and patient and situation-related data extracted from the text currently input, and repeatedly generating a scenario-based doctor-response dialogue according to the analysis results to reset the scenario-based situational drama when an error in the scenario-based situational drama occurs.

[0014] In another aspect, the virtual reality communication system including an error prevention function based on an artificial intelligence language model proposed in the present invention includes a VR device that provides a user with a virtual hospital environment for SBAR communication training of a nurse based on virtual reality, receives a voice input from the user, and plays back to the user a voice for a doctor response dialogue text converted by a voice-to-text conversion unit, a voice-to-text conversion unit that converts the input voice into text, requests voice conversion for the doctor response dialogue text generated by a dialogue generation unit, and extracts the converted voice, a data extraction unit that recognizes SBAR from the converted text and extracts patient and situation-related data, a data merging unit that merges the extracted patient and situation-related data with previously stored dialogue analysis data, and a dialogue generation unit that analyzes the merged data to identify missing information and generates a scenario-based doctor response dialogue as text.

[0015] According to embodiments of the present invention, SBAR (Situation, Background, Assessment, Recommendation) communication training for improving nurses' communication skills is possible without time or location constraints in a non-face-to-face manner through a virtual reality communication system that includes an error prevention function based on an artificial intelligence language model. According to embodiments of the present invention, the virtual reality communication system is replaced with a software program based on an artificial intelligence language model to automate the training. Compared to existing face-to-face training methods, it offers greater accessibility and convenience, higher learning accuracy, and consistent personalized training, thereby presenting an alternative to existing training methods.

[0016] FIG. 1 is a flowchart illustrating a virtual reality communication method including an error prevention function based on an artificial intelligence language model according to one embodiment of the present invention.

[0017] FIG. 2 is a diagram showing the configuration of a virtual reality communication system including an error prevention function based on an artificial intelligence language model according to one embodiment of the present invention.

[0018] FIG. 3 is a diagram showing an example of scenario information input from a voice-to-text conversion unit according to one embodiment of the present invention.

[0019] FIG. 4 is a diagram for explaining a process of classifying intent information and slot information extracted from user voice text based on a scenario and converting them into data according to one embodiment of the present invention.

[0020] FIG. 5 is a diagram for explaining a process of performing learning according to a scenario according to one embodiment of the present invention.

[0021] FIG. 6 is a diagram for explaining a process of extracting scenario information according to one embodiment of the present invention.

[0022] FIG. 7 is a diagram illustrating a process of classifying information required for an SBAR situation in an analysis result according to one embodiment of the present invention.

[0023] FIG. 8 is a diagram showing response data of a server according to one embodiment of the present invention.

[0024] As the need for SBAR (Situation, Background, Assessment, Recommendation) communication training to improve nurses' communication skills has arisen, human-to-human communication training has been implemented. The present invention aims to automate this process by replacing it with a software program based on an artificial intelligence language model. Compared to existing face-to-face training methods, this training method offers greater accessibility and convenience, higher learning accuracy, and consistent, personalized training, offering an alternative to existing training methods.

[0025] The present invention proposes a system for SBAR (Situation, Background, Assessment, Recommendation) communication training for nurses based on virtual reality, utilizing a conversational AI language model (e.g., ChatGPT). According to an embodiment of the present invention, a trainee wears VR goggles and practices communication between healthcare professionals through a conversation-based role-play with ChatGPT in a virtual hospital environment. The GPT's responses are classified and verified using a Large Language Model (LLM). If an error occurs in the response, the role settings of ChatGPT are readjusted to continue training. While ChatGPT is described as an example in one embodiment of the present invention, various conversational AI language models can be used.

[0026] In order for a nursing student to practice SBAR communication through a virtual reality communication method and system including an error prevention function based on an artificial intelligence language model according to an embodiment of the present invention, SBAR-related information can be extracted from input sentences coming in as real-time voice, essential patient information that has not yet been received can be recognized from the extracted information, and a sentence that guides the student to speak the information later can be output as a response.

[0027] However, recognizing only information of a specific domain from real-time natural language input sentences and using this result to generate a conversation to elicit information from the other party that is absolutely necessary for learning but has not yet been input is a truly difficult problem.

[0028] In addition, a technology that analyzes the structure of information in a sentence that simply lists information about a specific patient and the current nursing situation in a medical situation, rather than a sentence that requests or asks information for a specific purpose, and then generates questions about missing information based on this has not yet been disclosed.

[0029] According to an embodiment of the present invention, a virtual reality communication method and system is proposed, including an error prevention function based on an artificial intelligence language model that generates counter-questions based on natural sentences, rather than simply responding to a query. Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0030]

[0031] FIG. 1 is a flowchart illustrating a virtual reality communication method including an error prevention function based on an artificial intelligence language model according to one embodiment of the present invention.

[0032] According to one embodiment of the present invention, an AI-based voice conversation system was built in a VR (Virtual Reality) environment to enable nursing education requiring a counterpart role to be practiced without the help of others in a virtual environment through a virtual reality communication method including an error prevention function based on an AI language model. To configure a VR environment according to one embodiment of the present invention, a Unity program and a natural language processing API (Application Program Interface) server are connected, and the program created with Unity is linked to a VR device. The Unity program according to one embodiment of the present invention is described as a voice-to-text conversion unit.

[0033] A virtual reality communication method including an error prevention function based on an artificial intelligence language model according to one embodiment of the present invention comprises the steps of: providing a user with a virtual hospital environment for virtual reality-based SBAR communication training of nurses through a VR device, receiving a voice input from the user (110); converting the input voice into text (i.e., transcription) through a voice-to-text conversion unit (120); recognizing SBAR from the converted text through a data extraction unit and extracting patient and situation-related data (130); merging the extracted patient and situation-related data with previously stored conversation analysis data through a data merging unit (140); analyzing the merged data through a conversation generation unit to check for missing information and generating a scenario-based doctor-response dialogue as text (150); requesting voice conversion for the generated doctor-response dialogue text through a voice-to-text conversion unit (161), extracting the converted voice (162); and playing the doctor-response dialogue as voice to the user through the VR device (170).

[0034] According to one embodiment of the present invention, input and interruption of a user's (e.g., a nursing student's) voice through a VR device is detected by a Unity program (i.e., a voice-to-text conversion unit), and the VR device and a virtual reality communication system are connected, and the VR device, the virtual reality communication system, and scenario information are initialized.

[0035] In step (110), a virtual hospital environment for virtual reality-based SBAR communication training of nurses is provided to a user (e.g., a nursing student) through a VR device, and voice input is received from the user.

[0036] In step (120), the input voice is converted into text through a voice-to-text conversion unit.

[0037] A voice-to-text conversion unit according to an embodiment of the present invention waits for, for example, 2 seconds from the time when the user's voice detected has stopped, to see if there is any further input, and then transmits the converted text for the detected user's voice to the server.

[0038] In step (130), SBAR is recognized from the converted text through the data extraction unit, and patient and situation-related data is extracted.

[0039] The purpose is to receive the converted text and previously stored context information through a data extraction unit of a server according to one embodiment of the present invention, and to appropriately extract and classify information contained in the text and the intent of the user's voice.

[0040] According to one embodiment of the present invention, a model additionally trained based on a conversational AI language model (e.g., ChatGPT) tailored to a nursing scenario is used. Since the portion of text to focus on for information extraction varies depending on the nursing scenario, the voice-to-text converter according to the present invention receives the VR device identification number and usage scenario information together, and selects and uses the model trained for the corresponding scenario.

[0041] In step (140), the extracted patient and situation-related data are merged with previously stored conversation analysis data through the data merging unit.

[0042] According to an embodiment of the present invention, intention information and slot information extracted from user voice text are classified and data-ized according to a format predetermined based on a scenario, and are additionally stored in data extracted from past user voices and recorded in a virtual reality communication system.

[0043] In step (150), a scenario-based response dialogue is generated as text through a dialogue generation unit.

[0044] In an embodiment of the present invention, a server compares data accumulated from the start point of a virtual reality communication system with patient and situation information data required in a scenario, and generates a doctor response dialogue text requesting the user to select the slot information most required at the current point in time among the missing information.

[0045] According to an embodiment of the present invention, in order to input scenario information into a pre-trained conversational artificial intelligence language model and request a situational drama from a user, a doctor response dialogue for conveying patient and situation-related data required for the scenario is generated as text, and user input phrases input in real time according to the request are analyzed through the conversational artificial intelligence language model.

[0046] In order to generate the above scenario-based doctor response dialogue as text, the previous conversation analysis data stored in advance from the data merging unit is retrieved, and the patient and situation-related data extracted from the previous conversation analysis data and the text currently input are used through a pre-trained conversational artificial intelligence language model, and the scenario-based doctor response dialogue is repeatedly generated according to the analysis results in order to reset the situation drama when the scenario-based situation drama error occurs.

[0047] When the generated text is sent as a response to the speech-to-text conversion unit on the server, the speech-to-text conversion unit requests the web page to convert the text into speech (161) and extracts speech by scraping the conversion result (162).

[0048] Finally, the user's response dialogue is played back as voice through the VR device (170).

[0049]

[0050] FIG. 2 is a diagram showing the configuration of a virtual reality communication system including an error prevention function based on an artificial intelligence language model according to one embodiment of the present invention.

[0051] A virtual reality communication system (200) according to the present embodiment may include a processor (210), a bus (220), a network interface (230), a memory (240), and a database (250). The memory (240) may include an operating system (241) and a virtual reality communication routine (242) including an error prevention function based on an artificial intelligence language model. The processor (210) may include a data extraction unit (211), a data merging unit (212), a dialogue generation unit (213), a VR device (260), and a voice-to-text conversion unit (270). In other embodiments, the virtual reality communication system (200) may include more components than those in FIG. 2. However, it is not necessary to clearly illustrate most of the conventional components. For example, the virtual reality communication system (200) may include other components such as a display or a transceiver.

[0052] The memory (240) is a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. In addition, the memory (240) may store program codes for a virtual reality communication routine (242) including an operating system (241) and an error prevention function based on an artificial intelligence language model. These software components may be loaded from a computer-readable recording medium separate from the memory (240) using a drive mechanism (not shown). This separate computer-readable recording medium may include a computer-readable recording medium (not shown) such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (240) through a network interface (230) rather than a computer-readable recording medium.

[0053] The bus (220) may enable communication and data transmission between components of the virtual reality communication system (200). The bus (220) may be configured using a high-speed serial bus, a parallel bus, a storage area network (SAN), and / or other appropriate communication technology.

[0054] The network interface (230) may be a computer hardware component for connecting the virtual reality communication system (200) to a computer network. The network interface (230) may connect the virtual reality communication system (200) to the computer network via a wireless or wired connection.

[0055] The database (250) can store and maintain all information necessary for virtual reality communication, including error prevention functions based on an artificial intelligence language model. While FIG. 2 illustrates a database (250) built and included within the virtual reality communication system (200), the present invention is not limited thereto. Depending on the system implementation method or environment, the database may be omitted, or all or part of the database may exist as an external database built on a separate system.

[0056] The processor (210) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations of the virtual reality communication system (200). Instructions may be provided to the processor (210) by a memory (240) or a network interface (230), and via a bus (220). The processor (210) may be configured to execute program codes for a data extraction unit (211), a data merging unit (212), a dialogue generation unit (213), a VR device (260), and a voice-to-text conversion unit (270). These program codes may be stored in a recording device such as the memory (240).

[0057] The data extraction unit (211), data merging unit (212), dialogue generation unit (213), VR device (260), and voice-to-text conversion unit (270) can be configured to perform steps (110 to 170) of FIG. 1.

[0058] A virtual reality communication system (200) may include a data extraction unit (211), a data merging unit (212), a dialogue generation unit (213), a VR device (260), and a voice-to-text conversion unit (270).

[0059] A VR device (260) according to an embodiment of the present invention provides a user with a virtual hospital environment for virtual reality-based SBAR communication training for nurses, receives voice input from the user, and plays back to the user the voice for the doctor response dialogue text converted by the voice-to-text conversion unit (270).

[0060] According to one embodiment of the present invention, input and interruption of a user's (e.g., a nursing student's) voice through a VR device is detected by a Unity program (i.e., a voice-to-text conversion unit (270)), and the VR device and the virtual reality communication system are connected, and the VR device, the virtual reality communication system, and scenario information are initialized.

[0061] The voice-to-text conversion unit (270) according to an embodiment of the present invention converts the input voice into text.

[0062] The voice-to-text conversion unit (270) according to the embodiment of the present invention waits for, for example, 2 seconds from the point at which the user's voice detected has stopped, and then transmits the converted text for the detected user's voice to the server.

[0063] The data extraction unit (211) according to an embodiment of the present invention recognizes SBAR from the converted text and extracts patient and situation-related data.

[0064] The data extraction unit (211) according to an embodiment of the present invention extracts and classifies information included in the text and the intent of the user's voice using the converted text received from the voice-to-text conversion unit and previously stored previous context information.

[0065] The purpose of the data extraction unit (211) according to an embodiment of the present invention is to receive the converted text and previously stored previous context information, and to appropriately extract and classify information contained in the text and the intent of the user's voice.

[0066] The data extraction unit (211) according to an embodiment of the present invention uses a model additionally learned based on a conversational artificial intelligence language model (e.g., ChatGPT) tailored to a nursing scenario. Since the portion of text to focus on for information extraction varies depending on the nursing scenario, the voice-to-text conversion unit according to an embodiment of the present invention receives the VR device's identification number and usage scenario information together, and selects and uses a model learned for the corresponding scenario.

[0067] The data merging unit (212) according to an embodiment of the present invention merges the extracted patient and situation-related data with previously stored conversation analysis data.

[0068] The data merging unit (212) according to an embodiment of the present invention classifies the user voice intent information and slot information regarding patient and situation-related data extracted from the text into data according to a format predetermined based on the corresponding scenario, and stores the data by merging it with previous conversation analysis data extracted from the user voice and stored in advance.

[0069] The dialogue generation unit (213) according to an embodiment of the present invention analyzes the merged data to identify missing information and generates a scenario-based pseudo-response dialogue as text.

[0070] The dialogue generation unit (213) according to an embodiment of the present invention compares data accumulated from the starting point of virtual reality communication with patient and situation-related data required in the scenario, analyzes slot information required at the current point in time among the missing information, and generates a text to request from the user.

[0071] A dialogue generation unit (213) according to an embodiment of the present invention inputs scenario information into a pre-trained conversational artificial intelligence language model and generates a doctor response dialogue in text format to convey patient and situation-related data required for the scenario in order to request a situational drama from the user, and analyzes user input phrases input in real time according to the request through the conversational artificial intelligence language model.

[0072] The dialogue generation unit (213) according to an embodiment of the present invention obtains previously stored previous dialogue analysis data from the data merging unit to generate the scenario-based doctor-response dialogue as text, and uses the previous dialogue analysis data and patient and situation-related data extracted from the text currently input through a pre-trained conversational artificial intelligence language model, and repeatedly generates the scenario-based doctor-response dialogue according to the analysis results to reset the scenario when an error in the scenario-based situational drama occurs.

[0073] When the text generated through the dialogue generation unit (213) is transmitted as a response to the voice-to-text conversion unit (270), the voice-to-text conversion unit (270) requests the web page to convert the text into voice, and extracts the voice by scraping the conversion result.

[0074] Finally, the user's response dialogue is played back as voice through the VR device (260).

[0075]

[0076] FIG. 3 is a diagram showing an example of scenario information input from a voice-to-text conversion unit according to one embodiment of the present invention.

[0077] According to the prior art, research on generating responses from artificial intelligence in a goal-oriented virtual conversation environment has been conducted in various fields, but the conversation format covered by the present invention differs significantly from the prior art in that the user input does not take the form of a question or command, and is not intended for information request.

[0078] FIG. 3(a) shows an example of scenario information for a situational drama according to one embodiment of the present invention, and FIG. 3(b) shows an example of a situational drama generated through an interactive artificial intelligence language model according to one embodiment of the present invention.

[0079] Referring to Figures 3(a) and 3(b), problems are shown when attempting to role-play a Large Language Model (LLM) without fine-tuning and exception handling.

[0080] As the interview progresses, it is observed that the interviewer does not point out any missing information in the nurse's speech, and only mechanically spits out answers that would be expected if the nurse had mentioned all the necessary information.

[0081]

[0082] FIG. 4 is a diagram for explaining a process of classifying intent information and slot information extracted from user voice text based on a scenario and converting them into data according to one embodiment of the present invention.

[0083] FIG. 4(a) shows an example of a text analysis result of a user's voice according to one embodiment of the present invention, and FIG. 4(b) and FIG. 4(c) show examples of a result of classifying and datafication the intent information and slot information of a user's voice according to one embodiment of the present invention according to a format predetermined based on the corresponding scenario.

[0084] A virtual hospital environment for virtual reality-based SBAR communication training for nurses is provided to users (e.g., nursing students) through a VR device, and voice input is received from the users.

[0085] A voice-to-text conversion unit according to an embodiment of the present invention waits for, for example, 2 seconds from the time when the user's voice detected has stopped, to see if there is any further input, and then transmits the converted text for the detected user's voice to the server.

[0086] The purpose is to receive the converted text and previously stored context information through a data extraction unit of a server according to one embodiment of the present invention, and to appropriately extract and classify information contained in the text and the intent of the user's voice.

[0087]

[0088] FIG. 5 is a diagram for explaining a process of performing learning according to a scenario according to one embodiment of the present invention.

[0089] Figure 5(a) is an exemplary diagram for explaining an LLM learning process according to one embodiment of the present invention.

[0090] Referring to Figure 5(a), a scenario for a situational drama is input and training data is prepared to handle exceptional situations. For example, given a doctor-nurse dialogue pair, LLM must be trained to analyze the nurse's utterance as { "Situation 1": { "Category 1": [Information 1, Information 2, … ], …}.

[0091] Figure 5(b) is an example diagram of learning data according to one embodiment of the present invention.

[0092] Referring to Figure 5(b), training data is generated in the form of mapping analysis results (answers) to doctor-nurse dialogue pairs (questions) for some of the conversations that may occur in SBAR delivery situations. This training data is fed into the LLM, which has been trained with extensive data in advance, and fine-tuned to learn how to appropriately analyze conversations within the given scenario.

[0093]

[0094] FIG. 6 is a diagram for explaining a process of extracting scenario information according to one embodiment of the present invention.

[0095] FIG. 6(a) and FIG. 6(b) are exemplary diagrams showing the results of extracting scenario information according to one embodiment of the present invention.

[0096] Referring to Figures 6(a) and 6(b), scenario information can be automatically extracted through real-time inference via LLM, and the information can be input into LLM and then real-time conversation analysis can be requested.

[0097] After converting the scenario data of Fig. 5(a) to .csv and saving it, a specific script is executed to extract the data in json format as shown in Fig. 6(c).

[0098]

[0099] FIG. 7 is a diagram illustrating a process of classifying information required for an SBAR situation in an analysis result according to one embodiment of the present invention.

[0100] Figure 7(a) shows a dialogue system log, and Figure 7(b) shows an example of initializing dialogue_state, which stores utterance information at the start of the dialogue system, for an SBAR item.

[0101] As shown in Fig. 7(a), the finely tuned LLM accumulates and stores information necessary for the SBAR situation within the speech analyzed in real time in the dialogue_state above.

[0102] Referring to Figure 7(c), the current SBAR stage is marked as "progress." The required information for the current stage is compared with the dialogue_state (information analyzed by the LLM). If any information is missing, a response is output, asking for more information. Upon completion of the stage, the patient moves to the next stage, which has the educational effect of encouraging the patient to report their condition to the physician according to the SBAR sequence.

[0103]

[0104] FIG. 8 is a diagram showing response data of a server according to one embodiment of the present invention.

[0105] As in Figure 8(a), if all the necessary information has been received, proceed to the next step. If not, as in Figure 8(b), a response is output asking for the missing information.

[0106] According to one embodiment of the present invention, when a nursing student conducts communication practice in which patient information is shared among medical staff, the practice can be performed in an environment very close to communication training in an actual medical situation.

[0107] Missing information in medical settings can cause significant problems. The natural language information recognition method described above captures minute details of conversations that conventional technologies fail to capture, automatically detects missing information, and guides trainees to supplement the information themselves, thereby mitigating these risks in real-world medical settings.

[0108] Most question-answering systems are primarily intended to provide answers to users' questions, but are difficult to apply to conversations in special situations, which is the purpose of the present invention. The present invention includes a technology for generating counter-questions by identifying missing information required for each field in a user's question or statement, and thus can handle a wider variety of natural language and conversation situations than before.

[0109] The virtual reality communication system, including error prevention features based on an AI language model according to an embodiment of the present invention, can be used for education and training in multidisciplinary medical fields. It can also be used to teach SBAR communication skills to nursing students or other medical students.

[0110] According to an embodiment of the present invention, it can be used in a nursing practice education platform. By integrating the present invention into educational software or a web platform, accessibility to SBAR communication practice can be improved.

[0111] According to another embodiment of the present invention, it can be used in a virtual reality medical simulator. In conjunction with a VR headset, ChatGPT can be used to provide more realistic SBAR training.

[0112] According to another embodiment of the present invention, it can be used in a medical institution's intranet. By integrating it into the medical institution's existing system, both new and existing medical staff, including new nurses unfamiliar with SBAR, can be trained according to a prescribed format.

[0113]

[0114] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0115] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0116] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0117] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0118] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A step of providing a virtual hospital environment for virtual reality-based nurse SBAR communication training to a user through a VR device and receiving voice input from the user; A step of converting the input voice into text through a voice-to-text conversion unit; A step of recognizing SBAR from the converted text through a data extraction unit and extracting patient and situation-related data; A step of merging the extracted patient and situation-related data with previously stored conversation analysis data through a data merging unit; A step of analyzing the merged data through a dialogue generation unit to identify missing information and generating a scenario-based doctor-response dialogue as text; A step of requesting voice conversion for the generated pseudo-response dialogue text through a voice-to-text conversion unit and extracting the converted voice; and Steps to play back a voice response dialogue to the user through a VR device A virtual reality communication method including:

2. In paragraph 1, The step of recognizing SBAR from the converted text through the above data extraction unit and extracting patient and situation-related data is as follows: Using the converted text received from the above speech-to-text conversion unit and the previously stored previous context information, the information contained in the text and the intent of the user's voice are extracted and classified. How to communicate in virtual reality.

3. In paragraph 2, The step of recognizing SBAR from the converted text through the above data extraction unit and extracting patient and situation-related data is as follows: The identification number of the VR device and scenario information are input from the above voice-to-text conversion unit, and learning is performed according to the corresponding scenario based on an interactive artificial intelligence language model. How to communicate in virtual reality.

4. In paragraph 1, The step of merging the extracted patient and situation-related data with the previously stored conversation analysis data through the above data merging unit is as follows: The user's voice intent information and slot information regarding patient and situation-related data extracted from the above text are classified and dataized according to a pre-determined format based on the corresponding scenario, and are stored by merging them with the previously stored conversation analysis data extracted from the user's voice. How to communicate in virtual reality.

5. In paragraph 1, The step of analyzing the merged data through the above dialogue generation unit to identify missing information and generating a scenario-based doctor-response dialogue as text is as follows: From the starting point of virtual reality communication, the accumulated data is compared with the patient and situation-related data required in the scenario, and the missing information is analyzed to generate a text to request to the user by analyzing the slot information required at the current point in time. How to communicate in virtual reality.

6. In paragraph 5, The step of analyzing the merged data through the above dialogue generation unit to identify missing information and generating a scenario-based doctor-response dialogue as text is as follows: In order to input scenario information into a pre-trained conversational AI language model and request a situational drama from the user, a doctor response dialogue is generated as text to convey the patient and situation-related data required for the scenario, and the user input phrases input in real time according to the request are analyzed through the conversational AI language model. How to communicate in virtual reality.

7. In paragraph 5, The step of analyzing the merged data through the above dialogue generation unit to identify missing information and generating a scenario-based doctor-response dialogue as text is as follows: In order to generate the above scenario-based doctor response dialogue as text, the previous conversation analysis data stored in advance from the data merging unit is retrieved, and the patient and situation-related data extracted from the text currently input are used with the previous conversation analysis data and the text currently input through a pre-trained conversational artificial intelligence language model, and the scenario-based doctor response dialogue is repeatedly generated according to the analysis results in order to reset the situation drama when the scenario-based situation drama error occurs. How to communicate in virtual reality.

8. A VR device that provides a virtual hospital environment for virtual reality-based SBAR communication training for nurses, receives voice input from the user, and plays back to the user the voice for the doctor's response dialogue text converted by the voice-to-text conversion unit; A speech-to-text conversion unit that converts the input speech into text, requests speech conversion for the pseudo-response dialogue text generated by the dialogue generation unit, and extracts the converted speech; A data extraction unit that recognizes SBAR from the above-mentioned converted text and extracts patient and situation-related data; A data merging unit that merges the extracted patient and situation-related data with previously stored conversation analysis data; and A dialogue generation unit that analyzes the above merged data to identify missing information and generates scenario-based doctor-response dialogues as text. A virtual reality communication system including:

9. In paragraph 8, The above data extraction unit, Using the converted text received from the above speech-to-text conversion unit and the previously stored previous context information, the information contained in the text and the intent of the user's voice are extracted and classified. Virtual reality communication system.

10. In paragraph 9, The above data extraction unit, The identification number of the VR device and scenario information are input from the above voice-to-text conversion unit, and learning is performed according to the corresponding scenario based on an interactive artificial intelligence language model. Virtual reality communication system.

11. In paragraph 8, The above data merging section, The user's voice intent information and slot information regarding patient and situation-related data extracted from the above text are classified and dataized according to a pre-determined format based on the corresponding scenario, and are stored by merging them with the previously stored conversation analysis data extracted from the user's voice. Virtual reality communication system.

12. In paragraph 8, The above dialogue generation unit, From the starting point of virtual reality communication, the accumulated data is compared with the patient and situation-related data required in the scenario, and the missing information is analyzed to generate a text to request to the user by analyzing the slot information required at the current point in time. Virtual reality communication system.

13. In paragraph 12, The above dialogue generation unit, In order to input scenario information into a pre-trained conversational AI language model and request a situational drama from the user, a doctor response dialogue is generated as text to convey the patient and situation-related data required for the scenario, and the user input phrases input in real time according to the request are analyzed through the conversational AI language model. Virtual reality communication system.

14. In paragraph 12, The above dialogue generation unit, In order to generate the above scenario-based doctor response dialogue as text, the previous conversation analysis data stored in advance from the data merging unit is retrieved, and the patient and situation-related data extracted from the text currently input are used with the previous conversation analysis data and the text currently input through a pre-trained conversational artificial intelligence language model, and the scenario-based doctor response dialogue is repeatedly generated according to the analysis results in order to reset the situation drama when the scenario-based situation drama error occurs. Virtual reality communication system.

15. In a program stored in a computer-readable storage medium for executing a virtual reality communication method including an error prevention function based on an artificial intelligence language model, A step of providing a virtual hospital environment for virtual reality-based SBAR communication training of nurses to a user through a VR device and receiving voice input from the user; A step of converting the input voice into text through a voice-to-text conversion unit; A step of recognizing SBAR from the converted text through a data extraction unit and extracting patient and situation-related data; A step of merging the extracted patient and situation-related data with previously stored conversation analysis data through a data merging unit; A step of analyzing the merged data through a dialogue generation unit to identify missing information and generating a scenario-based doctor-response dialogue as text; A step of requesting voice conversion for the generated pseudo-response dialogue text through a voice-to-text conversion unit and extracting the converted voice; and Steps to play back a voice response dialogue to the user through a VR device A program stored on a computer-readable storage medium containing .

Images