Educational VR Content Provision Program and Medical Education VR Content Provision System

Abstract

To provide an education VR content provision system which can provide an education VR content that allows a user to grasp, at a glance, a spot to which attention should be paid all the time in a treatment while the treatment is performed.SOLUTION: There is provided a medical education VR content provision system which connects a medical education VR content site 400, an instructor side terminal 110 and a learner side terminal with each other via the high-speed communication network (5G). A server of the medical education VR content site 400 acquires motion information of a HMD, determines a visual field image, overlaps a pointer icon indicating the sight line direction of a portion that pays attention to the determined visual field image with the visual field image, and transmits the image as a gazed image with a pointer to the instructor side terminal 110 or the medical learner side terminal via the high-speed communication network (5G) to display the image on a screen.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a content providing system for medical education VR.

Background Art

[0002] VR (Virtual Reality) is also called "artificial reality" or "virtual reality", and with VR, an experience extremely close to a real experience can be obtained. Conventionally, it has been difficult to actually reproduce a highly dangerous emergency situation for the purpose of learning. However, by applying VR (Virtual Reality), it becomes possible to reproduce it safely, and various scenes are assumed for learning.

[0003] That is, it becomes possible to repeatedly experience and share an ideal situation closer to reality many times. For this reason, each person can perform simulations etc. in case of an emergency within the medical team even in normal times. Therefore, by repeating virtual experiences using VR, the experience value of medical learners also increases, and it is expected that they can calmly implement initial responses even in an actual emergency situation. For example, it is considered that it may contribute to improving the safety of dental treatment.

[0004] For example, in Patent Document 1, profile information such as the age and gender of each patient, medical record information, three-dimensional CT imaging information obtained by X-ray CT imaging, drilling information designed in advance for implanting fixtures in implant treatment, and three-dimensional shape data of implants etc. are stored in a memory (it may be separately acquired via the Internet) in a dental treatment training device (HMD).

[0005] And it is disclosed that a simulated patient body is photographed with an HMD, the three-dimensional visual field video photographed by this HMD is displayed on a screen, and further, a simulated practice is carried out while the HMD superimposes and displays these.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, although medical treatment requires high expertise, experience, etc., Patent Document 1 is for learning by making the content of simulation learning into three-dimensional images (3D), and it is impossible to learn where to fix the line of sight and gaze.

[0008] For example, in the overall management of dental treatment including implant treatment in a general dental clinic, as the particularity of anesthesia management in general anesthesia, it is particularly important that the surgical field and the airway area coincide and that respiratory management is required due to the operation of irrigation. The safety of the operation is to be protected by all medical staff involved. For this purpose, it is important to respond while adjusting various things with a broad perspective, not just the role of one's own occupation.

[0009] Therefore, it is considered important to have training, simulation, and communication in team medicine on a daily basis. Therefore, during treatment, it is desirable to have learning that can grasp at a glance where to always pay attention to proceed with the treatment.

Means for Solving the Problems

[0010] In order to solve the above problems, the content providing program for educational VR according to the present invention To a medical instructor or learner is a program for displaying on the educational VR image based on the movement of the worn head-mounted display, On the educational VR image for patient treatment displayed on the head-mounted display, with a pointer icon indicating the line of sight from the medical instructor to the point of attention to the patient wherein the computer has means for storing the educational VR image in a storage means, (A). means for storing the educational VR image in a storage means, (B). said Worn by a medical instructor or learnerMeans for reading the movement information of the head-mounted display at regular intervals, (C). Means for determining the center of the line of sight in the educational VR image and determining the field of view based on the movement information of the head-mounted display, (D) Means for displaying the visual field image of the determined visual field range on the screen of the terminal of the instructor or learner (E) Causing the terminal of the medical instructor or the learner to define the position where the pointer icon ends as the point of attention in this visual field image Means, (F). Regarding the definition of the end point Along with this, Define the starting point at the center of the visual field image, and the coordinates of this starting point and The End point Means for obtaining the coordinates of, (G). Of the field-of-view image, Of the starting point Taking the coordinates as a starting point, Of the starting point Between these coordinates and the coordinates of the End point Overlay the pointer icon on the field-of-view image, A striped triangular icon whose width gradually widens from the starting point towards the end point And generate this as a pointer-annotated view image in the storage means, Of the medical instructor or learner Means, (H). The Transmit the pointer-annotated visual image of the medical instructor to the learner terminal for display Means, Execute the function as.

[0011] Also, The terminal of the instructor has a head-mounted display, a controller, a monitor, and a personal computer, and is connected to the server of the educational VR content site via a communication network The terminal of the learner has a head-mounted display, a controller, a smartphone with a high-speed large-capacity communication function, and a personal computer, and the smartphone is connected to the server of the educational VR content site via a communication network The server Execute the functions as the means of (A) to (H) above

Advantages of the Invention

[0012] As described above, according to the present invention, the line-of-sight direction of the part or equipment to be gazed at on the head-mounted screen is displayed in the VR image seen by the pointer. Therefore, during treatment, it is possible to learn at a glance what to always pay attention to and proceed with the treatment.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Mode for Carrying Out the Invention

[0014] First, the outline of the present invention will be described. Note that it will be described by taking dentistry as an example. The present invention can be applied to (A) the dental treatment environment and the anesthesia management of oral and maxillofacial surgery, (B) the application of medical education and VR, (C) the learning and evaluation methods using VR content, etc.

[0015] <(A) The dental treatment environment and the anesthesia management of oral and maxillofacial surgery> According to the 2017 survey of the Ministry of Health, Labour and Welfare, implant treatment is said to be carried out in 24,014 facilities, which accounts for 35% of dental clinics. In May 2007, an accident occurred in a dental clinic in Tokyo where a 70-year-old woman who received implant treatment died due to arterial injury during the operation. As a result, the problem of simulating diagnosis by applying diagnostic instruments to a dental model regarding the life-saving ability of dentists was proposed, and a system for ensuring medical safety and acquiring the technology of life-saving medicine has been established.

[0016] According to a report by some researchers, in the overall management of dental treatments including implant treatment in a general dental hospital, as a special feature of anesthesia management in general anesthesia, the fact that the surgical field and the airway area coincide, and the respiratory management due to the irrigation and other operations are particularly important.

[0017] By draping, it is significantly different from normal dental treatment, and it is difficult to immediately grasp the patient's expression and condition. In case of an emergency, it becomes an even more different situation from normal dental treatment. Therefore, it is conceivable that it may be difficult to respond smoothly due to differences in the proficiency of medical staff, etc.

[0018] In the research conducted by the author, dental medical staff tended to seek instructions from each dentist when a problem occurred to the patient. However, in the case of actually encountering such an emergency situation, it is also conceivable that the dentist himself / herself may be flustered. The safety of the surgery is protected by all medical staff involved. For that purpose, it is important to respond while adjusting various things with a broad perspective, not just the role of one's own occupation.

[0019] For that purpose, it is considered important to have training, simulation, and communication in team medicine on a daily basis. That is, the present invention can be applied to learning for dental treatment environments and anesthesia management in oral and maxillofacial surgery.

[0020] <(B) Medical Education and Application of VR> Conventionally, the educational style of gathering students in one place for lectures was common, but in the current situation of the COVID-19 pandemic, online education is being promoted, and lectures online have centered on lectures in a classroom setting. In a profession that requires specialized and advanced techniques and knowledge such as medicine, training involving experience is considered important, and there are concerns that online lectures centered on classroom learning may result in insufficient experience in medical cases compared to clinical learning.

[0021] Virtual reality (VR) - based virtual experiences are considered superior to real - world experiences in that, for example, even if they fail at a certain point, they can go back to any past time and repeat training, or they can prepare ideal experiences that include pure educational programs. In actual clinical learning, cases may be biased depending on the timing and region, or may be too specialized. From the perspective of standardizing medical education, VR is considered one of the effective means of educational tools.

[0022] In recent years, 360 - degree omnidirectional cameras and Head Mount Displays (hereinafter referred to as HMDs) for viewing VR content have become more common, and it has become possible to easily record, play back, and share visual images alone. That is, VR medical education using HMDs is tailored to the level of the learners. <(Learning and Evaluation Methods Applying VR Content)> Using a 360 - degree camera, a VR content simulating an emergency situation is prototyped, and learners wearing an HMD or the like conduct training while viewing the video.

[0023] By looking at the line - of - sight video of the learner displayed on the PC, the evaluator can confirm whether the learner can grasp the condition of the patient that should be considered in an emergency. The evaluation sheet used at that time describes check - points that should be noted for each action, enabling the evaluator to conduct a uniform evaluation. Next, the outline of the VR content used in the present invention will be described.

[0024] (Produced Educational VR Content) (1) Responses to Incidents during Dental Treatment (2) Anesthesia Management during Intravenous Sedation in Oral Surgery VR content was produced with these two themes (Figure 10). Regarding the video in (1), it was created for hyperventilation syndrome and anaphylactic shock. Hyperventilation syndrome is a mild and incidental disease that does not lead to a fatal turn, but there are cases where dentists may be flustered because patients complain of severe breathing difficulties, etc., and may not be able to respond promptly and appropriately.

[0025] On the other hand, anaphylactic shock will lead to a fatal turn if left untreated, so it is necessary to take measures such as immediate intramuscular injection of adrenaline. However, anaphylactic shock is not an incidental disease that dentists frequently encounter. So far, they have learned through their own image training via books, educational videos, and internships. Therefore, VR content was created while incorporating checkpoints in the video that can confirm the diagnostic pointers and treatment details for both.

[0026] For example, in the scenario of anaphylactic shock, after the dentist performs local anesthesia in the oral cavity, after a while, the patient complains of feeling unwell. In response, the dentist places the patient in the shock position and starts monitoring the vital signs by attaching a biometric monitor.

[0027] Normally, if it is a vasovagal reflex, the blood pressure will naturally recover. However, if the blood pressure continues to drop and a reddish skin condition is observed on the forearm, the scenario is to suspect anaphylactic shock and administer an epinephrine auto-injector by intramuscular injection. As a pointer, the scenario is configured to ask whether the learner can detect and understand the differential diagnosis from vasovagal reflex and the important skin symptoms for that purpose.

[0028] The evaluator can confirm from the line-of-sight video of the learner shown on a separate monitor whether the learner's line of sight is directed without missing the patient's state of gripping the hand due to mental tension or pain of local anesthesia, the patient's expression until complaining of feeling unwell after anesthesia, and the reddening of the skin on the forearm, etc. If the learner has directed their line of sight, the evaluator can confirm what the learner has detected or understood through a written test or oral questioning.

[0029] For example, when a patient is given a strong pain stimulus in the oral cavity, which is the trigeminal nerve area, while being overly tense, there is a possibility of triggering a vasovagal reflex. Also, when anaphylactic shock is suspected, questions are asked about what other symptoms may occur. Additionally, regarding peripheral knowledge such as the fact that since it is an EpiPen trainer in the video, the color of the syringe is white, while the color of the syringe for adult EpiPens is either yellow or green.

[0030] In this way, while having learners experience the response to emergencies using VR, for each checkpoint, the evaluator can confirm whether the learner was looking in the right direction and accurately grasped and understood the situation through oral or written tests.

[0031] (2) Regarding the VR, a video was produced assuming a situation where mild respiratory depression occurred during implant insertion under intravenous sedation at a dental hospital. Due to the particularity of anesthesia management in dentistry, where the surgical field and the airway area coincide and there are treatment operations such as irrigation, respiratory management is particularly important.

[0032] Currently, implant insertion is frequently performed even in general dentistry, but there is a tendency for more overall management under intravenous sedation rather than general anesthesia. By draping, it is difficult to immediately grasp the patient's expression and condition, which are significantly different from those in normal dental treatment. Therefore, it is important for not only dentists but also dental medical staff such as dental hygienists to consider the patient's condition and be proficient in monitoring vital signs.

[0033] By using VR, it is also possible to view videos of the surrounding situation, so it becomes possible to learn the roles and movements of each medical staff while observing the progress of the surgery. In the research conducted on dental medical staff, inexperienced dental medical staff have a narrow field of vision within VR content and tend to focus their gaze around the patient's face, while dental medical staff who usually pay attention to monitoring and the overall condition have a wider field of vision and tend to send their gaze overall.

[0034] Having a wide field of vision should enable one to obtain various visual information. By knowing the attention pointer in VR, it is expected to be useful for detecting dangerous situations even in the actual medical field.

[0035] (Points for attention in video production) As a point for attention in video production, when the learner moves their line of sight within the HMD, it is difficult for the evaluator to know where the learner is looking. Whether to program to display a pointer on the screen that both the learner and the evaluator can understand where they are gazing (Fig. 6), or deliberately place a biometric monitor at a distant position so that the learner cannot see the biometric monitor unless they consciously look in a way of turning their neck. Gaze

[0036] VR can experience a more immersive situation, but when learners lack knowledge such as in initial training, there is a possibility that they may not be able to understand the occurring phenomena and may not gain sufficient understanding. Therefore, a 2D video was created based on the VR content video, and educational content with explanations necessary for organizing the situation and knowledge on the screen was created separately.

[0037] In addition to the VR video, the situation of the monitor is projected on a separate screen so that it can always be observed. Devices such as putting a check mark or inserting an explanatory slide are made in scenes that should be noted (see Fig. 9(b)). Normally, when producing such content, multiple cameras and cameramen are required, but in the case of a 360-degree camera, fewer units are sufficient. Therefore, less space and personnel are required, and it is considered suitable for a clean environment such as an operating room.

[0038] Precautions during shooting and editing Regarding 360-degree camera equipment, various models are currently in circulation. Even with a high resolution of 4K, in practice, the resolution may decrease when enlarged for viewing, or there are models that cannot use a PC for video editing and can only be edited on a smartphone. It is important to select according to the intended use.

[0039] With a 360-degree camera, an entire direction is photographed using a single camera with a fisheye lens or multiple cameras, and the video is distorted (compressed, encoded) into a 16:9 planar video (as shown in Image 1) and stored. When playing back, it is conversely distorted (expanded, decoded) to virtually reproduce the original full-direction video.

[0040] To pack a wide (i.e., high-information) video in all directions into a 16:9 planar video, if the resolution of the planar image is low (i.e., the information volume is small), when viewing, a part of it is enlarged, and thus the information volume becomes even smaller (i.e., the image quality becomes poor and rough). To prevent a decrease in image quality, it is necessary to set the 16:9 video to be recorded at the highest possible resolution.

[0041] Currently, 4K image quality is at the highest level, but due to the large amount of data, there is a possibility that it cannot be viewed smoothly depending on the PC, etc. during playback and editing. In addition, it is necessary to arrange the camera as close as possible to the subject that you mainly want to show the front of the camera, in order to reduce the magnification ratio when viewing and avoid a decrease in image quality.

[0042] Furthermore, depending on whether the device that the learner views on is an HMD or a smartphone, the appearance will differ depending on the font size and style. What device to let the learner view on is also an important matter in editing. It is important to be compatible with the manufacturer and model method used for shooting during editing.

[0043] (Distribution of VR content) Since it is medical information, security in distribution is important. There are several distribution sites for educational content in dentistry. There is also a method of downloading to an HMD. Since a stronger sense of immersion can be obtained, a more immersive experience is desired. However, since the device is expensive, there is concern about the cost.

[0044] If distributing to multiple people, a method of mounting the learners' own smartphones on VR goggles is good.

[0045] (High-speed content distribution) Although it is an issue for the future, it is considered that by connecting the medical and educational sites in real time using VR, education in a form similar to on-the-job training (OJT) will be possible. In order to transmit and receive high-precision images used in the medical field, a large-capacity and high-speed communication environment is indispensable. For this purpose, a communication method without delay such as 5G (the fifth-generation mobile communication system) is used.

[0046] The concentration of doctors in big cities and the shortage of doctors in local areas have become serious problems. It is considered that by having real-time exchanges of opinions with specialists in remote areas, it will contribute to medical education without status differences. The following describes the embodiments.

[0047] <Embodiments> In this embodiment, the learning of VR by dentists with little experience among medical professionals (dentists, doctors, midwives, nurses, dental hygienists, dental technicians, physical therapists, occupational therapists, emergency medical technicians, etc.) will be described as an example. Also, in this embodiment, the content displayed on the screen of the display unit etc. will be referred to as video, and what is processed in the computer will be referred to as image for description.

[0048] FIG. 1 is a schematic configuration diagram of the medical learning content providing system according to the present embodiment. As shown in FIG. 1, this system connects a leader side 100, a medical learner side (hereinafter referred to as the learner side 200), a medical site side (educational performance) 300, a dental education content service site 400 (hereinafter referred to as the site 400), etc. via a communication network 500 (Internet or dedicated network).

[0049] The leader side 100 includes a leader terminal 110 (personal computer), an HMD (hereinafter referred to as the leader's HMD 130) worn by a supervising physician (hereinafter referred to as the leader MDi), an HMD controller (hereinafter referred to as the leader side HMD controller 120), etc. The medical site side 300 shoots the medical site with a VR camera (360-degree camera: dome camera) 310 and transmits the captured image to the site 400 using a personal computer or server (not shown).

[0050] The site 400 includes a site side server 410 and a person in charge terminal (not shown), etc. The site side server 410 includes a transmission / reception processing unit 420 (program), an educational image generation processing unit 430 (program), a site side memory 440, an educational VR memory 460, etc. The learner side 200 is composed of a 5G-compatible smartphone (hereinafter referred to as the smartphone 270), a personal computer 280 (a monitor is also acceptable), etc.

[0051] An HMD (hereinafter referred to as the learner's HMD 240) is worn on the head of the learner Ai. A controller (hereinafter referred to as the learner's HMD controller 260) is connected to the learner's HMD 240. Furthermore, the learner's HMD 240 is provided with a learner gaze image generation processing unit 230, an HMD image display processing unit 220, a memory 250, etc. Educational VR content, etc. is loaded into the memory 250. The HMD image display processing unit 220 performs 3D display processing on the educational VR content (image) in the memory 250. The learner gaze image generation processing unit 230 is a pointer definition program described later, and defines the pointer Pi indicated by the learner HMD controller 260 in the image to be 3D displayed. Note that in the case of an HMD in the form of mounting a smartphone, the learner gaze image generation processing unit 230 and the HMD image display processing unit 220 are not necessary.

[0052] FIG. 2 is a schematic configuration diagram of the educational image generation processing unit 430 (program) of the present embodiment. As shown in FIG. 2, the educational image generation processing unit 430 includes a viewing direction image reading unit 12, a pointer start / end definition unit 14, a pointer-attached image generation unit 15, an image registration unit 15, and the like. The viewing direction image reading unit 12 reads the movement (including posture and position) of the HMD from the learner side 200 or the instructor side 100, obtains the line-of-sight direction based on this movement of the HMD, and defines the viewing range Si in the educational VR content in the site-side memory 440.

[0053] The image of this viewing range Si is referred to as a viewing image. In the case of the viewing image by the instructor side 100, it is referred to as the instructor-side viewing image KGi, and in the case of the learner side 200, it is referred to as the evaluation tester viewing image LGi. Also, when the learner Ai is evaluated by an evaluator, it is referred to as the examinee viewing image EGi. Related information described later is added to these images to identify the images.

[0054] The read viewing image is read out to the image memory 11 and transmitted (5G) to the learner side 200 or the instructor side 100 for display (3D or 2D). The pointer start / end definition unit 14 transmits and downloads a pointer definition program (for defining the end point of the pointer) in response to an access from the learner side 200 or the instructor side 100. Then, every time the pointer end point Pbi from the learner side 200 or the instructor side 100 is received, a frame is defined on the field-of-view image (KGi, LGi, or EGi) of the image memory 11, and the center coordinates of this frame are defined as the pointer start point Pai (the center coordinates of the field-of-view image), and the pointer end point Pbi is defined in the frame (the coordinates of the field-of-view image).

[0055] The pointer-attached image generation unit 15 obtains a straight line connecting the center coordinates of the layer (hereinafter referred to as the pointer start point Pai) and the pointer end point Pbi, and defines the instructor's gaze direction pointer icon Pi based on this straight line in the layer. This image is referred to as the gaze direction pointer image PWi. Then, the gaze direction pointer image PWi is overlaid on the field-of-view image of the image memory 11, and this is stored in the site-side memory 440 as the gaze image with pointer PGi (HGi, PWi) and transmitted.

[0056] The image registration unit 17 copies (registers) the educational VR content (educational VR image: 360 degrees) of the educational VR memory 460 associated with the gaze image with pointer PGi to the site-side memory 440, and copies (registers) the gaze image with pointer PGi to the memory 16. These images are collectively referred to as the educational VR content with pointer MPi. Then, in response to a request for the educational VR content with pointer MPi from the learner side 200 (the examinee side) or the evaluator side, the educational VR content with pointer MPi is transmitted to the terminal.

[0057] Next, the operation of the medical learning content providing system will be described in more detail with a sequence diagram. First, the generation of the instructor-side educational VR image will be described.

[0058] (Generation of Instructor-Side Educational VR Image) FIG. 3 is a sequence diagram for generating a gaze image with pointer on the instructor side. In the present embodiment, the instructor-side personal computer 110, the instructor-side MDI controller 120, and the instructor-side MDI 130 are collectively referred to as the instructor-side terminal.

[0059] Before explaining the generation of the point-annotated attention image, the registration of basic information will be described. As shown in FIG. 3, the learner side 200 operates the smartphone 270 or a personal computer (a smartphone in this embodiment) to access the site-side server 410 of the site 400 (ID, password) (d10), and stores the learner basic information in the site-side memory 440 (d11).

[0060] This learner basic information consists of the learner's name, specialized field name, age, gender, phone number, email address, registration date and time, learner-side HMD address, etc. Specifically, the smartphone 270 on the learner side 200 and the site-side server 410 communicate and connect via the communication network 500, and the transmission / reception processing unit 420 of the site-side server 410 stores it in the site-side memory 440.

[0061] On the other hand, the person in charge of the medical site 300 (performance) operates a personal computer (not shown) to access the site 400 (site-side server 410), and stores the medical education site basic information in the site-side memory 440 (d12, d14).

[0062] The basic information of the education site consists of the site name (e.g., university name, clinic name, etc.), the name of the registrant, the specialized field name, age, gender, learning name (implant, intramuscular injection, anesthesia,..., personal computer address), date and time of year, education site image name (also referred to as an educational VR image name or educational VR content name), etc.

[0063] On the other hand, the instructor MDi on the instructor side 100 operates the instructor-side personal computer 110 to access the site 400 (site-side server 410), and stores the instructor basic information in the site-side memory 440 (d16, d18).

[0064] The instructor basic information consists of the instructor's name, specialized name, affiliated name, number of years of experience, age, gender, registration date and time, personal computer address, instructor-side HMD address, etc. The medical site side (for educational performance) 300 operates a personal computer to transmit educational VR content (educational VR images) to the site-side server 410 (d20). This educational VR content is associated with the name of the shooting site, the name of the educational VR content (incidents, anesthesia, etc. in dental treatment), the patient's age, gender, height, name, pre-existing conditions, shooting time, camera number, etc.

[0065] An example of the educational VR content (educational VR images) is shown in FIGS. 10(a) and 10(b). FIG. 10(a) is an educational VR image (educational VR content) for dealing with incidents during dental treatment, and FIG. 10(b) is an educational VR image (educational VR content) for anesthesia management under intravenous sedation in oral surgery.

[0066] The transmission / reception processing unit 420 of the site-side server 410 receives the educational VR content (educational VR images) from the medical site side 300 and stores it in the educational VR memory 460 (d24). The educational VR content (educational VR images) is, for example, content regarding responses to incidents during dental treatment, content regarding anesthesia management under intravenous sedation in oral surgery, etc.

[0067] On the other hand, the instructor MDi on the instructor side 100 operates the instructor personal computer 110 to access the site 400 (site-side server 410) to obtain the educational VR content (d26) and display it on the instructor-side HMD 130 (also referred to as the initial image) (d28).

[0068] Initially, it is preferable to output the images of each predetermined visual field when viewing the educational VR image from a predetermined direction to the instructor-side personal computer 110 and display them on the instructor-side HMD 130. Specifically, for the acquisition of the aforementioned educational VR content, a screen for inputting the instructor's name, instructor code, educational content name, content type, shooting date and time, etc. (hereinafter collectively referred to as instructor educational VR request information) and the presence or absence of pointer generation is transmitted from the site-side server 410 for input.

[0069] The transmission and reception processing unit 420 of the site server 410 receives this input information, and collates it with the basic information of the instructor in the site memory 440 for acceptance. For a registered instructor, the request information for instructor education VR is output to the education image generation processing unit 430 to start it.

[0070] The education image generation processing unit 430 allocates the education VR content (education VR image) corresponding to the education content name, content type, shooting site name, shooting date and time, etc. included in the request information for instructor education VR from the education VR memory 460, and outputs this education VR content (education VR image) to the transmission and reception processing unit 420 for transmission to the instructor's personal computer 110 (packet, 5G).

[0071] At this time, if a pointer generation request is included, for the transmission of the aforementioned education VR content (education VR image), a control program (also referred to as a pointer generation program) for generating a pointer to be described later, the request information number for instructor education VR, the number of the education VR content (content identification code), etc. are added and transmitted.

[0072] On the other hand, the computer of the instructor side HMD 130 outputs the attitude θi and position Mi of the gyro sensor (collectively also referred to as the instructor side HMD movement information Ei) to the instructor side personal computer 110 at regular intervals (for example, 10 msec, 50 msec, 100 msec, 200 msec,...).

[0073] Every time the instructor side personal computer 110 receives the instructor side HMD movement information Ei (attitude θi, position Mi, identification information of the instructor side HMD 130, date and time, etc.) of the instructor side HMD 130, it transmits it to the site server 410 (d30).

[0074] For this transmission, the request information number for instructor education VR, the education VR content number, the date and time, etc. (hereinafter collectively referred to as the instructor side VR identification information) are added. The transmission and reception processing unit 420 of the site server 410 outputs this instructor-side HMD movement information Ei to the education image generation processing unit 430.

[0075] The education image generation processing unit 430 reads the instructor-side HMD movement information Ei, defines a viewing point in the viewing direction based on the movement of the instructor-side HMD 130, and defines a predetermined viewing range Si centered on this viewing point based on the posture θi and position Mi.

[0076] Then, an image of this viewing range Si is generated as the instructor-side viewing image KGi (with instructor-side VR identification information added) (d32). This process is performed by the viewing direction image reading unit 12 shown in FIG. 2.

[0077] Then, it is transmitted (5G) to the instructor-side personal computer 110 by the transmission and reception processing unit 420 (d34). Each time the instructor-side personal computer 110 receives the instructor-side viewing image KGi, it outputs it to the instructor-side HMD 130 and causes it to be displayed on its display (it may be for 2D or 3D) (d36).

[0078] When there is a part that the instructor-side viewing image KGi is gazing at, the instructor MDi operates the HMD controller 120 to move the cursor to a predetermined position to determine the end point of the pointer and defines the end point Pbi (d38).

[0079] The instructor-side HMD 130 outputs information on this end point Pbi (including the coordinates in the image of the screen, the number of the instructor-side viewing image KGi, and the instructor-side VR identification information) to the instructor-side personal computer 110, and the instructor-side personal computer 110 transmits it to the site server 410 (d40).

[0080] The transmission and reception processing unit 420 of the site server 410 outputs the pointer end point Pbi to the education image generation processing unit 430, and the education image generation processing unit 430 performs pointer-added fixation image generation processing (d42).

[0081] (Pointer-added fixation image generation processing (d42)) The pointer-annotated visual image generation process will be described with reference to FIG. 2. The pointer start / end point definition unit 14 shown in FIG. 2 defines a frame layer (corresponding to the size of the visual field range) for generating the pointer Pi on the instructor-side visual image KGi in the image memory 11, and causes the visual field direction image reading unit 12 to read (store) the current instructor-side visual image KGi into the image memory 11.

[0082] Furthermore, a frame layer (frame) is defined on the displayed instructor-side visual image KGi. Then, it is determined whether the pointer end point Pbi has been specified by the instructor MDi. The pointer end point Pbi is preferably specified by the instructor-side HMD controller 120. Specifically, the cursor is moved to a predetermined end point position in the direction where the instructor gazes, and a cross mark or the like (star mark, circle mark, etc.) is displayed by a mouse click. This is referred to as the definition of the pointer end point Pbi.

[0083] With the definition of this pointer end point Pbi, the coordinates of the center of the displayed instructor-side visual image KGi (hereinafter referred to as the center Pai) are obtained (the position in the educational VR content), and the pointer end point Pbi is obtained, and the center Pai and the pointer end point Pbi are defined in the frame of the image memory 11 (see FIG. 11(a)). Then, the pointer start point-annotated image definition unit 14 activates the pointer-annotated image generation unit 15.

[0084] The pointer-annotated image generation unit 15 sets the instructor-side visual image KGi as the instructor-gazed image HGi, and defines a pointer Pi (icon) indicating the instructor's gaze direction connecting the center Pai and the pointer end point Pbi in the layer (see FIG. 11(b)). This frame image is referred to as the gaze direction pointer image PWi.

[0085] The following information is associated with this gaze direction pointer image PWi: educational VR content number, educational content name, instructor-side HMD movement information Ei, date, time, start point Pai, end point Pbi, etc.

[0086] Then, the pointer-attached image generation unit 15 overlays the gaze direction pointer image PWi on the instructor's gaze image HGi in the image memory 11, and stores this as the pointer-attached gaze image PGi (HGi, PWi) in the site-side memory 440.

[0087] At this time, related information such as the educational VR content number, the center Pai of the instructor's field-of-view image KGi (the center of the frame: the center of the pointer-attached gaze image PGi), the instructor's name, and the date and time are associated. This is collectively referred to as the pointer-attached gaze image information PJi (the pointer-attached gaze image PGi, the educational VR content number, the center Pai of the instructor's field-of-view image KGi (the center of the frame), the instructor's name, the date and time, etc.).

[0088] FIG. 6 shows the pointer-attached gaze image PGi displayed on the instructor's side HDM 130 or the instructor's side personal computer 110. For example, the instructor's field-of-view image KG11 is taken as the instructor's gaze image HG1, the instructor's gaze direction pointer icon Pi is defined to become the instructor's gaze direction pointer image PW1 and the pointer-attached gaze image PG1, ··· the instructor's field-of-view image KG51 is taken as the instructor's gaze image HG10, the instructor's gaze direction pointer icon Pi is defined to become the instructor's gaze direction pointer image PW10 and the pointer-attached gaze image PG10.

[0089] This instructor's field-of-view image KGi (KG1, KG2, ···) and the pointer-attached gaze image PGi (PG1, PG2, ···) are sequentially output to the instructor's personal computer 110 and displayed (see FIGS. 6 and 7).

[0090] FIG. 6(a) is an example in which a pointer in the line-of-sight direction to be gazed at when the patient complains of feeling unwell is superimposed and displayed. FIG. 6(b) is a scene of intramuscular injection with an EpiPen Trainer (registered trademark), and it is an image in which the part that the learner gazes at is indicated by the pointer Pi. That is, it is possible to teach the patient's expression and the place to pay attention to during intramuscular injection.

[0091] Specifically, the pointer-annotated viewing image PGi (number of educational content information, content name, Pai, Pbi, pointer code, date and time, terminal address, etc.) in the memory 16 is output to the transmission / reception processing unit 420 and transmitted to the instructor-side personal computer 110 (d43). This transmission is performed via 5G.

[0092] Then, the instructor-side personal computer 110 displays (3D) this pointer-annotated viewing image PGi on the instructor-side HMD 130 (d44).

[0093] Then, the instructor-side HMD 130 and the instructor-side personal computer 110 return to the above-mentioned d30 (d46), and display an image corresponding to the movement of the instructor-side HMD 130, and define the pointer Pi (a triangular icon preferably starting from the center of the image and spreading towards the end point) as described above. In FIG. 6 and the like, the icon is shown in stripes. It may also be a simple solid-colored icon without stripes. In addition, the image registration unit 17 of the educational image generation processing unit 430 copies (registers) the educational VR content (educational VR image: 360 degrees) associated with the pointer-annotated viewing image PGi to the site-side memory 440, and also copies (registers) the pointer-annotated viewing image PGi to the memory 16 (d41).

[0094] When copying the pointer-annotated viewing image PGi, it is preferable to rewrite the pointer-annotated viewing image PGi of the same field of view to this pointer-annotated viewing image PGi. These images are collectively referred to as the pointer-annotated educational VR content MPi.

[0095] That is, it is possible to provide educational VR content incorporating an image of a pointer for diagnosis by an experienced instructor or a checkpoint that can confirm the treatment content. In addition, by using VR, it is also possible to view the video of the surrounding situation, so it is possible to learn the roles and movements of each medical staff while observing the progress of the surgery.

[0096] In addition, inexperienced dental medical staff can understand, by viewing this VR content, that dental medical staff who usually monitor and consider the overall condition have a wide field of vision and tend to send overall glances.

[0097] Furthermore, by learning about the eye - care pointers in VR, it will be useful for detecting dangerous situations even in actual medical settings.

[0098] (Learning side) Next, the learning using the learner - side HMD will be explained with reference to the sequence diagram of FIG. 4. Regarding the learning side 200, the educational content will be described as learning content. As shown in FIG. 4, the learner operates the smartphone 270 to access the site 400 and obtain a learner content acquisition screen (not shown).

[0099] For example, enter the learner content name, or the pointer - attached viewing image PGi (educational content information number, content name, Pai, Pbi, pointer code, date - time, terminal address ···), or the presence or absence of a test on this screen, and transmit this information as learner - side VR request information to the site - side server 410 (d50).

[0100] The transmission - reception processing unit 420 of the site - side server 410 determines whether the learner basic information having the learning name included in the learner - side VR request information is stored in the site - side memory 440. If it is stored, the request is accepted.

[0101] Then, it determines the type of image included in the learner - side VR request information. For example, if a pointer - attached viewing image PGi (educational content information number, content name, Pai, Pbi, pointer code, date - time, terminal address ···) is requested, the learning VR content with the learning content name included in this learner - side VR request information is read from the site - side memory 440 and transmitted (d52).

[0102] On the other hand, the learner-side HMD 240 detects the learner HMD movement Fi (posture, position) at regular intervals (d54). Every time the learner-side HMD 240 detects the movement Fi of this learner HMD, it outputs it to the smartphone 270 (d55).

[0103] The smartphone 270 transmits (5G) the movement Fi (posture, position) of the learner HMD to the site-side server 410 (d56). The site-side server 410, for the movement Fi (posture, position) of the learner HMD, the transmission / reception processing unit 420 corresponding thereto obtains the field of view in the pointer-attached educational VR content MPi registered in the site-side memory 440 (similar to the process of d42), reads out the pointer-attached fixation image PGi (PG1, PG2,...), the instructor-side field-of-view image KGi (KG1, KG2,...) of this field of view (d58), and transmits it to the smartphone 270 (d60).

[0104] When the image within the range of the field of view is the pointer-attached fixation image PGi (PG1, PG2,...), it is that image, and when it is the instructor-side field-of-view image KGi (KG1, KG2,...), it is that image. The smartphone 270 receives the transmitted image and outputs it to the learner-side HMD 240 (d62).

[0105] The HMD image processing unit 220 of the learner-side HMD 240 stores this in the memory (d64) and performs 3D display (see Fig. 6) (d66). Therefore, the learner can pre-learn where to fixate during the treatment.

[0106] (Evaluation) Next, the case of testing (also referred to as evaluation) the learner will be described using the sequence diagram of Fig. 5. In the present embodiment, the examiner is referred to as the evaluator, and the learner to be evaluated is referred to as the examinee. As shown in Fig. 5, the evaluator operates the terminal (hereinafter referred to as the instructor terminal) to access the site 400 (site-side server 410) (ID, password), and stores the evaluator basic information in the site-side memory 440 (d70). The educational VR content is referred to as evaluation test VR content.

[0107] For example, it memorizes the learner's name, the evaluator's name, the name of the evaluation content, the date and time, the location, etc. The name of the evaluation content is preferably either the VR content for the evaluation test (VR content for education) or the VR content for education, and is designated by the evaluator.

[0108] On the other hand, the learner operates the smartphone 270 to send information such as the name of the VR content for evaluation (hereinafter referred to as the VR content for evaluation name information) to the site-side server 410 (d72). For example, information such as implant, intramuscular injection, anesthesia, ···, and the computer address) and information such as the date and time are input.

[0109] The transmission / reception processing unit 420 of the site-side server 410 receives the VR content for evaluation name information and causes the education image processing unit 430 to assign the VR content for evaluation corresponding to this VR content for evaluation name information (d74).

[0110] The education image processing unit 430 causes the transmission / reception processing unit 420 to transmit the initial image of the VR content for evaluation to the learner's smartphone 270 (d76).

[0111] At this time, the data of the pointer generation program is also transmitted. The learner's smartphone 270 that receives the evaluation outputs this initial image to the learner's HMD 240 for 3D display (d78).

[0112] On the other hand, the computer of the learner's HMD 240 detects the movement (posture, position) of the learner's HMD 240 at regular intervals as the movement LEi of the examinee's HMD based on the program (d79).

[0113] Then, the detected movement LEi of the examinee's HMD is output to the smartphone 270 (d80). The smartphone 270 transmits the information on the movement LEi (posture, position) of the examinee's HMD to the site-side server 410 (5G) (d82).

[0114] The site-side server 410 receives the information of the movement LEi of the examinee-side HMD, outputs it to the educational image generation processing unit 430, and performs the determination process of the examinee's visual field image LGi (d84). The determination process of the examinee's visual field image LGi obtains the visual field in the evaluation test VR content corresponding to the movement LEi (posture, position) of the examinee-side HMD, and determines the visual field image KGi (KG1, KG2,...) of this visual field as the evaluation image TPi for the learner taking the evaluation test.

[0115] Then, this evaluation image TPi (evaluation test VR content) is transmitted to the smartphone 270 (d86). The smartphone 270 receives the evaluation image TPi (evaluation test VR content) and outputs it to the HMD 240 (d88).

[0116] The HMD 240 performs 3D display of this (d90). When there is a part that the learner being evaluated focuses on in this evaluation image TGi, the HMD controller 260 is operated to move the cursor to a predetermined position to determine the end point of the pointer, and the end point Pbi is defined (d92).

[0117] The HMD controller 260 outputs the information of this end point Pbi (including the coordinates in the screen image, the number of the evaluation image KGi, and the name of the evaluation VR content) to the smartphone 240 via the HMD 240, and the smartphone 270 transmits this pointer end point Pbi to the site-side server 410 (d96). Note that the HMD 240 displays, for example, a cross mark indicating the pointer end point Pbi from the HMD controller 260 (d98).

[0118] On the other hand, the transmission / reception processing unit 420 of the site-side server 410 stores the pointer end point Pbi in the memory (d100), and the educational image generation processing unit 430 determines whether the point end point PBi has been received (d102).

[0119] When the educational image generation processing unit 430 determines that the point end point PBi has been received, it performs the same point-annotated fixation image generation process as above (d104).

[0120] The point-annotated view image generation process is such that the pointer start / end point definition unit 14 shown in FIG. 2 defines a frame layer (corresponding to the size of the visual field range) for generating the pointer Pi on the evaluation tester's view image LGi in the image memory 11, and causes the view direction image reading unit 12 to read (store) the current evaluation tester's view image LGi into the image memory 11.

[0121] And a frame is defined for this evaluation tester's view image LGi. Then, the pointer end point Pbi is defined on the frame of the image memory 11 (see FIG. 11(a)). And on the layer, an evaluation examinee-side fixation direction pointer LPi (icon) connecting the center Pai and the pointer end point Pbi is defined (see FIG. 11(b)). The image composed of this pointer and the frame is called the fixation direction pointer image PWi.

[0122] The following are associated with this fixation direction pointer image PWi: the educational VR content number, the educational content name, the instructor-side HMD movement information Ei, the date, time, the start point Pai, the end point Pbi, etc.

[0123] Then, the pointer-added image generation unit 15 overlays the fixation direction pointer image PWi on the view image in the image memory 11, and stores this as the point-annotated view image PGi (HGi, PWi) in the site-side memory 440 (d108).

[0124] At this time, the following associated information is associated: the educational VR content number, the center Pai of the view image KGi (the center of the frame: the center of the point-annotated view image PGi), the instructor's name, the examinee's name, the date, time, etc. This is collectively called the point-annotated view image information PJi (the point-annotated view image PGi, the educational VR content number, the center Pai (the center of the frame) of the instructor-side view image KGi, the instructor's name, the date, time, etc.).

[0125] Then, the point-annotated view images PGi (PG1, PG2, ···) are transmitted to the smartphone 270 (d110). The smartphone 270 outputs this pointer-annotated fixation image PGi (educational content information number, content name, Pai, Pbi, pointer code, date and time, terminal address, etc.) to the learner-side HMD 240 (d112).

[0126] The HMD 240 performs 3D display of this (d114). Then, the instructor-side HMD 130 and the smartphone 240 return to the aforementioned d79 (d116), display an image corresponding to the movement of the learner-side HMD 130, and define and display the pointer Pi (a triangular icon starting from the center of the image and spreading toward the end point) as described above (d116).

[0127] Also, the smartphone 270 outputs the pointer-annotated fixation image PGi (educational content information number, content name, Pai, Pbi, pointer code, date and time, terminal address, etc.) to the personal computer 280 and displays it on the screen (see Fig. 7). Therefore, the evaluator can confirm the line-of-sight direction at which the examinee is looking, and the examinee taking the evaluation test can also confirm it.

[0128] For example, as the pointer, it is preferable to use evaluation content with a scenario configuration that asks about the differential diagnosis with vasovagal reflex and whether the learner can detect and understand important skin symptoms for that purpose.

[0129] Therefore, the evaluator can confirm from the pointer defined by the learner shown on the separate monitor whether the evaluator is keeping an eye on the patient's condition, such as the state of the patient gripping their hand due to mental tension or the pain of local anesthesia, the patient's expression until they complain of discomfort after anesthesia, and the redness of the skin on the forearm, without missing any details.

[0130] If the learner is looking, the evaluator can confirm what the learner has detected or understood through a written test or oral questioning. For example, when a strong pain stimulus is applied to the oral cavity, which is the trigeminal nerve region, while the patient is in an overly tense state, there is a possibility of causing a vasovagal reflex, or when anaphylactic shock is suspected, what other symptoms may occur, and also, since it is an epinephrine trainer in the video, the color of the syringe is white, but the color of the syringe for adults of EpiPen is either yellow or green. It is possible to ask about peripheral knowledge such as this.

[0131] In this way, while having the learner experience dealing with accidental patients in VR, the evaluator can confirm through oral or written tests whether the learner was looking in the direction of each checkpoint and whether they accurately grasped and understood the situation.

[0132] <Other Embodiments> As a point to note in video production, when the learner moves their line of sight inside the HMD, it is difficult for the evaluator to know where the learner is looking. An example was shown where a program is required to display a pointer on the screen that both the learner and the evaluator can understand where the learner is gazing (see FIGS. 6 and 7). However, VR content that can be evaluated without preparing this program may also be acceptable. Gaze For example, as shown in FIG. 8, a biological monitor or the like is deliberately placed at a distant position and arranged in a place where the learner cannot see the biological monitor unless they consciously look in a way such as shaking their head.

[0133] Also, although VR can experience a more immersive situation, when the learner has insufficient knowledge such as in initial training, there is a possibility that they may not be able to understand the occurring phenomena and may not gain sufficient understanding.

[0134] Therefore, as shown in FIGS. 9(a) and 9(b), a 2D video was created based on the video of the educational VR content, and educational content with explanations necessary for organizing the situation and knowledge added on the screen was created separately.

[0135]

[0136] ​For example, as shown in Fig. 9(a), in addition to the VR video, the status of the monitor is projected on a separate screen (ai) so that it can always be observed. Devises such as putting a check mark (Qi) or inserting an explanatory slide are made for the scenes that need attention (Fig. 9(b)). Also, as a comment Bi, for example, display "Strange breathing is a breathing pattern that is the opposite of natural breathing where the chest sinks during inhalation and protrudes during inhalation, and it occurs due to upper airway obstruction."

[0137] Normally, when producing such content, a plurality of cameras and cameramen are required. However, in the case of a 360-degree camera, fewer units are needed, so less space and personnel are required, and it is considered suitable for a clean environment such as an operating room.

[0138] Also, as shown in Fig. 8, for educational VR content, the biological monitor is arranged so that the biological monitor does not enter the field of view unless the learner consciously turns the right (HMD) (this is a situation slightly different from the original clinical situation. Normally, an anesthesiologist should be near the biological monitor).

[0139] That is, it is arranged so that the biological monitor does not enter unless one turns towards H1→H2→H3. With such content, a program for pointer definition becomes unnecessary.

Explanation of symbols

[0140] 100 Instructor side 200 Learner side 300 Medical site side 400 Site 500 Communication network 110 Instructor terminal 410 Site-side server 430 Educational image generation processing unit 430 Educational image generation processing unit

Claims

1. Based on the movement of a head-mounted display worn by a medical instructor or learner, A program for displaying, on an educational VR image for treating a patient displayed on the head-mounted display, a pointer icon indicating the line of sight from a medical instructor to the point of attention to the patient on the educational VR image, A computer, (A). Means for storing an educational VR image in a storage means; (B). Means for reading movement information of the head-mounted display worn by the medical instructor or learner at regular intervals; (C). Means for determining a line-of-sight center on the educational VR image and determining a field-of-view range based on the movement information of the head-mounted display; (D). Means for displaying a field-of-view image of the determined field-of-view range on a screen of the instructor's or learner's terminal; (E). Means for defining, by the medical instructor's or learner's terminal, the position where the pointer icon ends as the point of attention on this field-of-view image; (F). Means for defining a starting point at the center of this field-of-view image along with the definition of the end point, and obtaining the coordinates of this starting point and the coordinates of the end point; (G). Starting from the coordinates of the starting point of the field-of-view image, between the coordinates of this starting point and the coordinates of the end point, overlapping a striped triangular icon whose width gradually widens from the starting point towards the end point as a pointer icon on the field-of-view image, and generating this as a pointer-annotated view image of the medical instructor or learner in the storage means; (H). An educational VR content providing program for executing the function as means for transmitting and displaying the pointer-annotated view image of the medical instructor on the learner's terminal.

2. The instructor's terminal has a head-mounted display, a controller, a monitor, and a personal computer, and is connected to a server of an educational VR content site via a communication network, The server, The learner's terminal has a head-mounted display, a controller, a smartphone having a high-speed large-capacity communication function, and a personal computer, and is connected to the server of the educational VR content site via the communication network using the smartphone, The educational VR content providing program according to claim 1, which executes the functions as the means (A) to (H).

Images