Information processing system, information processing method, and information processing program

Abstract

To provide an information processing system, an information processing method, and an information processing program that allow appropriate training to be performed.SOLUTION: An information processing system includes at least one processor. The processor acquires a first visual field image showing a visual field of a first user and a second visual field image showing a visual field of a second user; performs control of causing a second display device viewed by the second user to display first interest information indicating a position of at least one first region of interest included in the first visual field image; and performs control of causing a first display device viewed by the first user to display second interest information indicating a position of at least one second region of interest included in the second visual field image.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, a method is known in which a display device such as an Augmented Reality (AR) device that allows a user to visually recognize a real space and a displayed image is applied to a training system for a user to learn an operation. For example, Patent Document 1 discloses that, in relation to learning the probe operation of an ultrasound imaging device, a model video is displayed superimposed on the learner's visual field image on a glasses-type device worn by the learner. It also discloses that the display content of the model video is dynamically changed according to the characteristics of the learner's work operation that are included in the learner's visual field image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-144233 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there is a demand for technology that can assist teachers in teaching students in real time in order to perform more advanced training according to work procedures, tools used, objects, situations, and the proficiency levels of student users, etc. For example, when a teacher and a student perform similar actions in parallel, there is a demand for technology that allows the student to imitate the teacher's actions or allows the teacher to give advice to the student's actions.

[0005] The present disclosure provides an information processing system, an information processing method, and an information processing program that enable appropriate training. [Means for solving the problem]

[0006] A first aspect of the present disclosure is an information processing system comprising at least one processor, which acquires a first field of view image showing a field of view of a first user and a second field of view image showing a field of view of a second user, controls first interest information indicating the position of at least one first area of ​​interest included in the first field of view image to be displayed on a second display device viewed by the second user, and controls second interest information indicating the position of at least one second area of ​​interest included in the second field of view image to be displayed on a first display device viewed by the first user.

[0007] In the above aspect, the processor may control the second display device to display, as the first information of interest, a semi-transparent image of the first field of view image.

[0008] In the above aspect, the processor may extract at least one first region of interest from the first field of view image, and control the second display device to display information indicating the outline of the extracted at least one first region of interest as first interest information.

[0009] In the above aspect, the processor may extract at least one first region of interest from the first field of view image, extract at least one second region of interest from the second field of view image, match the first and second regions of interest that are of the same type, determine whether the positions of the matched first and second regions of interest are different, and if the determination is positive, control the second display device to display information indicating the difference between the positions of the matched first and second regions of interest as first interest information.

[0010] In the above aspect, when the above judgment is negative, the processor may control the second display device to display, as the first interest information, information indicating a difference between the position of the associated first region of interest and the position of the second region of interest in a display form different from that when the above judgment is positive.

[0011] In the above aspect, the processor may acquire the first field of view image and the second field of view image over time, derive a correction value as a difference between the position of the first field of view image and the position of the second field of view image extracted and associated from each of the first field of view image and the second field of view image at an initial point in time, and make a judgment based on the difference corrected using the correction value between the position of the first field of view image and the position of the second field of view image extracted and associated from each of the first field of view image and the second field of view image at a point in time later than the initial point in time.

[0012] In the above aspect, the processor may control the display of the left image, obtained by dividing the first field of view image into two halves, left and right, as the first interest information, on the left side of the area on the second display device where the field of view of the second user is displayed, and may control the display of the right image, obtained by dividing the first field of view image into two halves, left and right, as the first interest information, on the right side of the area on the second display device where the field of view of the second user is displayed.

[0013] In the above aspect, the processor may acquire first field of view images over time, and determine whether there has been a change in the position or orientation of the first area of ​​interest from the first field of view images acquired over time, and if the determination is positive, control the second display device to display information indicating that there has been a change in the position or orientation of the first area of ​​interest.

[0014] In the above aspect, the processor may acquire first gaze information of the first user, and control the second display device to display the first gaze information of the first user.

[0015] In the above aspect, the processor may acquire second gaze information of the second user, identify a region of interest of interest that is a second region of interest that the second user is focusing on in the second field of view image based on the second gaze information, and if the region of interest of interest is a region of interest of a predetermined type, acquire related information related to the region of interest of interest, and control the related information to be superimposed on the region of interest of interest and displayed on the second display device.

[0016] In the above aspect, the processor may acquire a first posture image and a second posture image obtained by photographing the first user and the second user, respectively, from at least one same direction, and control the first posture image and the second posture image to be displayed outside an area on the second display device in which the field of view of the second user is displayed.

[0017] In the above aspect, the processor may acquire a voice of the first user, convert the voice of the first user into text information, and control the second display device to display the text information.

[0018] In the above aspect, the processor may generate a second composite image by combining information indicating the position of the first area of ​​interest with the second field of view image, generate a first composite image by combining information indicating the position of the second area of ​​interest with the first field of view image, control the second composite image to be displayed on the second display device, and control the first composite image to be displayed on the first display device.

[0019] In the above aspect, the first field of view image and the second field of view image indicate the respective fields of view of the first user and the second user when operating the same type of tool, and the first area of ​​interest and the second area of ​​interest may be at least one of a body part of the first user or the second user operating the tool, at least a part of the tool, an object to which the tool is to be used, and an edge of the first field of view image or the second field of view image, respectively.

[0020] In the above embodiment, the processor may obtain parameter information relating to the tool and control the second display device to display the parameter information.

[0021] In the above aspect, the processor may control the first information of interest to be displayed on a third display device for viewing by a third user.

[0022] In the above aspect, the processor may acquire a first sub-field of view image showing the field of view of the first sub-user and a second sub-field of view image showing the field of view of the second sub-user, control information indicating the position of the first sub-region of interest contained in the first sub-field of view image to be displayed on a second sub-display device viewed by the second sub-user, and control information indicating the position of the second sub-region of interest contained in the second sub-field of view image to be displayed on the first sub-display device viewed by the first sub-user.

[0023] In the above aspect, the processor may acquire a second sub-field of view image showing the field of view of the second sub-user, and control the second display device to display the first interest information and information indicating the position of the second sub-region of interest contained in the second sub-field of view image.

[0024] A second aspect of the present disclosure is an information processing method in which a computer executes a process of acquiring a first field of view image showing the field of view of a first user and a second field of view image showing the field of view of a second user, controlling first interest information indicating the position of at least one first area of ​​interest included in the first field of view image to be displayed on a second display device viewed by the second user, and controlling second interest information indicating the position of at least one second area of ​​interest included in the second field of view image to be displayed on the first display device viewed by the first user.

[0025] A second aspect of the present disclosure is an information processing program configured to cause a computer to execute a process of acquiring a first field of view image showing the field of view of a first user and a second field of view image showing the field of view of a second user, controlling first interest information indicating the position of at least one first area of ​​interest included in the first field of view image to be displayed on a second display device viewed by the second user, and controlling second interest information indicating the position of at least one second area of ​​interest included in the second field of view image to be displayed on the first display device viewed by the first user. Effect of the Invention

[0026] According to the above aspects, the information processing system, the information processing method, and the information processing program disclosed herein enable appropriate training. [Brief description of the drawings]

[0027] [Figure 1] 1 is a schematic diagram showing an example of an overall configuration of an endoscope system. [Diagram 2] 1 is a schematic diagram showing an example of an overall configuration of an endoscope system. [Diagram 3] 1 is a schematic diagram showing an example of an overall configuration of a training system. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of an HMD. [Diagram 5] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device. [Figure 6] FIG. 2 is a block diagram showing an example of a functional configuration of an information processing device. [Figure 7] FIG. 13 is a diagram showing an example of a teacher's first field of view image. [Figure 8] FIG. 13 is a diagram showing an example of a second field of view image of a student. [Figure 9] FIG. 13 is a diagram showing an example of an image displayed on a teacher's HMD. [Figure 10] FIG. 13 is a diagram showing an example of an image displayed on a student's HMD. [Figure 11] FIG. 13 is a diagram showing an example of an image displayed on a student's HMD. [Figure 12] FIG. 13 is a diagram showing an example of a state in which additional information is displayed on a student's HMD. [Figure 13] FIG. 13 is a diagram showing an example of a state in which additional information is displayed on a student's HMD. [Figure 14] FIG. 13 is a diagram showing an example of a state in which additional information is displayed on a student's HMD. [Figure 15] FIG. 13 is a diagram showing an example of a state in which additional information is displayed on a student's HMD. [Figure 16] 11 is a flowchart illustrating an example of information processing. [Figure 17] FIG. 13 is a diagram showing an example of an image displayed on a student's HMD. [Figure 18] FIG. 13 is a diagram showing an example of an image displayed on a student's HMD. [Figure 19]FIG. 1 is a schematic diagram showing an example of the overall configuration of a training system when there are multiple students. [Figure 20] FIG. 1 is a schematic diagram showing an example of the overall configuration of a training system for training in pairs. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0029] First, an endoscope system 10 to which the technology of the present disclosure can be applied will be described with reference to Figures 1 and 2. The endoscope system 10 is used by a doctor 12 in an endoscopic examination or the like. The endoscopic examination is assisted by staff such as a nurse 14.

[0030] 1, the endoscope system 10 includes an endoscope scope 16, a display device 18, a light source device 20, a control device 22, and an image processing device 24. The light source device 20, the control device 22, and the image processing device 24 are installed on a wagon 34. The wagon 34 has a plurality of stands arranged vertically, and the image processing device 24, the control device 22, and the light source device 20 are installed from the lower stand to the upper stand. In addition, the display device 18 is installed on the top stand of the wagon 34.

[0031] The endoscope system 10 is a modality for performing medical treatment inside the body of a subject 26 (e.g., a patient) using an endoscope scope 16. The endoscope scope 16 is used by a doctor 12 and inserted into a body cavity (e.g., a hollow organ such as the large intestine, stomach, duodenum, and trachea) of the subject 26. The endoscope system 10 causes the endoscope scope 16 inserted into the body cavity of the subject 26 to capture images of the inside of the body cavity of the subject 26, and performs various medical procedures in the body cavity as necessary.

[0032] As an example, in this embodiment, the endoscope scope 16 is inserted into the large intestine 28 of the subject 26. The endoscope system 10 captures an image of the inside of the large intestine 28 of the subject 26, thereby acquiring and outputting an image showing the state inside the large intestine 28. In this embodiment, the endoscope system 10 has an optical imaging function of capturing an image of reflected light obtained by irradiating the inside of the large intestine 28 with light 30 and reflecting it off a wall 32 of the large intestine 28.

[0033] The control device 22 controls the entire endoscope system 10. The image processing device 24, under the control of the control device 22, performs various types of image processing on the image obtained by imaging the intestinal wall 32 by the endoscope scope 16.

[0034] The display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an organic EL (Electro-Luminescence) display. Instead of or in addition to the display device 18, a smartphone, a tablet terminal, or the like may be used.

[0035] A screen 35 is displayed on the display device 18. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged side by side within the screen 35. In the example shown in FIG. 1, a first display area 36 and a second display area 38 are shown as examples of the plurality of display areas. The first display area 36 is used as a main display area, and the second display area 38 is used as a sub-display area.

[0036] An endoscopic moving image 39 is displayed in the first display area 36. The endoscopic moving image 39 is a moving image acquired by imaging an intestinal wall 32 in the large intestine 28 of the subject 26 with the endoscope scope 16. In the example shown in Fig. 1, a moving image showing the intestinal wall 32 is shown as an example of the endoscopic moving image 39.

[0037] 1, the intestinal wall 32 shown in the endoscopic video 39 includes a lesion 42 as an observation target region gazed upon by the doctor 12. The doctor 12 can visually recognize the state of the intestinal wall 32 including the lesion 42 through the endoscopic video 39. Examples of the lesion 42 include a neoplastic polyp and a non-neoplastic polyp.

[0038] The image displayed in the first display area 36 is one frame 40 included in a moving image that includes a plurality of frames 40 arranged in chronological order. In other words, the first display area 36 displays the plurality of frames 40 arranged in chronological order at a default frame rate (e.g., several tens of frames per second).

[0039] The second display area 38 displays medical information 44, which is information related to medical care. Examples of the medical information 44 include information that assists the medical judgment of the doctor 12. Examples of such information include various types of information related to the subject 26 and information obtained by performing image analysis on the endoscopic video image 39 using CAD (Computer Aided Diagnosis / Detection) technology.

[0040] 2, the endoscope 16 includes an operation section 46 and an insertion section 48. The insertion section 48 is partially curved by operating the operation section 46. The insertion section 48 is inserted into the large intestine 28 while curving in accordance with the shape of the large intestine 28 in accordance with the operation of the operation section 46 by the doctor 12.

[0041] The distal end 50 of the insertion section 48 is provided with a camera 52, an illumination device 54, and a treatment tool opening 56. The camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50. Note that, although an example in which the camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50 is given here, the present invention is not limited thereto, and the camera 52 and the illumination device 54 may be provided on the side surface of the distal end 50, so that the endoscope 16 is configured as a side-viewing mirror.

[0042] The camera 52 is inserted into the body cavity of the subject 26 to capture an image of an observation target region. In this embodiment, the camera 52 captures an image of the inside of the subject 26 (e.g., inside the large intestine 28) to obtain an endoscopic moving image 39. Examples of the camera 52 include a complementary metal oxide semiconductor (CMOS) camera and a charge coupled device (CCD) camera.

[0043] The illumination device 54 has illumination windows 54A and 54B. The illumination device 54 irradiates light 30 through the illumination windows 54A and 54B. Examples of the light 30 irradiated from the illumination device 54 include visible light (e.g., white light) and non-visible light (e.g., near-infrared light). The illumination device 54 also irradiates special light through the illumination windows 54A and 54B. Examples of the special light include light for Blue Light Imaging (BLI) and light for Linked Color Imaging (LCI). The camera 52 captures an image of the inside of the large intestine 28 by an optical method while the light 30 is irradiated inside the large intestine 28 by the illumination device 54.

[0044] The treatment tool opening 56 is an opening for allowing a treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 is also used as a suction port for sucking blood, internal waste, and the like, and as a delivery port for delivering fluid.

[0045] A treatment tool insertion port 60 is formed in the operation section 46, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60. The treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56. In the example shown in FIG. 2, a puncture needle is shown as the treatment tool 58 protruding from the treatment tool opening 56. Note that the treatment tool 58 is not limited to a puncture needle, and may be, for example, a grasping forceps, a papillotomy knife, a snare, a catheter, a guide wire, a cannula, a puncture needle with a guide sheath, or the like.

[0046] The endoscope 16 is connected to a light source device 20 and a control device 22 via a universal cord 62. An image processing device 24 and a reception device 64 are connected to the control device 22. In addition, a display device 18 is connected to the image processing device 24. That is, the control device 22 is connected to the display device 18 via the image processing device 24.

[0047] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and a remote control device.

[0048] The control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the image processing device 24.

[0049] The light source device 20 emits light under the control of the control device 22, and supplies the light to the illumination device 54. A light guide is built into the illumination device 54, and the light supplied from the light source device 20 passes through the light guide and is irradiated from illumination windows 54A and 54B. The control device 22 causes the camera 52 to capture an image, acquires an endoscopic moving image 39 from the camera 52, and outputs it to a predetermined output destination (for example, the image processing device 24).

[0050] The image processing device 24 supports the endoscopic examination by performing various image processing on the endoscopic moving image 39 input from the control device 22. The image processing device 24 outputs the endoscopic moving image 39 that has been subjected to various image processing to a predetermined output destination (for example, the display device 18 and the control device 22).

[0051] In this example, the image processing device 24 is shown as an external device for expanding the functions performed by the control device 22, and therefore an example is shown in which the control device 22 and the display device 18 are indirectly connected via the image processing device 24, but this is not limiting. For example, the display device 18 may be directly connected to the control device 22. In this case, for example, the function of the image processing device 24 may be included in the control device 22, or the control device 22 may be equipped with a function to cause a server (not shown) to execute the same process as that executed by the image processing device 24 and receive and use the results of the process by the server.

[0052] The endoscope system 10 is also communicably connected to an information processing device 300 and other external devices (not shown). Examples of the external devices include a server and / or a client terminal (such as a personal computer, a smartphone, or a tablet terminal) that manages various information such as an electronic medical record. The external device receives information transmitted from the endoscope system 10 and executes processing using the received information (such as processing to store the information in an electronic medical record, etc.).

[0053] However, it is known that it is difficult to learn how to hold, the direction, and the strength of force to use the endoscope 16 in the endoscope system 10. For example, as shown in Fig. 1, the doctor 12 operates the operation unit 46 with the left hand and the insertion unit 48 with the right hand while looking at the display device 18 so that the tip 50 of the endoscope 16 reaches the desired observation area. In particular, when the large intestine 28 is the subject of the examination, there is a large individual difference in the course of the large intestine 28, so it is necessary to flexibly change the operation depending on the state of the subject 26 (for example, whether the subject appears calm or in pain, etc.).

[0054] Therefore, the training system 100 of the present disclosure allows a teacher user and a student user to check each other's actions when performing similar actions in parallel, thereby allowing the student to imitate the teacher's actions and the teacher to give advice on the student's actions.

[0055] A training system 100 of the present disclosure will be described with reference to Fig. 3. In this embodiment, the training system 100 includes two endoscope systems 10 used by a teacher doctor 12 (hereinafter referred to as "teacher 12A") and a student doctor 12 (hereinafter referred to as "student 12B"), respectively. The two endoscope systems 10 may be constructed in the same facility or in remote locations.

[0056] Hereinafter, the endoscopic system 10 used by the teacher 12A will be referred to as the "teacher-side endoscopic system 10A," and the endoscopic system 10 used by the student 12B will be referred to as the "student-side endoscopic system 10B." In addition, in the following description, when distinguishing between the configuration included in the teacher-side endoscopic system 10A and the configuration included in the student-side endoscopic system 10B, "A" and "B" will be added to the end of the respective reference numerals. The teacher 12A is an example of a first user of the present disclosure, and the student 12B is an example of a second user of the present disclosure.

[0057] The subjects 26A and 26B used in the training system 100 may be human bodies, but are preferably made of training models (phantoms). This is because, in the present training system 100, training is performed by the student 12B imitating the operations of the teacher 12A, so it is preferable to make the conditions of the subjects 26A and 26B the same.

[0058] In the training system 100, the teacher 12A and the student 12B wear a head mounted display (HMD) 200 as an example of a display device that they each view.

[0059] An example of a hardware configuration of the HMD 200 will be described with reference to Fig. 4. As shown in Fig. 4, the HMD 200 includes a CPU (Central Processing Unit) 251, a non-volatile storage unit 252, and a memory 253 as a temporary storage area. The HMD 200 also includes a display unit 254, an operation unit 255 such as a touch panel, and an I / F (Interface) unit 256. The HMD 200 also includes a motion sensor 260 that detects a motion of a user wearing the HMD 200, a gaze sensor 261 that detects a gaze, a camera 262, and a microphone 263.

[0060] The storage unit 252 is realized by, for example, a storage medium such as a flash memory. The storage unit 252 stores a control program 257 that performs overall control of the HMD 200. The CPU 251 reads the control program 257 from the storage unit 252, loads it into the memory 253, and executes the loaded control program 257.

[0061] The display unit 254 is configured to allow a user wearing the HMD 200 to view an image (virtual image). For example, the display unit 254 may use a transmissive or non-transmissive display, or may project an image onto the user's retina. The I / F unit 256 performs wired or wireless communication with the information processing device 300 and other external devices.

[0062] The motion sensor 260 is a sensor that detects the motion of the user wearing the HMD 200, and is, for example, an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. The gaze sensor 261 is a sensor that detects the gaze point of the user wearing the HMD 200. The camera 262 is a camera that captures the real space observed by the user wearing the HMD 200, and is, for example, a digital camera such as a CMOS camera. The microphone 263 collects the voice of the user wearing the HMD 200 and surrounding sounds.

[0063] The CPU 251, memory unit 252, memory 253, display unit 254, operation unit 255, I / F unit 256, motion sensor 260, gaze sensor 261, camera 262, and microphone 263 are connected via a bus 258 such as a system bus and a control bus so as to be able to exchange various information with each other.

[0064] Hereinafter, when distinguishing between the HMD 200 worn by the teacher 12A and the HMD 200 worn by the student 12B, they will be referred to as "HMD 200A" and "HMD 200B", respectively. In addition, in the following description, when distinguishing between the configuration included in the HMD 200A and the configuration included in the HMD 200B, "A" and "B" will be added to the end of each reference numeral. The HMD 200A is an example of a first display device seen by a first user of the present disclosure, and the HMD 200B is an example of a second display device seen by a second user of the present disclosure.

[0065] The training system 100 also includes an information processing device 300. The information processing device 300 is configured to be connectable to the control device 22A and HMD 200A of the teacher side endoscope system 10A, and the control device 22B and HMD 200B of the student side endoscope system 10B via a wired or wireless network. The training system 100 is an example of an information processing system of the present disclosure.

[0066] An example of a hardware configuration of the information processing device 300 will be described with reference to Fig. 5. As shown in Fig. 5, the information processing device 300 includes a CPU 351, a non-volatile storage unit 352, and a memory 353 as a temporary storage area. The information processing device 300 also includes a display 354 such as a liquid crystal display, an operation unit 355 such as a touch panel, a keyboard, and a mouse, and an I / F unit 356. The I / F unit 356 performs wired or wireless communication with the control device 22A, the control device 22B, the HMD 200A, the HMD 200B, and other external devices.

[0067] The storage unit 352 is realized by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. A training program 357 for the information processing device 300 is stored in the storage unit 352. The CPU 351 reads the training program 357 from the storage unit 352, expands it in the memory 353, and executes the expanded training program 357. The training program 357 is an example of an information processing program of the present disclosure.

[0068] The CPU 351, storage unit 352, memory 353, display 354, operation unit 355, and I / F unit 356 are connected to each other via a bus 358 such as a system bus or a control bus so as to be able to transmit and receive various information to each other. As the information processing device 300, for example, a personal computer, a server computer, a smartphone, a tablet terminal, a wearable terminal, or the like can be appropriately applied.

[0069] An example of a functional configuration of the information processing device 300 will be described with reference to Fig. 6. As shown in Fig. 6, the information processing device 300 includes an acquisition unit 380, an extraction unit 382, ​​a determination unit 384, and a display control unit 386. The CPU 351 executes the training program 357, whereby the CPU 351 functions as the acquisition unit 380, the extraction unit 382, ​​the determination unit 384, and the display control unit 386.

[0070] The acquisition unit 380 acquires a first field of view image 210A showing the field of view of the teacher 12A from the HMD 200A worn by the teacher 12A. The acquisition unit 380 also acquires a second field of view image 210B showing the field of view of the student 12B from the HMD 200B worn by the student 12B. Here, the first field of view image 210A and the second field of view image 210B show the respective fields of view when the teacher 12A and the student 12B operate the same type of tool (e.g., the endoscope 16).

[0071] As an example, FIG. 7 shows a first visual field image 210A showing the visual field of the teacher 12A operating the teacher-side endoscope system 10A. FIG. 8 shows a second visual field image 210B showing the visual field of the student 12B operating the student-side endoscope system 10B. In general, exemplary states such as how to hold the endoscope scope 16, the standing position relative to the subject 26 and the display device 18, and the visual field (objects to be captured in the field of view and their positions) are determined in advance in an endoscopic examination. The first visual field image 210A shows an appropriate state in which the display device 18A, the operation unit 46A held in the left hand, the insertion unit 48A held in the right hand, and the state of the subject 26A (insertion port) can all be viewed at the same time. On the other hand, the second visual field image 210B shows an inappropriate state in which the operation unit 46B (left hand) is too close to the face, and the display device 18B and the subject 26B cannot be viewed.

[0072] The display control unit 386 controls the HMD 200B worn by the student 12B to display the first interest information indicating the position of at least one first region of interest included in the first field of view image 210A of the teacher 12A acquired by the acquisition unit 380. That is, the display control unit 386 causes the HMD 200B worn by the student 12B to display the arrangement in which the teacher 12A views the regions of interest. This allows the student 12B to see the field of view of the teacher 12A when operating the teacher-side endoscope system 10A as a model, and to learn the appropriate method of operating the student-side endoscope system 10B from the difference between the field of view of the teacher 12A and the student 12B.

[0073] Similarly, the display control unit 386 controls the HMD 200A worn by the teacher 12A to display second interest information indicating the position of at least one second region of interest included in the second field of view image 210B of the student 12B acquired by the acquisition unit 380. That is, the display control unit 386 causes the HMD 200A worn by the teacher 12A to display the arrangement in which the student 12B is viewing the region of interest. This allows the teacher 12A to see the field of view when the student 12B operates the endoscope system 10B, and to point out inappropriate points and provide guidance on a more preferable operating method.

[0074] Specifically, the extraction unit 382 may extract at least one first region of interest from the first field of view image 210A. Similarly, the extraction unit 382 may extract at least one second region of interest from the second field of view image 210B. The display control unit 386 may control the HMD 200B of the student 12B to display, as the first interest information, information indicating the outline of at least one first region of interest extracted by the extraction unit 382. Similarly, the display control unit 386 may control the HMD 200A of the teacher 12A to display, as the second interest information, information indicating the outline of at least one second region of interest extracted by the extraction unit 382.

[0075] As an example, Fig. 9 shows a first field of view image 210A on which information showing the outline of the second region of interest is superimposed, which is displayed on the HMD 200A worn by the teacher 12A. In Fig. 9, the display device 18B (outline 18Br), the operation unit 46B (outline 46Br), and the insertion unit 48B (outline 48Br) are illustrated as the second region of interest. Fig. 10 shows a second field of view image 210B on which information showing the outline of the first region of interest is superimposed, which is displayed on the HMD 200B worn by the student 12B. In Fig. 10, the display device 18A (outline 18Ar), the operation unit 46A (outline 46Ar), the insertion unit 48A (outline 48Ar), and the subject 26A (outline 26Ar) are illustrated as the first region of interest.

[0076] 9 and 10, the first region of interest and the second region of interest may each be at least one of a body part (e.g., a hand and a foot) of the teacher 12A or the student 12B who operates the instrument, at least a part of the instrument, and an object (e.g., a subject 26) on which the instrument is to be used. The at least a part of the instrument is, for example, the operation unit 46 and the insertion unit 48 of the endoscope 16. Also, as shown in Figs. 9 and 10, the display form for each region of interest may be changed by using different line types to indicate the contours.

[0077] It is also known that in endoscopic examinations, after first determining the field of view, an expert often proceeds with the operation by moving only the line of sight without changing the direction of the face (i.e., the field of view is fixed).On the other hand, a novice has a tendency to proceed with the operation by directing the whole face towards the object (the operation unit 46, the insertion unit 48, the display device 18, etc.) that catches his / her attention at the time (i.e., the field of view fluctuates).

[0078] Therefore, as shown in FIG. 11, the first region of interest may be the edge of the first field of view image 210A. In this case, the display control unit 386 may control the HMD 200B of the student 12B to display information (contour 210Ar) indicating the edge of the first field of view image 210A as the first interest information. According to this embodiment, the student 12B can notice that his / her field of view is in a range different from that of the teacher 12A, and can return to an appropriate field of view. Similarly, the second region of interest may be the edge of the second field of view image 210B. In this case, the display control unit 386 may control the HMD 200A of the teacher 12A to display information indicating the edge of the second field of view image 210B as the second interest information.

[0079] In addition, when the display unit 254B of the HMD 200B of the student 12B is a non-transmissive type, the display control unit 386 generates a second composite image by combining information indicating the position of the first region of interest of the teacher 12A with the second field of view image 210B of the student 12B captured by the camera 262B of the HMD 200B. Then, the display control unit 386 controls the display unit 254B to display the generated second composite image. Similarly, when the display unit 254A of the HMD 200A of the teacher 12A is a non-transmissive type, the display control unit 386 generates a first composite image by combining information indicating the position of the second region of interest of the student 12B with the first field of view image 210A of the teacher 12A captured by the camera 262A of the HMD 200A. Then, the display control unit 386 controls the display unit 254A to display the generated first composite image.

[0080] On the other hand, when the display unit 254B of the HMD 200B of the student 12B is a transmissive type, the display control unit 386 controls the display unit 254B to project information indicating the position of the first region of interest of the teacher 12A. Similarly, when the display unit 254A of the HMD 200A of the teacher 12A is a transmissive type, the display control unit 386 controls the display unit 254A to project information indicating the position of the second region of interest of the student 12B. In other words, when the display unit 254 is a transmissive type, the real space is visually recognized as a real image.

[0081] Furthermore, the display control unit 386 may perform control so that additional information other than the first interest information and the second interest information is displayed on the HMD 200. Below, a case where additional information is displayed on the HMD 200B worn by the student 12B will be described, but similar additional information can also be displayed on the HMD 200A worn by the teacher 12A.

[0082] For example, as shown in Fig. 12, the first gaze information 212A (illustrated by a cross) of the teacher 12A may be superimposed on the second visual field image 210B of the student 12B. Specifically, the acquisition unit 380 may acquire the first gaze information 212A of the teacher 12A detected by the gaze sensor 261A provided in the HMD 200A of the teacher 12A. The display control unit 386 may control the first gaze information 212A of the teacher 12A to be displayed on the HMD 200B of the student 12B. According to this embodiment, the student 12B can know where the teacher 12A is currently gazing, and can deepen his / her understanding of what operation he / she should perform.

[0083] Also, for example, as shown in FIG. 12, rotation information 214A (illustrated by an arrow) indicating the direction when the teacher 12A rotated the insertion part 48A may be superimposed on the second field of view image 210B of the student 12B. Specifically, the acquisition unit 380 may acquire the first field of view image 210A of the teacher 12A over time. The determination unit 384 may determine whether or not there is a change in the position or orientation of the first region of interest from the first field of view image 210A acquired over time by the acquisition unit 380. When the determination by the determination unit 384 is positive, the display control unit 386 may control the HMD 200B of the student 12B to display information indicating that there has been a change in the position or orientation of the first region of interest. According to this embodiment, the student 12B can easily determine what operation the teacher 12A has performed, and therefore can deepen his / her understanding of what operation he / she should perform.

[0084] Also, for example, as shown in Fig. 13, an image showing the postures of the teacher 12A and the student 12B may be displayed outside the area where the field of view of the student 12B is displayed. In the example of Fig. 13, a first posture image 220A and a second posture image 220B obtained by photographing the teacher 12A and the student 12B from the left side are displayed side by side on the left side, and a first posture image 221A and a second posture image 221B obtained by photographing the teacher 12A and the student 12B from the right side are displayed side by side on the right side. According to this form, the student 12B can compare the posture of the teacher 12A with his / her own posture, and therefore can deepen his / her understanding of what operation to perform.

[0085] Specifically, the acquisition unit 380 may acquire a first posture image of the teacher 12A and a second posture image of the student 12B, which are obtained by photographing the teacher 12A and the student 12B from at least one same direction, respectively. The first posture image and the second posture image are photographed, for example, by a camera installed on the ceiling, wall, or the like of a room where an endoscopic examination is performed. The display control unit 386 may perform control so that the first posture image and the second posture image acquired by the acquisition unit 380 are displayed outside an area in which the field of view of the student 12B is displayed on the HMD 200B of the student 12B.

[0086] Also, for example, as shown on the left side of FIG. 14, the contents of the teacher 12A's speech may be displayed as text information 222A outside the area where the field of view of the student 12B is displayed. Specifically, the acquisition unit 380 may acquire the voice of the teacher 12A detected by the microphone 263A of the HMD 200A of the teacher 12A. The display control unit 386 may convert the voice of the teacher 12A acquired by the acquisition unit 380 into text information 222A and control the text information 222A to be displayed on the HMD 200B of the student 12B. As a method of converting voice into text information, a known method can be appropriately applied. In addition, the display control unit 386 may display the utterances of the teacher 12A so that they flow from bottom to top in order, so that the latest utterance contents can be understood. According to such a form, it is possible to prevent the student 12B from missing the contents of the teacher 12A's speech, and therefore it is possible to deepen the understanding of what operation to perform.

[0087] Also, for example, as shown in the upper right of Fig. 14, the first gaze information 212A and operation history of the teacher 12A may be displayed as text information 224A outside the area where the visual field of the student 12B is displayed. For example, the display control unit 386 may perform image analysis of the first visual field image 210A of the teacher 12A to identify the name of the thing in the line of sight indicated by the first gaze information 212A of the teacher 12A (for example, "insertion part") and control the name to be displayed as text information 224A on the HMD 200B of the student 12B. According to such a form, it is possible to prevent the student 12B from overlooking the gaze and operation history of the teacher 12A, and therefore it is possible to deepen the understanding of what operation should be performed.

[0088] Also, for example, as shown in the lower right part of Fig. 14, the parameter information 226A of the endoscope 16A of the teacher 12A may be displayed outside the area where the visual field of the student 12B is displayed. Specifically, the acquisition unit 380 may acquire the parameter information 226A of the endoscope 16A from the control device 22A of the teacher-side endoscope system 10A. The display control unit 386 may control the parameter information 226A acquired by the acquisition unit 380 to be displayed on the HMD 200B of the student 12B. According to such a form, the student 12B can easily grasp what settings and operations the teacher 12A is currently performing, and can deepen his / her understanding of what operations he / she should perform.

[0089] Also, instead of or in addition to the parameter information 226A of the endoscope 16A of the teacher 12A, the parameter information 226B of the endoscope 16B of the student 12B may be displayed. Specifically, the acquisition unit 380 may acquire the parameter information of the endoscope 16B from the control device 22B of the student side endoscope system 10B. The display control unit 386 may control the parameter information acquired by the acquisition unit 380 to be displayed on the HMD 200B of the student 12B.

[0090] Also, for example, as shown in Fig. 15, when a user focuses on a certain area in a visual field image, information related to the area may be controlled to be displayed on the HMD 200. Fig. 15 shows an example in which, when a student 12B focuses on a display device 18B in a second visual field image 210B, the screen 35 displayed on the display device 18 by the endoscope system 10 is enlarged and displayed on the HMD 200B.

[0091] Specifically, the acquisition unit 380 may acquire second gaze information of the student 12B detected by the gaze sensor 261B provided in the HMD 200B of the student 12B. The display control unit 386 may identify a region of interest of interest, which is a second region of interest that the student 12B is paying attention to in the second field of view image 210B, based on the second gaze information of the student 12B acquired by the acquisition unit 380. Furthermore, when the identified region of interest of interest is a region of interest of a predetermined type, the display control unit 386 may acquire related information related to the region of interest of interest. Then, the display control unit 386 may control the acquired related information to be superimposed on the region of interest of interest and displayed on the HMD 200B of the student 12B.

[0092] For example, the second line of sight information indicates that the student 12B is paying attention to the display device 18B. At this time, the display control unit 386 specifies that the display device 18B is the region of interest of interest among the display device 18B, the operation unit 46B, and the insertion unit 48B, which are the second region of interest included in the second field of view image 210B. When the region of interest of interest is the display device 18B, the display control unit 386 acquires information of the screen 35B (including the endoscopic moving image 39B and the medical information 44B) displayed on the display device 18B as related information from the control device 22B of the student-side endoscope system 10B. Then, the display control unit 386 controls the HMD 200B of the student 12B to display the acquired information of the screen 35B by superimposing it on the region of the display device 18B (region of interest of interest) of the second field of view image 210B. According to this embodiment, the student 12B can confirm the information of the screen 35B more clearly than by looking at the screen 35B displayed on the display device 18B.

[0093] For example, the second gaze information indicates that the student 12B is paying attention to the operation unit 46B. At this time, the display control unit 386 specifies that the operation unit 46B is the region of interest of interest. When the region of interest of interest is the operation unit 46B, the display control unit 386 acquires parameter information of the endoscope scope 16B as related information from the control device 22B of the student side endoscope system 10B. Then, the display control unit 386 controls the HMD 200B of the student 12B to display the acquired parameter information by superimposing it on the region of the operation unit 46B (region of interest of interest) in the second field of view image 210B.

[0094] Furthermore, for example, when the region of interest is the operation unit 46B, the display control unit 386 may acquire an image of the operation unit 46B captured by a camera other than the camera of the second field of view image 210B (camera 262B of the HMD 200B) as related information. For example, an image captured by a camera capturing an image of the operation unit 46B from the side may be acquired. Then, the display control unit 386 may control the second field of view image 210B to be displayed on the HMD 200B of the student 12B by superimposing the acquired image on the region of the operation unit 46B (region of interest).

[0095] 15, the display control unit 386 may enlarge and display the related information. For example, the display control unit 386 may enlarge or reduce the related information displayed on the HMD 200B, or stop displaying the information, in response to an instruction from the student 12B. An instruction from the student 12B may be received, for example, by the motion sensor 260B of the HMD 200B detecting that the student 12B has made a predetermined motion (for example, nodding twice). For example, an instruction area may be provided on the end of the HMD 200B, and the gaze sensor 261B may detect that the gaze is directed toward the instruction area for a predetermined period of time or more, and the instruction may be received. For example, the instruction may be received using an input means that does not affect the operation, such as a foot switch.

[0096] Furthermore, for example, when a phantom is used as the subject 26B, the display control unit 386 may replace the region of the subject 26B in the second field of view image 210B with an image of a human body and perform control to display it on the HMD 200B. For example, a person's body, face, eyes, etc. may be superimposed on the region of the subject 26B, and a painful expression or a distressed voice may be output in response to an operation by the student 12B.

[0097] Furthermore, for example, when a phantom is used as the subject 26B, the display control unit 386 may replace the region of the endoscopic moving image 39B in the second field of view image 210B with a large intestine CT (Computed Tomography) (colonography) image and perform control to display it on the HMD 200B. For example, the structure of the phantom and the colonography image may be associated in advance, and colonography images may be displayed sequentially according to the position and orientation of the tip 50B of the endoscope 16B at the time of insertion. The position and orientation of the tip 50B may be detected by, for example, a magnetic sensor or the like.

[0098] Furthermore, for example, when a phantom is used as the subject 26B, the display control unit 386 may perform control to replace the region of the endoscopic moving image 39B in the second field of view image 210B with a simulation image including a treatment target such as a polyp, and display it on the HMD 200B. Furthermore, when various treatments such as enlargement (air supply) of the intestine, processing of remaining matter (water supply and air suction), and polypectomy are performed using the treatment tool 58B while the simulation image is displayed, simulation images showing changes inside the body may be displayed sequentially.

[0099] A plurality of pieces of additional information may be displayed simultaneously on the HMD 200B, or a part of the additional information selected by the user may be displayed on the HMD 200B. For example, the additional information displayed on the HMD 200B may be switched at any timing by the user.

[0100] Next, the operation of the information processing device 300 according to this embodiment will be described with reference to Fig. 16. In the information processing device 300, the CPU 351 executes the training program 357, thereby executing the information processing shown in Fig. 16. The information processing is executed, for example, when a user issues an instruction to start execution via the operation unit 355.

[0101] In step S10, the acquisition unit 380 acquires a first field of view image 210A showing the field of view of the first user (teacher 12A) and a second field of view image 210B showing the field of view of the second user (student 12B). In step S12, the display control unit 386 controls the second display device (HMD 200B) viewed by the second user (student 12B) to display first interest information indicating the position of at least one first area of ​​interest included in the first field of view image 210A acquired in step S10. In step S14, the display control unit 386 controls the first display device (HMD 200A) viewed by the first user (teacher 12A) to display second interest information indicating the position of at least one second area of ​​interest included in the second field of view image 210B acquired in step S10, and ends this information processing.

[0102] As described above, the training system 100 according to one embodiment of the present disclosure includes at least one processor, and the processor acquires a first field of view image 210A showing the field of view of a first user (teacher 12A) and a second field of view image 210B showing the field of view of a second user (student 12B). The processor also controls the second display device (HMD 200B) viewed by the second user (student 12B) to display first interest information showing the position of at least one first area of ​​interest included in the first field of view image 210A. The processor also controls the first display device (HMD 200A) viewed by the first user (teacher 12A) to display second interest information showing the position of at least one second area of ​​interest included in the second field of view image 210B.

[0103] That is, according to the training system 100 of the present disclosure, when a teacher and a student perform similar movements in parallel, the student can imitate the teacher's movements and the teacher can give advice to the student's movements, thereby enabling appropriate training.

[0104] In addition, the first information of interest and the second information of interest are not limited to the information indicating the outline of the first region of interest and the information indicating the outline of the second region of interest described above. Modified examples of the first information of interest and the second information of interest will be described below.

[0105] [Variation 1] The display control unit 386 may control the HMD 200B worn by the student 12B to display a semi-transparent image of the first field of view image 210A as the first interest information. By displaying the semi-transparent first field of view image 210A on the HMD 200B, the student 12B can grasp where the first area of ​​interest is located in the field of view of the teacher 12A. Similarly, the display control unit 386 may control the HMD 200A worn by the teacher 12A to display a semi-transparent image of the second field of view image 210B as the second interest information. Note that the "semi-transparent image" refers to an image that is in a state between transparent (i.e., 100% transparency) and opaque (i.e., 0% transparency), and is not limited to an image with a transparency of 50%.

[0106] [Variation 2-1] The display control unit 386 may control to display each other's interest information only when the difference between the visual field of the teacher 12A and the visual field of the student 12B is relatively large. Specifically, the determination unit 384 may associate the first and second regions of interest of the same type, and determine whether the positions of the associated first and second regions of interest are different. The first and second regions of interest of the same type are, for example, the display device 18A and the display device 18B, the operation unit 46A and the operation unit 46B, the insertion unit 48A and the insertion unit 48B, the subject 26A and the subject 26B, etc.

[0107] When the determination by the determination unit 384 is positive (when the deviation is relatively large), the display control unit 386 may control the HMD 200B worn by the student 12B to display, as the first interest information, information indicating the difference between the position of the associated first region of interest and the position of the second region of interest. For example, as shown in FIG. 17, the display control unit 386 may display the difference between the position of the display device 18A (first region of interest) and the position of the display device 18B (second region of interest) with an arrow. Similarly, when the determination by the determination unit 384 is positive, the display control unit 386 may control the HMD 200A worn by the teacher 12A to display, as the second interest information, information indicating the difference between the position of the associated first region of interest and the position of the second region of interest.

[0108] Furthermore, when the determination by the determination unit 384 is negative (when the deviation is relatively small), the display control unit 386 may control the display of information indicating the difference between the positions of the associated first and second regions of interest as the first information of interest on the HMD 200B worn by the student 12B in a display form different from that when the determination by the determination unit 384 is positive (when the deviation is relatively large). Specifically, it is preferable that the information indicating the difference is displayed more conspicuously as the deviation is larger. For example, when the determination by the determination unit 384 is negative, the arrow as shown in FIG. 17 may not be displayed or may be lighter in color.

[0109] [Variation 2-2] It may be difficult to align the visual fields even after adjustments due to differences in physique and environment between teacher 12A and student 12B. Therefore, before the start of training, the visual fields of teacher 12A and student 12B may be aligned, and any deviation that still occurs may be corrected so that it is not considered as deviation when the determination unit 384 makes a determination.

[0110] Specifically, first, the student 12B checks the first interest information of the teacher 12A displayed on the HMD 200B worn by the student 12B, and performs alignment so that the field of view is as similar as possible to that of the teacher 12A. The time when this alignment is completed is set as the initial time. Whether or not the alignment is completed may be determined by the determination unit 384 based on the deviation between the first field of view image 210A and the second field of view image 210B, for example, or the user may input that the alignment is completed.

[0111] The acquisition unit 380 acquires the first field of view image 210A and the second field of view image 210B at the initial time. The extraction unit 382 extracts a first region of interest from the first field of view image 210A at the initial time, and extracts a second region of interest from the second field of view image 210B at the initial time. The determination unit 384 derives a difference between the position of the first region of interest and the position of the second region of interest, which are extracted and associated with each other from the first field of view image 210A and the second field of view image 210B at the initial time, as a correction value. For example, the determination unit 384 derives that, at the initial time, the display device 18B is shifted 5 cm to the left relative to the display device 18A, and the operation unit 46B is shifted 5 cm to the right relative to the operation unit 46A.

[0112] Thereafter, while the teacher 12A and the student 12B are each performing their own operations, the acquisition unit 380 acquires the first field of view image 210A and the second field of view image 210B over time. The extraction unit 382 extracts a first region of interest from each of the first field of view images 210A acquired over time, and extracts a second region of interest from each of the second field of view images 210B acquired over time.

[0113] The determination unit 384 derives a difference between the positions of the first and second regions of interest, which are extracted from the first field of view image 210A and the second field of view image 210B at a time point after the initial time point and are associated with each other, by correcting the difference using a correction value. For example, if the display device 18B is shifted 5 cm to the left relative to the display device 18A at a certain time point after the start of training, the difference is the same as the shift at the initial time point, so the determination unit 384 derives the difference as 0 cm. On the other hand, if the operation unit 46B is shifted 10 cm to the right relative to the operation unit 46A, the difference is greater than the shift at the initial time point, so the determination unit 384 derives the difference as 5 cm to the right. Then, the determination unit 384 determines whether the positions of the first and second regions of interest are different based on the difference corrected using the correction value derived based on the shift at the initial time point.

[0114] [Variation 3] In the above description, the second information of interest is displayed so as to be superimposed on the first field of view image 210A, and the first information of interest is displayed so as to be superimposed on the second field of view image 210B, but the present invention is not limited to this. For example, as shown in Fig. 18, the other person's field of view image may be split and displayed on the left and right of the own field of view. In Fig. 18, the first field of view image 210A of the teacher 12A is split and displayed on the left and right of the second field of view image 210B of the student 12B.

[0115] Specifically, the display control unit 386 may control the display of the left image 210AL of the first visual field image 210A divided into two images, left and right, on the left side of the area in the HMD 200B where the visual field of the student 12B is displayed, as the first information of interest. The display control unit 386 may also control the display of the right image 210AR of the first visual field image 210A divided into two images, left and right, on the right side of the area in the HMD 200B where the visual field of the student 12B is displayed, as the first information of interest.

[0116] Similarly, the display control unit 386 may control the second interest information to display, as one of the images obtained by dividing the second visual field image 210B into two, a left image, on the left side of the area in which the visual field of the teacher 12A is displayed in the HMD 200A. The display control unit 386 may also control the second interest information to display, as one of the images obtained by dividing the second visual field image 210B into two, a right image, on the right side of the area in which the visual field of the teacher 12A is displayed in the HMD 200A.

[0117] Here, the area in the HMD 200 where the visual field of the student 12B (teacher 12A) is displayed is an area where the user visually recognizes the real space through a lens (display) if the HMD 200 is a transmissive type, or an area where an image of the real space (visual field image) captured by the camera 262 is displayed if the HMD 200 is a non-transmissive type.

[0118] In the above embodiment, the form of the second interest information displayed on the HMD 200A of the teacher 12A and the form of the first interest information displayed on the HMD 200B of the student 12B may not be the same. For example, the HMD 200A of the teacher 12A may be displayed in a form as shown in Fig. 9, and the HMD 200B of the student 12B may be displayed in a form as shown in Fig. 18. The form to be displayed may be determined in advance, for example, or may be arbitrarily selected by the user.

[0119] In the above embodiment, the first field of view image 210A and the second field of view image 210B are captured by the camera 262 of the HMD 200, but the present invention is not limited to this. For example, the first field of view image 210A and / or the second field of view image 210B may be acquired using a camera installed at any location such as the wall and ceiling of the room where the user does training, and a camera attached to any part of the user such as the head, shoulders, and chest. For example, a wider range of field of view image may be generated by combining these external cameras with the camera 262 of the HMD 200. For example, the image to be displayed on the display unit 254 of the HMD 200 may be captured using the camera of the HMD 200, and the image used to extract the first and second regions of interest may be captured using an external camera.

[0120] In the above embodiment, the training is performed by one teacher 12A and one student 12B, but the present invention is not limited to this. For example, as shown in Fig. 19, the training system 100 of the present disclosure can be applied to a case where a single teacher 12A and a plurality of students 12B and 12C train together.

[0121] That is, the display control unit 386 may control the first interest information included in the first field of view image 210A of the teacher 12A to be displayed on the HMD 200C worn by a student 12C other than the student 12B. In this case, the HMD 200A of the teacher 12A may display second interest information indicating the position of the second area of ​​interest included in the second field of view image 210B of the student 12B, or third interest information indicating the position of the third area of ​​interest included in the third field of view image of the student 12C. In addition, both may be displayed simultaneously, or may be displayed switchably according to an instruction from the teacher 12A. The student 12C is an example of a third user of the present disclosure, and the HMD 200C is an example of a third display device viewed by the third user of the present disclosure.

[0122] 1, in the endoscope system 10, operations may be performed with the assistance of a nurse 14 or the like. For example, when removing a polyp, the doctor 12 operates the operation unit 46 and the insertion unit 48 to guide the tip 50 of the endoscope scope 16 to the polyp, and the nurse pushes in the treatment tool 58 at the doctor's command.

[0123] As shown in Fig. 20, the training system 100 of the present disclosure is also capable of training a teacher pair and a student pair, and is suitable for the above-mentioned collaborative work. In Fig. 20, a pair of a teacher 12A (doctor) and a sub-teacher 14A (nurse) and a pair of a student 12B (doctor) and a sub-student 14B (nurse) train together. In other words, when a doctor and a nurse perform similar actions in parallel, the student can imitate the teacher's actions and the teacher can give advice on the student's actions.

[0124] In this case, the acquisition unit 380 may acquire a first sub-field of view image showing the field of view of the sub-teacher 14A and a second sub-field of view image showing the field of view of the sub-student 14B. The display control unit 386 may control the HMD 200BS worn by the sub-student 14B to display information indicating the position of the first sub-region of interest included in the first sub-field of view image of the sub-teacher 14A. The display control unit 386 may also control the HMD 200AS worn by the sub-teacher 14A to display information indicating the position of the second sub-region of interest included in the second sub-field of view image of the sub-student 14B. The sub-teacher 14A is an example of a first sub-user of the present disclosure, and the sub-student 14B is an example of a second sub-user of the present disclosure. The HMD 200AS is an example of a first sub-display device seen by the first sub-user of the present disclosure, and the HMD 200BS is an example of a second sub-display device seen by the second sub-user of the present disclosure.

[0125] In this case, in addition to the view sharing between the teacher and the student, the view sharing between the pair may be performed. That is, for example, the HMD 200B of the student 12B (doctor) may be controlled to display information indicating the positions of the first interest area of ​​the teacher 12A (doctor) and the second sub-interest area of ​​the sub-student 14B (nurse) of the pair. Specifically, the acquisition unit 380 may acquire a second sub-view image indicating the view of the sub-student 14B. The display control unit 386 may control to display the first interest information of the teacher 12A (doctor) and information indicating the position of the second sub-interest area included in the second sub-view image of the sub-student 14B (nurse) on the HMD 200B of the student 12B (doctor).

[0126] In the above embodiment, the first display device viewed by the first user, the second display device viewed by the second user, the third display device viewed by the third user, the first sub-display device viewed by the first sub-user, and the second sub-display device viewed by the second sub-user are described as an example of the application of the HMD 200, but the present invention is not limited thereto. As the display device viewed by each user in the present disclosure, for example, various portable displays such as a glasses-type wearable device (so-called smart glasses) and a contact lens-type wearable device (so-called smart contact lens) may be applied. For example, a stationary display disposed near each user may be applied. For example, a projector that projects an image on a screen suspended from the ceiling of the room where each user is staying or on the wall of the room may be applied. For example, AR glasses suspended from the ceiling of the room where each user is staying may be applied, and each user may look into the AR glasses. For example, an aerial display that forms an image in the air may be applied. In addition, the type of display device viewed by each user may be different; for example, a stationary display may be used as the first display device viewed by a first user, and an HMD 200 may be used as the second display device viewed by a second user.

[0127] In the above embodiment, some or all of the functions of the information processing device 300, such as the acquisition unit 380, the extraction unit 382, ​​the determination unit 384, and the display control unit 386, may be provided in the control device 22 of the endoscope system 10, etc.

[0128] In the above embodiment, the following various processors can be used as the hardware structure of the processing unit that executes various processes, such as the acquisition unit 380, the extraction unit 382, ​​the determination unit 384, and the display control unit 386. As described above, the above various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processes, such as an ASIC (Application Specific Integrated Circuit), etc.

[0129] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0130] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as typified by System on Chip (SoC), etc. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0131] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0132] In the above embodiment, the various programs in the information processing device 300 and the HMD 200 are stored in advance in the storage unit, but the present invention is not limited to this. The various programs may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The various programs may also be downloaded from an external device via a network. Furthermore, the technology disclosed herein extends to a storage medium that non-temporarily stores a program in addition to the program.

[0133] The technology of the present disclosure can also be appropriately combined with the above-mentioned embodiment examples and examples. The above-mentioned description and illustrated contents are detailed descriptions of the parts related to the technology of the present disclosure, and are merely one example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the description and illustrated contents shown above, within the scope of the gist of the technology of the present disclosure.

[0134] The following supplementary notes are further disclosed regarding the above embodiment. [Appendix 1] at least one processor; The processor, Acquire a first field of view image showing a field of view of a first user and a second field of view image showing a field of view of a second user; Controlling a second display device viewed by the second user to display first information of interest indicating a position of at least one first region of interest included in the first field of view image; and controlling a first display device viewed by the first user to display second interest information indicating a position of at least one second region of interest included in the second field of view image. Information processing system. [Appendix 2] The processor, and controlling the second display device to display a semi-transparent image of the first visual field image as the first interest information. 2. An information processing system according to claim 1. [Appendix 3] The processor, Extracting at least one first region of interest from the first field of view image; and controlling the second display device to display, as the first information of interest, information indicating a contour of at least one of the extracted first regions of interest. 3. An information processing system according to claim 1 or 2. [Appendix 4] The processor, Extracting at least one first region of interest from the first field of view image; Extracting at least one second region of interest from the second field of view image; Corresponding the first region of interest and the second region of interest of the same type; determining whether a position of the first region of interest and a position of the second region of interest that are associated with each other are different; When the determination is affirmative, control is performed so as to cause the second display device to display, as the first information of interest, information indicating a difference between a position of the associated first region of interest and a position of the associated second region of interest. 4. An information processing system according to any one of claims 1 to 3. [Appendix 5] The processor, When the determination is negative, control is performed so that information indicating a difference between a position of the associated first region of interest and a position of the associated second region of interest is displayed, as the first information of interest, on the second display device in a display form different from that when the determination is positive. 5. An information processing system according to claim 4. [Appendix 6] The processor, The first field of view image and the second field of view image are acquired over time; Derive a difference between the position of the first region of interest and the position of the second region of interest, which are extracted and associated with each other from the first field of view image and the second field of view image at an initial time point, as a correction value; The determination is performed based on a difference between the position of the first region of interest and the position of the second region of interest that are extracted from the first field of view image and the second field of view image at a time point later than the initial time point and that are associated with each other and corrected using the correction value. 6. The information processing system according to claim 4 or 5. [Appendix 7] The processor, Controlling the second display device to display, as the first interest information, a left image of an image obtained by dividing the first visual field image into two parts, left and right, on the left side of an area in which the visual field of the second user is displayed on the second display device; As the first interest information, a right image of the first visual field image divided into two parts, left and right, is controlled to be displayed on the right side of an area in which the visual field of the second user is displayed on the second display device. 7. An information processing system according to any one of claims 1 to 6. [Appendix 8] The processor, Acquiring the first field of view image over time; determining whether or not there is a change in the position or orientation of the first region of interest from the first field of view images acquired over time; When the determination is affirmative, control is performed so as to cause the second display device to display information indicating that a change has occurred in the position or orientation of the first region of interest. 8. An information processing system according to any one of claims 1 to 7. [Appendix 9] The processor, Acquire first gaze information of the first user; and controlling the second display device to display first line-of-sight information of the first user on the second display device. 9. An information processing system according to any one of claims 1 to 8. [Appendix 10] The processor, Acquire second gaze information of the second user; Identifying a region of interest in the second field of view image based on the second gaze information, the region of interest being the second region of interest that the second user is paying attention to; When the region of interest is a region of interest of a predetermined type, acquiring related information related to the region of interest; and controlling the second display device to display the related information on the second display device by superimposing the related information on the region of interest. 10. An information processing system according to any one of claims 1 to 9. [Appendix 11] The processor, acquiring a first posture image and a second posture image obtained by photographing the first user and the second user from at least one same direction, respectively; Controlling the second display device to display the first attitude image and the second attitude image outside an area in which the field of view of the second user is displayed. 11. An information processing system according to any one of claims 1 to 10. [Appendix 12] The processor, Acquire the voice of the first user; Converting the first user's speech into text information; and controlling the second display device to display the text information. 12. An information processing system according to any one of claims 1 to 11. [Appendix 13] The processor, generating a second composite image by combining information indicating the position of the first region of interest with the second field of view image; generating a first composite image by combining information indicating the position of the second region of interest with the first field of view image; Controlling the second display device to display the second composite image; Controlling the first composite image to be displayed on the first display device. 13. An information processing system according to any one of claims 1 to 12. [Appendix 14] The first field of view image and the second field of view image show respective fields of view when the first user and the second user operate the same type of tool, The first region of interest and the second region of interest are at least one of a body part of the first user or the second user who operates the tool, at least a part of the tool, an object to be used with the tool, and an edge of the first field of view image or the second field of view image, respectively. 14. An information processing system according to any one of claims 1 to 13. [Appendix 15] The processor, obtaining parameter information relating to the instrument; and controlling the second display device to display the parameter information. 15. An information processing system according to claim 14. [Appendix 16] The processor, and controlling the first interest information to be displayed on a third display device for viewing by a third user. 16. An information processing system according to any one of claims 1 to 15. [Appendix 17] The processor, Obtaining a first sub-field of view image showing a field of view of a first sub-user and a second sub-field of view image showing a field of view of a second sub-user; Controlling a second sub-display device viewed by the second sub-user to display information indicating a position of a first sub-region of interest included in the first sub-field of view image; and controlling the display device to display information indicating the position of the second sub-region of interest included in the second sub-field of view image on a first sub-display device viewed by the first sub-user. 17. An information processing system according to any one of claims 1 to 16. [Appendix 18] The processor, obtaining a second sub-field of view image indicative of a field of view of a second sub-user; and controlling the second display device to display the first information of interest and information indicating a position of a second sub-region of interest included in the second sub-field of view image. 18. An information processing system according to any one of claims 1 to 17. [Appendix 19] Acquire a first field of view image showing a field of view of a first user and a second field of view image showing a field of view of a second user; Controlling a second display device viewed by the second user to display first information of interest indicating a position of at least one first region of interest included in the first field of view image; and controlling a first display device viewed by the first user to display second interest information indicating a position of at least one second region of interest included in the second field of view image. An information processing method in which processing is performed by a computer. [Appendix 20] Acquire a first field of view image showing a field of view of a first user and a second field of view image showing a field of view of a second user; Controlling a second display device viewed by the second user to display first information of interest indicating a position of at least one first region of interest included in the first field of view image; and controlling a first display device viewed by the first user to display second interest information indicating a position of at least one second region of interest included in the second field of view image. An information processing program that enables a computer to execute processing. [Explanation of symbols]

[0135] 10 Endoscope System 10A Teacher's endoscope system 10B, 10C Student side endoscope system 12. Doctor 12A Teacher 12B, 12C students 14. Nurse 14A Sub-Teacher 14B Sub-student 16 Endoscope 18, 18A, 18B display device 18Ar, 18Br, 46Ar, 46Br, 48Ar, 48Br, 210Ar Contour 20 Light source device 22, 22A, 22B, 22C Control device 24 Image Processing Device 26, 26A, 26B Subject 28 Large intestine 30 light 32 Intestinal wall 34 Wagon 35, 35B screen 36 1st display area 38 Second display area 39 Endoscopic video images 40 frames 42 Lesions 44 Medical Information 46, 46A, 46B operation section 48, 48A, 48B Insertion section 50 Tip 50A Tip surface 52 Camera 54 Lighting Equipment 54A, 54B Lighting window 56 Treatment opening 58 Treatment tools 60 Treatment tool insertion port 62 Universal Code 64 Reception device 100 Training System 200, 200A, 200B, 200C, 200AS, 200BS HMD 210A First Field of View Image 210AL (left side of first field image) 210AR (right side of first field image) 210B Second field of view image 212A First line of sight information 214A Rotation Information 220A, 221A 1st position image 220B, 221B 2nd posture image 222A, 224A Text information 226A Parameter Information 251, 351 CPU 252, 352 storage section 253, 353 memory 254 Display section 255, 355 Operation section 256, 356 I / F section 257 Control Program 258, 358 Bus 260 Motion Sensor 261 Eye Movement Sensor 262 Camera 263 Mike 300 Information processing device 354 Display 357 Training Program 380 Acquisition Department 382 Extraction part 384 Judgment section 386 Display control section

Claims

1. At least one processor; The processor, Acquire a first field of view image showing a field of view of a first user and a second field of view image showing a field of view of a second user; Controlling a second display device viewed by the second user to display first interest information indicating a position of at least one first region of interest included in the first field of view image; and controlling a first display device viewed by the first user to display second interest information indicative of a position of at least one second region of interest included in the second field of view image. Information processing system.

2. The processor, and controlling the second display device to display a semi-transparent image of the first visual field image as the first interest information. The information processing system according to claim 1 .

3. The processor, Extracting at least one first region of interest from the first field of view image; and controlling the second display device to display, as the first information of interest, information indicating a contour of the extracted at least one first region of interest. The information processing system according to claim 1 .

4. The processor, Extracting at least one first region of interest from the first field of view image; Extracting at least one second region of interest from the second field of view image; Corresponding the first region of interest and the second region of interest of the same type; determining whether a position of the first region of interest and a position of the second region of interest that are associated with each other are different; When the determination is affirmative, control is performed so that information indicating a difference between a position of the associated first region of interest and a position of the associated second region of interest is displayed on the second display device as the first information of interest. The information processing system according to claim 1 .

5. The processor, When the determination is negative, control is performed so that information indicating a difference between a position of the associated first region of interest and a position of the associated second region of interest is displayed, as the first information of interest, on the second display device in a display form different from that when the determination is positive.

5. The information processing system according to claim 4.

6. The processor, The first field of view image and the second field of view image are acquired over time; Derive a difference between the position of the first region of interest and the position of the second region of interest that are extracted and associated with each other from the first field of view image and the second field of view image at an initial time point as a correction value; The determination is performed based on a difference between the position of the first region of interest and the position of the second region of interest that are extracted from the first field of view image and the second field of view image at a time point later than the initial time point and that are associated with each other and corrected using the correction value.

5. The information processing system according to claim 4.

7. The processor, Controlling the second display device to display, as the first interest information, a left image of an image obtained by dividing the first visual field image into two parts, left and right, on the left side of an area in which the visual field of the second user is displayed on the second display device; As the first interest information, a right image of an image obtained by dividing the first visual field image into two parts, left and right, is controlled to be displayed on the right side of an area in which the visual field of the second user is displayed on the second display device. The information processing system according to claim 1 .

8. The processor, Acquiring the first field of view image over time; determining whether or not there is a change in the position or orientation of the first region of interest from the first field of view images acquired over time; When the determination is affirmative, control is performed so as to cause the second display device to display information indicating that a change has occurred in a position or orientation of the first region of interest. The information processing system according to claim 1 .

9. The processor, Acquire first line of sight information of the first user; and controlling the second display device to display first line-of-sight information of the first user. The information processing system according to claim 1 .

10. The processor, Acquire second gaze information of the second user; Identifying a region of interest in the second field of view image, which is the second region of interest that the second user is paying attention to, based on the second gaze information; When the region of interest is a region of interest of a predetermined type, acquiring related information related to the region of interest; and controlling the second display device to display the related information on the second display device by superimposing the related information on the region of interest. The information processing system according to claim 1 .

11. The processor, acquiring a first posture image and a second posture image obtained by photographing the first user and the second user from at least one same direction, respectively; and controlling the second display device to display the first attitude image and the second attitude image outside an area in which the second user's field of view is displayed. The information processing system according to claim 1 .

12. The processor, Acquire a voice of the first user; Converting the first user's speech into text information; and controlling the second display device to display the text information. The information processing system according to claim 1 .

13. The processor, generating a second composite image by combining information indicating the position of the first region of interest with the second field of view image; generating a first composite image by combining information indicating the position of the second region of interest with the first field of view image; Controlling the second display device to display the second composite image; Controlling the first composite image to be displayed on the first display device. The information processing system according to claim 1 .

14. The first field of view image and the second field of view image indicate respective fields of view when the first user and the second user operate the same type of tool, The first region of interest and the second region of interest are at least one of a body part of the first user or the second user who operates the tool, at least a part of the tool, an object to be used with the tool, and an edge of the first field of view image or the second field of view image, respectively. The information processing system according to claim 1 .

15. The processor, obtaining parameter information relating to the instrument; and controlling the second display device to display the parameter information.

15. The information processing system according to claim 14.

16. The processor, and controlling the first interest information to be displayed on a third display device for viewing by a third user. The information processing system according to claim 1 .

17. The processor, Obtaining a first sub-field of view image showing a field of view of a first sub-user and a second sub-field of view image showing a field of view of a second sub-user; Controlling a second sub-display device viewed by the second sub-user to display information indicating a position of a first sub-region of interest included in the first sub-field of view image; and controlling a first sub-display device to display information indicating a position of a second sub-region of interest included in the second sub-field of view image on the first sub-display device viewed by the first sub-user. The information processing system according to claim 1 .

18. The processor, obtaining a second sub-field of view image indicative of a field of view of a second sub-user; and controlling the second display device to display the first information of interest and information indicating a position of a second sub-region of interest included in the second sub-field of view image. The information processing system according to claim 1 .

19. Acquire a first field of view image showing a field of view of a first user and a second field of view image showing a field of view of a second user; Controlling a second display device viewed by the second user to display first interest information indicating a position of at least one first region of interest included in the first field of view image; and controlling a first display device viewed by the first user to display second interest information indicative of a position of at least one second region of interest included in the second field of view image. An information processing method in which processing is performed by a computer.

20. Acquire a first field of view image showing a field of view of a first user and a second field of view image showing a field of view of a second user; Controlling a second display device viewed by the second user to display first interest information indicating a position of at least one first region of interest included in the first field of view image; and controlling a first display device viewed by the first user to display second interest information indicative of a position of at least one second region of interest included in the second field of view image. An information processing program that enables a computer to execute processing.

Images