Information processing system, information processing method, and information processing program

Abstract

To provide an information processing system, an information processing method, and an information processing program that improve the visibility of at least part of a visual field.SOLUTION: An information processing system includes at least one processor. The processor acquires a visual field image showing a visual field of a user; acquires, in a case where the visual field image includes a target region of interest that is a predetermined type of region of interest, relevant information related to the target region of interest; and performs control of causing a first display device viewed by the user to display the relevant information to be superimposed on the target region of interest.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there are known display devices such as Augmented Reality (AR) devices that allow a user to visually recognize a real space and a display image. For example, Patent Document 1 discloses a method of acquiring an imaged display image and an image for decoding by capturing an image of a subject, acquiring a light ID by decoding the image for decoding, acquiring an AR image and recognition information corresponding to the light ID from a server, and displaying an imaged display image in which the AR image is superimposed in a region of the imaged display image corresponding to the recognition information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 104666 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been a demand for technology that can improve the visibility of at least a part of a visual field image viewed by a user through a display device such as a head-mounted display, smart glasses, etc. For example, in an endoscopic examination, a doctor operates an endoscope scope while viewing an endoscopic video until the scope reaches a desired observation target site. However, depending on the doctor's standing position, etc., the doctor may be far from the display device on which the endoscopic video is displayed, making it difficult to view the endoscopic video.

[0005] The present disclosure provides an information processing system, an information processing method, and an information processing program capable of improving visibility in at least a part of a field of view. [Means for solving the problem]

[0006] A first aspect of the present disclosure is an information processing system comprising at least one processor, which acquires a field of view image showing a user's field of view, and when the field of view image includes a target region of interest that is a predetermined type of region of interest, acquires associated information related to the target region of interest, and controls the associated information to be displayed on a first display device viewed by the user so as to be superimposed on the target region of interest.

[0007] In the above first aspect, the processor may acquire gaze information of the user, identify a region of interest at which the user is focusing in the field of view image based on the gaze information, and if the region of interest at which the user is focusing is a region of target interest, acquire related information related to the region of target interest, and control the first display device to display the related information so that it is superimposed on the region of target interest.

[0008] In the first aspect above, the processor may control the display of the related information in an area larger than the target region of interest in the field of view image.

[0009] In the first aspect, the processor may switch whether or not to display the related information on the first display device in response to an instruction from a user.

[0010] In the first aspect above, the processor may accept a predetermined action or speech of the user as an instruction from the user.

[0011] In the above first aspect, the target area of ​​interest may be an area of ​​a second display device different from the first display device, and the processor may obtain, as the relevant information, content displayed by the second display device.

[0012] In the first aspect, the second display device may be a device that displays an image captured by at least one of an endoscope, a microscope, and an operating field camera.

[0013] A second aspect of the present disclosure is an information processing method in which a computer executes a process of acquiring a field of view image showing a user's field of view, and if the field of view image includes a target region of interest that is a predetermined type of region of interest, acquiring associated information related to the target region of interest, and controlling the associated information to be displayed on a first display device viewed by the user so that it is superimposed on the target region of interest.

[0014] A third aspect of the present disclosure is an information processing program that causes a computer to execute a process of acquiring a field of view image showing a user's field of view, and when the field of view image includes a target region of interest that is a predetermined type of region of interest, acquiring associated information related to the target region of interest, and controlling the associated information to be displayed on a first display device viewed by the user so that it is superimposed on the target region of interest. Effect of the Invention

[0015] According to the above aspects, the information processing system, the information processing method, and the information processing program disclosed herein can improve the visibility of at least a part of the field of view. [Brief description of the drawings]

[0016] [Figure 1] 1 is a schematic diagram showing an example of an overall configuration of an endoscope system. [Diagram 2] 1 is a schematic diagram showing an example of an overall configuration of an endoscope system. [Diagram 3] 1 is a schematic diagram illustrating an example of an overall configuration of an information processing system. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of an HMD. [Diagram 5] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device. [Figure 6] FIG. 2 is a block diagram showing an example of a functional configuration of an information processing device. [Figure 7] FIG. 13 is a diagram showing an example of a visual field image. [Figure 8] FIG. 2 is a diagram showing an example of an image displayed on an HMD. [Figure 9]11 is a flowchart illustrating an example of information processing. [Figure 10] FIG. 13 is a diagram showing an example of a visual field image. [Figure 11] FIG. 2 is a diagram showing an example of an image displayed on an HMD. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0018] First, an endoscope system 10 to which the technology of the present disclosure can be applied will be described with reference to Fig. 1 and Fig. 2. The endoscope system 10 is used by a doctor 12 in an endoscopic examination or the like. The endoscopic examination is assisted by staff such as a nurse 14.

[0019] 1, the endoscope system 10 includes an endoscope scope 16, a display device 18, a light source device 20, a control device 22, and an image processing device 24. The light source device 20, the control device 22, and the image processing device 24 are installed on a wagon 34. The wagon 34 has a plurality of stands arranged vertically, and the image processing device 24, the control device 22, and the light source device 20 are installed from the lower stand to the upper stand. In addition, the display device 18 is installed on the top stand of the wagon 34.

[0020] The endoscope system 10 is a modality for performing medical treatment inside the body of a subject 26 (e.g., a patient) using an endoscope scope 16. The endoscope scope 16 is used by a doctor 12 and inserted into a body cavity (e.g., a hollow organ such as the large intestine, stomach, duodenum, and trachea) of the subject 26. The endoscope system 10 causes the endoscope scope 16 inserted into the body cavity of the subject 26 to capture images of the inside of the body cavity of the subject 26, and performs various medical procedures in the body cavity as necessary.

[0021] As an example, in this embodiment, the endoscope scope 16 is inserted into the large intestine 28 of the subject 26. The endoscope system 10 captures an image of the inside of the large intestine 28 of the subject 26, thereby acquiring and outputting an image showing the state inside the large intestine 28. In this embodiment, the endoscope system 10 has an optical imaging function of capturing an image of reflected light obtained by irradiating the inside of the large intestine 28 with light 30 and reflecting it off a wall 32 of the large intestine 28.

[0022] The control device 22 controls the entire endoscope system 10. The image processing device 24, under the control of the control device 22, performs various types of image processing on the image obtained by imaging the intestinal wall 32 by the endoscope scope 16.

[0023] The display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an organic EL (Electro-Luminescence) display. Moreover, a smartphone, a tablet terminal, and the like may be used instead of or in addition to the display device 18. The display device 18 is an example of a second display device of the present disclosure.

[0024] A screen 35 is displayed on the display device 18. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged side by side within the screen 35. In the example shown in FIG. 1, a first display area 36 and a second display area 38 are shown as examples of the plurality of display areas. The first display area 36 is used as a main display area, and the second display area 38 is used as a sub-display area.

[0025] An endoscopic moving image 39 is displayed in the first display area 36. The endoscopic moving image 39 is a moving image acquired by imaging an intestinal wall 32 in the large intestine 28 of the subject 26 with the endoscope scope 16. In the example shown in Fig. 1, a moving image showing the intestinal wall 32 is shown as an example of the endoscopic moving image 39.

[0026] 1, the intestinal wall 32 shown in the endoscopic video 39 includes a lesion 42 as an observation target region gazed upon by the doctor 12. The doctor 12 can visually recognize the state of the intestinal wall 32 including the lesion 42 through the endoscopic video 39. Examples of the lesion 42 include a neoplastic polyp and a non-neoplastic polyp.

[0027] The image displayed in the first display area 36 is one frame 40 included in a moving image that includes a plurality of frames 40 arranged in chronological order. In other words, the first display area 36 displays the plurality of frames 40 arranged in chronological order at a default frame rate (e.g., several tens of frames per second).

[0028] The second display area 38 displays medical information 44, which is information related to medical care. Examples of the medical information 44 include information that assists the medical judgment of the doctor 12. Examples of such information include various types of information related to the subject 26 and information obtained by performing image analysis on the endoscopic video image 39 using CAD (Computer Aided Diagnosis / Detection) technology.

[0029] 2, the endoscope 16 includes an operation section 46 and an insertion section 48. The insertion section 48 is partially curved by operating the operation section 46. The insertion section 48 is inserted into the large intestine 28 while curving in accordance with the shape of the large intestine 28 in accordance with the operation of the operation section 46 by the doctor 12.

[0030] The distal end 50 of the insertion section 48 is provided with a camera 52, an illumination device 54, and a treatment tool opening 56. The camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50. Note that, although an example in which the camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50 is given here, the present invention is not limited thereto, and the camera 52 and the illumination device 54 may be provided on the side surface of the distal end 50, so that the endoscope 16 is configured as a side-viewing mirror.

[0031] The camera 52 is inserted into the body cavity of the subject 26 to capture an image of an observation target region. In this embodiment, the camera 52 captures an image of the inside of the subject 26 (e.g., inside the large intestine 28) to obtain an endoscopic moving image 39. Examples of the camera 52 include a complementary metal oxide semiconductor (CMOS) camera and a charge coupled device (CCD) camera.

[0032] The illumination device 54 has illumination windows 54A and 54B. The illumination device 54 irradiates light 30 through the illumination windows 54A and 54B. Examples of the light 30 irradiated from the illumination device 54 include visible light (e.g., white light) and non-visible light (e.g., near-infrared light). The illumination device 54 also irradiates special light through the illumination windows 54A and 54B. Examples of the special light include light for Blue Light Imaging (BLI) and light for Linked Color Imaging (LCI). The camera 52 captures an image of the inside of the large intestine 28 by an optical method while the light 30 is irradiated inside the large intestine 28 by the illumination device 54.

[0033] The treatment tool opening 56 is an opening for allowing a treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 is also used as a suction port for sucking blood, internal waste, and the like, and as a delivery port for delivering fluid.

[0034] A treatment tool insertion port 60 is formed in the operation section 46, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60. The treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56. In the example shown in FIG. 2, a puncture needle is shown as the treatment tool 58 protruding from the treatment tool opening 56. Note that the treatment tool 58 is not limited to a puncture needle, and may be, for example, a grasping forceps, a papillotomy knife, a snare, a catheter, a guide wire, a cannula, a puncture needle with a guide sheath, or the like.

[0035] The endoscope 16 is connected to a light source device 20 and a control device 22 via a universal cord 62. An image processing device 24 and a reception device 64 are connected to the control device 22. In addition, a display device 18 is connected to the image processing device 24. That is, the control device 22 is connected to the display device 18 via the image processing device 24.

[0036] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and a remote control device.

[0037] The control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the image processing device 24.

[0038] The light source device 20 emits light under the control of the control device 22, and supplies the light to the illumination device 54. A light guide is built into the illumination device 54, and the light supplied from the light source device 20 passes through the light guide and is irradiated from illumination windows 54A and 54B. The control device 22 causes the camera 52 to capture an image, acquires an endoscopic moving image 39 from the camera 52, and outputs it to a predetermined output destination (for example, the image processing device 24).

[0039] The image processing device 24 supports the endoscopic examination by performing various image processing on the endoscopic moving image 39 input from the control device 22. The image processing device 24 outputs the endoscopic moving image 39 that has been subjected to various image processing to a predetermined output destination (for example, the display device 18 and the control device 22).

[0040] In this example, the image processing device 24 is shown as an external device for expanding the functions performed by the control device 22, and therefore an example is shown in which the control device 22 and the display device 18 are indirectly connected via the image processing device 24, but this is not limiting. For example, the display device 18 may be directly connected to the control device 22. In this case, for example, the function of the image processing device 24 may be included in the control device 22, or the control device 22 may be equipped with a function to cause a server (not shown) to execute the same process as that executed by the image processing device 24 and receive and use the results of the process by the server.

[0041] The endoscope system 10 is also communicably connected to an information processing device 300 and other external devices (not shown). Examples of the external devices include a server and / or a client terminal (such as a personal computer, a smartphone, or a tablet terminal) that manages various information such as an electronic medical record. The external device receives information transmitted from the endoscope system 10 and executes processing using the received information (such as processing to store the information in an electronic medical record, etc.).

[0042] In the endoscope system 10, the doctor 12 operates the operation unit 46 with his left hand and the insertion unit 48 with his right hand, while watching the display device 18, so that the tip 50 of the endoscope 16 reaches the desired observation area. In particular, when the large intestine 28 is the subject of the examination, there are large individual differences in the course of the large intestine 28, so it is necessary to flexibly change the operation depending on the state of the subject 26 (for example, whether the subject appears calm or in pain, etc.).

[0043] To perform such complicated operations, it is necessary to fit the display device 18, the operation unit 46 in the left hand, the insertion unit 48 in the right hand, and the subject 26 all within the field of view. However, in that case, the distance from the doctor to the display device 18 becomes long, which may reduce the visibility of the endoscopic moving image 39. Therefore, the information processing system 100 of the present disclosure improves the visibility of at least a part of the field of view on the display device viewed by the user.

[0044] An information processing system 100 of the present disclosure will be described with reference to Fig. 3. In this embodiment, the information processing system 100 includes an information processing device 300 and an endoscope system 10 used by a doctor 12. In the information processing system 100, the doctor 12 wears a head mounted display (HMD) 200 as an example of a first display device viewed by a user.

[0045] An example of a hardware configuration of the HMD 200 will be described with reference to Fig. 4. As shown in Fig. 4, the HMD 200 includes a CPU (Central Processing Unit) 251, a non-volatile storage unit 252, and a memory 253 as a temporary storage area. The HMD 200 also includes a display unit 254, an operation unit 255 such as a touch panel, and an I / F (Interface) unit 256. The HMD 200 also includes a motion sensor 260 that detects a motion of a user wearing the HMD 200, a gaze sensor 261 that detects a gaze, a camera 262, and a microphone 263.

[0046] The storage unit 252 is realized by, for example, a storage medium such as a flash memory. The storage unit 252 stores a control program 257 that performs overall control of the HMD 200. The CPU 251 reads the control program 257 from the storage unit 252, loads it into the memory 253, and executes the loaded control program 257.

[0047] The display unit 254 is configured to allow a user wearing the HMD 200 to view an image (virtual image). For example, the display unit 254 may use a transmissive or non-transmissive display, or may project an image onto the user's retina. The I / F unit 256 performs wired or wireless communication with the information processing device 300 and other external devices.

[0048] The motion sensor 260 is a sensor that detects the motion of the user wearing the HMD 200, and is, for example, an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. The gaze sensor 261 is a sensor that detects the gaze point of the user wearing the HMD 200. The camera 262 is a camera that captures the real space observed by the user wearing the HMD 200, and is, for example, a digital camera such as a CMOS camera. The microphone 263 collects the voice of the user wearing the HMD 200 and surrounding sounds.

[0049] The CPU 251, memory unit 252, memory 253, display unit 254, operation unit 255, I / F unit 256, motion sensor 260, gaze sensor 261, camera 262, and microphone 263 are connected via a bus 258 such as a system bus and a control bus so as to be able to exchange various information with each other.

[0050] The information processing device 300 is configured to be connectable to the control device 22 of the endoscope system 10, the HMD 200, and other external devices (not shown), respectively, via a wired or wireless network.

[0051] An example of a hardware configuration of the information processing device 300 will be described with reference to Fig. 5. As shown in Fig. 5, the information processing device 300 includes a CPU 351, a non-volatile storage unit 352, and a memory 353 as a temporary storage area. The information processing device 300 also includes a display 354 such as a liquid crystal display, an operation unit 355 such as a touch panel, a keyboard, and a mouse, and an I / F unit 356. The I / F unit 356 performs wired or wireless communication with the control device 22, the HMD 200, and other external devices.

[0052] The storage unit 352 is realized by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 352 stores an information processing program 357 for the information processing device 300. The CPU 351 reads the information processing program 357 from the storage unit 352, loads it into the memory 353, and executes the loaded information processing program 357.

[0053] The CPU 351, storage unit 352, memory 353, display 354, operation unit 355, and I / F unit 356 are connected to each other via a bus 358 such as a system bus or a control bus so as to be able to transmit and receive various information to each other. As the information processing device 300, for example, a personal computer, a server computer, a smartphone, a tablet terminal, a wearable terminal, or the like can be appropriately applied.

[0054] An example of a functional configuration of the information processing device 300 will be described with reference to Fig. 6. As shown in Fig. 6, the information processing device 300 includes an acquisition unit 380, an identification unit 382, ​​and a display control unit 384. The CPU 351 executes the information processing program 357, so that the CPU 351 functions as the acquisition unit 380, the identification unit 382, ​​and the display control unit 384.

[0055] The acquisition unit 380 acquires a visual field image 210 showing the visual field of the doctor 12 from the HMD 200 worn by the doctor 12. As an example, FIG. 7 shows a visual field image 210 showing the visual field of the doctor 12 operating the endoscope system 10. In general, exemplary states such as how to hold the endoscope scope 16, the standing position relative to the subject 26 and the display device 18, and the visual field (objects to be captured in the field of view and their positions) are determined in advance in an endoscopic examination. The visual field image 210 in FIG. 7 is an appropriate state in which the display device 18, the operation unit 46 held in the left hand, the insertion unit 48 held in the right hand, and the state of the subject 26 (insertion port) can all be viewed simultaneously. However, the distance from the doctor's viewpoint to the display device 18 becomes farther, and the visibility of the display device 18 is reduced. The doctor 12 is an example of a user of the present disclosure.

[0056] The identification unit 382 acquires related information related to a target region of interest, which is a region of interest of a predetermined type, when the field of view image 210 acquired by the acquisition unit 380 includes the target region of interest. In other words, the target region of interest is an area to which related information is associated in advance and can be acquired.

[0057] An example of such a target region of interest is a region of various display devices. When the region of the display device 18 is included in the visual field image 210, the identification unit 382 may acquire the contents displayed by the display device 18 as related information. Note that the display device 18 is not limited to the endoscope as shown in FIG. 1, and may be, for example, a device that displays an image captured by at least one of an endoscope, a microscope, and an operating field camera.

[0058] 7, since the field of view image 210 of the doctor 12 includes the area of ​​the display device 18, which is the target area of ​​interest, the identifying unit 382 acquires related information related to the display device 18. Specifically, the identifying unit 382 acquires, from the control device 22 of the endoscope system 10, information on the screen 35 (including the endoscopic video image 39 and medical information 44) that the endoscope system 10 is causing the display device 18 to display, as the related information.

[0059] The display control unit 384 controls the related information acquired by the identification unit 382 to be displayed on the HMD 200 worn by the doctor 12 so as to be superimposed on the target region of interest in the field of view image 210. As an example, FIG. 8 shows the screen 35 displayed on the display device 18 by the endoscope system 10 enlarged and then superimposed on the region of the display device 18 in the field of view image 210 (the target region of interest). In this way, the display control unit 384 may control the related information to be displayed in an area larger than the target region of interest in the field of view image 210. That is, the display control unit 384 may enlarge and display the related information.

[0060] Moreover, if the related information is superimposed only because it is included in the field of view, it may be bothersome. Therefore, the related information may be superimposed only when the doctor 12 is paying attention to the target region of interest. Specifically, the acquisition unit 380 may acquire the line of sight information of the doctor 12 detected by the line of sight sensor 261 provided in the HMD 200 of the doctor 12.

[0061] The identifying unit 382 may identify a region of interest in the visual field image 210 that is the region of interest on which the doctor 12 is focusing, based on the gaze information acquired by the acquiring unit 380. For example, it is assumed that the gaze information of the doctor 12 indicates that he is focusing on the display device 18. In this case, the identifying unit 382 identifies the display device 18 as the region of interest of interest among the regions of interest included in the visual field image 210 (the display device 18, the operation unit 46, the insertion unit 48, the subject 26, etc.).

[0062] Furthermore, when the identified region of interest is also a region of interest of a target (e.g., a display device, etc.), the identifying unit 382 may acquire related information related to the region of interest of the target. For example, when the region of interest of interest is the display device 18, it corresponds to the region of interest of a target. Therefore, the identifying unit 382 acquires information on the screen 35 displayed on the display device 18 from the control device 22 of the endoscope system 10 as related information.

[0063] Also, the doctor 12 may be allowed to check or stop checking the related information at any timing. Specifically, the display control unit 384 may switch whether or not to display the related information on the HMD 200 in response to an instruction from the doctor 12.

[0064] The display control unit 384 may accept a predetermined action or speech of the doctor 12 as an instruction from the doctor 12. For example, when the action sensor 260 of the HMD 200 detects that the doctor 12 has made a predetermined action (e.g., nodding twice), the display control unit 384 may determine that the instruction has been accepted. Alternatively, for example, an instruction area may be provided at an end of the HMD 200, and when the gaze sensor 261 detects that the gaze is directed toward the instruction area for a predetermined period of time or more, the display control unit 384 may determine that the instruction has been accepted. Alternatively, for example, the microphone 263 of the HMD 200 may be used to accept voice input instructions from the doctor 12.

[0065] Next, the operation of the information processing device 300 according to this embodiment will be described with reference to Fig. 9. In the information processing device 300, the CPU 351 executes the information processing program 357, thereby executing the information processing shown in Fig. 9. The information processing is executed, for example, when a user issues an instruction to start execution via the operation unit 355.

[0066] In step S10, the acquisition unit 380 acquires a visual field image 210 showing the visual field of the user (doctor 12). In step S12, the identification unit 382 determines whether or not the visual field image 210 acquired in step S10 includes a target region of interest that is a region of interest of a predetermined type.

[0067] If the target region of interest is included, the process proceeds to step S14, where the identification unit 382 acquires related information related to the target region of interest. In step S16, the display control unit 384 controls the first display device (HMD 200) viewed by the user (doctor 12) to display the related information acquired in step S14 so as to be superimposed on the target region of interest in the visual field image 210 acquired in step S10. If the target region of interest is not included in step S12, or when step S16 is completed, this information processing ends.

[0068] As described above, the information processing system 100 according to an embodiment of the present disclosure includes at least one processor, and the processor acquires a visual field image 210 indicating a visual field of a user (doctor 12). When a target region of interest that is a region of interest of a predetermined type is included in the visual field image 210, the processor acquires related information related to the target region of interest. The processor also controls the first display device (HMD 200) viewed by the user (doctor 12) to display the related information so as to be superimposed on the target region of interest.

[0069] That is, according to the information processing system 100 of the present disclosure, related information can be displayed for a target region of interest, thereby improving visibility of at least a portion of the field of view.

[0070] In the above embodiment, the target region of interest is the second display device (display device 18), and the related information is the display content of the second display device. However, the present invention is not limited to this. For example, the operation unit 46 of the endoscope 16 may be applied as the target region of interest. In this case, for example, parameter information of the endoscope 16 may be applied as the related information. Also, for example, an image of the operation unit 46 captured by a camera other than the camera capturing the field of view image 210 (camera 262 of the HMD 200) may be applied as the related information. For example, an image captured by a camera capturing the operation unit 46 from the side may be acquired.

[0071] In the above embodiment, the technique of the present disclosure is applied to the endoscope system 10, but the present disclosure is not limited to this. The technique of the present disclosure can be applied to the medical field and various other fields.

[0072] As an example, Figures 10 and 11 show an example in which the target area of ​​interest is a departure information board 18T at a railway passenger station and the related information is a train transfer information board 35T. As shown in Figure 10, when one wishes to know the departure time of a train on the station platform, visibility is poor if the distance from the departure information board 18T is large. In such a case, by applying the technology of the present disclosure, it becomes possible to visually confirm the departure time of the train without approaching the departure information board 18T.

[0073] Also, as shown in Fig. 11, a transfer guide may be displayed as related information. The destination may be a predetermined destination such as the nearest station to home, work, or school, or may be input by the user each time. The contents of the departure guide board 18T and the transfer guide information may be acquired from a website that provides transfer guides and timetables, etc., using an API (Application Programming Interface) with the departure point and / or destination as a search query.

[0074] In the above embodiment, the visual field image 210 is captured by the camera 262 of the HMD 200, but the present invention is not limited to this. For example, the visual field image 210 may be acquired using a camera installed at any location such as a wall or ceiling in the room where the user is present, and a camera attached to any part of the user such as the head, shoulders, or chest. For example, a wider range of visual field images may be generated by combining these external cameras with the camera 262 of the HMD 200. For example, an image to be displayed on the display unit 254 of the HMD 200 may be captured using the camera 262 of the HMD 200, and an image used to extract the target region of interest may be captured using an external camera.

[0075] In the above embodiment, the embodiment has been described in which the HMD 200 is applied as the first display device seen by the user, but the present disclosure is not limited thereto. As the first display device seen by the user in the present disclosure, for example, various portable displays such as a glasses-type wearable device (so-called smart glasses) and a contact lens-type wearable device (so-called smart contact lens) may be applied. For example, a stationary display disposed near each user may be applied. For example, a screen hung from the ceiling of the room where each user is staying or a projector that projects an image onto the wall of the room may be applied. For example, AR glasses hung from the ceiling of the room where each user is staying may be applied, and each user may look into the AR glasses. For example, an aerial display that forms an image in the air may be applied.

[0076] In the above embodiment, some or all of the functions of the information processing device 300, such as the acquisition unit 380, the identification unit 382, ​​and the display control unit 384, may be provided in the control device 22 of the endoscope system 10, etc.

[0077] In the above embodiment, the following various processors can be used as the hardware structure of the processing unit that executes various processes, such as the acquisition unit 380, the identification unit 382, ​​and the display control unit 384. As described above, the above various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processes, such as an ASIC (Application Specific Integrated Circuit), etc.

[0078] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0079] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as typified by System on Chip (SoC), etc. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0080] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0081] In the above embodiment, the various programs in the information processing device 300 and the HMD 200 are stored in advance in the storage unit, but the present invention is not limited to this. The various programs may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The various programs may also be downloaded from an external device via a network. Furthermore, the technology disclosed herein extends to a storage medium that non-temporarily stores a program in addition to the program.

[0082] The technology of the present disclosure can also be appropriately combined with the above-mentioned embodiment examples and examples. The above-mentioned description and illustrated contents are detailed descriptions of the parts related to the technology of the present disclosure, and are merely one example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the description and illustrated contents shown above, within the scope of the gist of the technology of the present disclosure.

[0083] The following supplementary notes are further disclosed regarding the above embodiment. [Appendix 1] At least one processor; The processor, obtaining a visual field image indicative of a user's visual field; When the field of view image includes a target region of interest that is a region of interest of a predetermined type, obtaining related information related to the target region of interest; and controlling the display of the related information on a first display device viewed by the user so as to be superimposed on the region of interest of interest. Information processing system. [Appendix 2] The processor, Acquire gaze information of the user; Identifying a region of interest in which the user is paying attention in the visual field image based on the line of sight information; If the region of interest is the region of interest of interest, obtain related information related to the region of interest of interest of interest; and controlling the first display device to display the related information so as to be superimposed on the region of interest of the subject. 2. An information processing system according to claim 1. [Appendix 3] The processor, and controlling the display of the related information in a region of the field of view image that is larger than the region of interest of the target. 3. An information processing system according to claim 1 or 2. [Appendix 4] The processor, Switching whether or not to display the related information on the display device in response to an instruction from the user 4. An information processing system according to any one of claims 1 to 3. [Appendix 5] The processor, Accepting a predetermined action or speech of the user as an instruction from the user 5. An information processing system according to claim 4. [Appendix 6] the region of interest of interest is a region of a second display device distinct from the first display device; The processor acquires, as the related information, the content displayed by the second display device. 6. An information processing system according to any one of claims 1 to 5. [Appendix 7] The second display device is a device that displays an image captured by at least one of an endoscope, a microscope, and an operating field camera. 7. An information processing system according to claim 6. [Appendix 8] obtaining a visual field image indicative of a user's visual field; When the field of view image includes a target region of interest that is a region of interest of a predetermined type, obtaining related information related to the target region of interest; and controlling the display of the related information on a first display device viewed by the user so as to be superimposed on the region of interest of interest. An information processing method in which processing is performed by a computer. [Appendix 9] obtaining a visual field image indicative of a user's visual field; When the field of view image includes a target region of interest that is a region of interest of a predetermined type, obtaining related information related to the target region of interest; and controlling the display of the related information on a first display device viewed by the user so as to be superimposed on the region of interest of interest. An information processing program that enables a computer to execute processing. [Explanation of symbols]

[0084] 10 Endoscope System 12. Doctor 14. Nurse 16 Endoscope 18 Display device 20 Light source device 22 Control device 24 Image Processing Device 26 Subject 28 Large intestine 30 light 32 Intestinal wall 34 Wagon 35 screens 36 1st display area 38 Second display area 39 Endoscopic video images 40 frames 42 Lesions 44 Medical Information 46 Control section 48 Insertion section 50 Tip 50A Tip surface 52 Camera 54 Lighting Equipment 54A, 54B Lighting window 56 Treatment opening 58 Treatment tools 60 Treatment tool insertion port 62 Universal Code 64 Reception device 100 Information Processing Systems 200 HMD 210 Field of view image 251, 351 CPUs 252, 352 storage section 253, 353 memory 254 Display section 255, 355 Operation section 256, 356 I / F section 257 Control Program 258, 358 Bus 260 Motion Sensor 261 Eye Movement Sensor 262 Camera 263 Mike 300 Information processing device 354 Display 357 Information Processing Program 380 Acquisition Department 382 Specific part 384 Display control section

Claims

1. At least one processor; The processor, obtaining a visual field image indicative of a user's visual field; When the field of view image includes a target region of interest that is a region of interest of a predetermined type, obtaining related information related to the target region of interest; and controlling the display of the related information on a first display device viewed by the user so as to be superimposed on the region of interest of interest. Information processing system.

2. The processor, Acquire gaze information of the user; Identifying a region of interest in which the user is paying attention in the visual field image based on the line of sight information; If the region of interest is the region of interest of interest, obtain related information related to the region of interest of interest of interest; and controlling the first display device to display the related information so as to be superimposed on the region of interest of the subject. The information processing system according to claim 1 .

3. The processor, and controlling the display of the related information in a region of the field of view image that is larger than the region of interest of the target. The information processing system according to claim 1 .

4. The processor, Switching whether or not to display the related information on the first display device in response to an instruction from the user The information processing system according to claim 1 .

5. The processor, Accepting a predetermined action or speech of the user as an instruction from the user 5. The information processing system according to claim 4.

6. the region of interest of interest is a region of a second display device distinct from the first display device; The processor acquires, as the related information, the content displayed by the second display device. The information processing system according to claim 1 .

7. The second display device is a device that displays an image captured by at least one of an endoscope, a microscope, and an operating field camera.

7. The information processing system according to claim 6.

8. obtaining a visual field image indicative of a user's visual field; When the field of view image includes a target region of interest that is a region of interest of a predetermined type, obtaining related information related to the target region of interest; and controlling the display of the related information on a first display device viewed by the user so as to be superimposed on the region of interest of interest. An information processing method in which processing is performed by a computer.

9. obtaining a visual field image indicative of a user's visual field; When the field of view image includes a target region of interest that is a region of interest of a predetermined type, obtaining related information related to the target region of interest; and controlling the display of the related information on a first display device viewed by the user so as to be superimposed on the region of interest of interest. An information processing program that enables a computer to execute processing.

Images