Medical display system, control device, and control method

Abstract

This technology relates to a medical display system, a control device, and a control method which are capable of providing necessary information without distracting an operator. The medical display system according to this technology comprises a control device that has: an acquisition unit that acquires a surgical field image signal generated by a medical imaging device which images a surgical field of a patient and a presentation information signal generated by a device which includes an external device excluding the medical imaging device; and a control unit that generates a display image on the basis of the surgical field image signal and the presentation information signal. The control unit generates the display image to be displayed so that presentation information generated from the presentation information signal is disposed in an upper region among display regions of the display image, a surgical field image generated on the basis of the surgical field image signal is disposed in a middle region having the same length as the upper region in the horizontal direction, and user interface information for controlling the control device is disposed in a region below the upper region. This technology can be applied to a control device which controls display of a display unit.

Description

[Technical field]

[0001] The present technology relates to a medical display system, a control device, and a control method, and in particular to a medical display system, a control device, and a control method that are capable of presenting necessary information without disturbing the concentration of a surgeon. [Background technology]

[0002] A surgeon who performs surgery while viewing an operative field image captured by a medical imaging device needs to make decisions regarding the surgery by viewing information on other medical devices in addition to the operative field image.

[0003] Various techniques have been proposed to enable surgeons to easily obtain information on multiple medical devices. For example, Patent Document 1 discloses a technique for displaying an image of the surgical field and information on multiple medical devices on a single screen. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-228286 A Summary of the Invention [Problem to be solved by the invention]

[0005] When displaying information on multiple medical devices together with an operative field image on one screen, it is usually required that the main operative field image be displayed larger than the information on the other medical devices.

[0006] It is also necessary to ensure that the surgeon's concentration on viewing the surgical field image is not disturbed by information from other medical devices.

[0007] The present technology has been developed in light of these circumstances, and makes it possible to present the necessary information to the surgeon without disturbing his or her concentration. [Means for solving the problem]

[0008] A medical display system according to one aspect of the present technology includes a control device having an acquisition unit that acquires an operative field image signal generated by a medical imaging device that images a operative field of a patient and a presentation information signal generated by an apparatus including an external device excluding the medical imaging device, and a control unit that generates a display image based on the operative field image signal and the presentation information signal, and the control unit arranges the presentation information generated from the presentation information signal in an upper region of a display region of the display image, and has a length equal to that of the upper region in the horizontal direction. and the upper region is overlapped In the middle region, an operative field image generated based on the operative field image signal is arranged, and in the region below the upper region, user interface information for controlling the control device is arranged, and the user's line of sight position is If it is not in the upper region The presentation information arranged in the upper region A predetermined distance for the surgical field image arranged in the central region Transmittance setting to generate the display image.

[0009] A control device according to one aspect of the present technology includes an acquisition unit that acquires an operative field image signal generated by a medical imaging device that images a operative field of a patient and a presentation information signal generated by an apparatus including an external device excluding the medical imaging device, and a control unit that generates a display image based on the operative field image signal and the presentation information signal, and the control unit arranges the presentation information generated from the presentation information signal in an upper region of a display region of the display image, and has a length equal to that of the upper region in a horizontal direction. and the upper region is overlapped In the middle region, a surgical field image generated based on the surgical field image signal is arranged, and in the region below the upper region, own device The user interface information for controlling the user's gaze position is arranged. If it is not in the upper region The presentation information arranged in the upper region A predetermined distance for the surgical field image arranged in the central region Transmittance setting to generate the display image.

[0010] A control method according to one aspect of the present technology includes a control device acquiring an operative field image signal generated by a medical imaging device that captures an operative field of a patient and a presentation information signal generated by an apparatus including an external device excluding the medical imaging device, and generating a display image based on the operative field image signal and the presentation information signal. The generating of the display image includes arranging presentation information generated from the presentation information signal in an upper region of a display region of the display image, and arranging a display area having a length equal to that of the upper region in a horizontal direction. and the upper region is overlapped In the middle region, an operative field image generated based on the operative field image signal is arranged, and in the region below the upper region, user interface information for controlling the control device is arranged, and the user's line of sight position is If it is not in the upper region The presentation information arranged in the upper region A predetermined distance for the surgical field image arranged in the central region Transmittance setting This includes

[0011] In an image processing system according to one aspect of the present technology, an operative field image signal generated by a medical imaging device that images an operative field of a patient and a presentation information signal generated by an apparatus including an external device excluding the medical imaging device are acquired, and a display image is generated based on the operative field image signal and the presentation information signal. Presentation information generated from the presentation information signal is arranged in an upper region of a display region of the display image, and a display image having a length equal to that of the upper region in the horizontal direction is generated. and the upper region is overlapped In the middle region, an operative field image generated based on the operative field image signal is arranged, and in the region below the upper region, user interface information for controlling the control device is arranged, and the user's line of sight position is If it is not in the upper region The presentation information arranged in the upper region A predetermined distance for the surgical field image arranged in the central region is Transmittance setting The display image is generated.

[0012] In the control device according to one aspect of the present technology, an operative field image signal generated by a medical imaging device that images an operative field of a patient and a presentation information signal generated by a device including an external device excluding the medical imaging device are acquired, and a display image is generated based on the operative field image signal and the presentation information signal. Presentation information generated from the presentation information signal is arranged in an upper region of a display region of the display image, and a display image having a length equal to that of the upper region in the horizontal direction is generated. and the upper region is overlapped In the middle region, an operative field image generated based on the operative field image signal is arranged, and in the region below the upper region, user interface information for controlling the control device is arranged, and the user's line of sight position is If it is not in the upper region The presentation information arranged in the upper region A predetermined distance for the surgical field image arranged in the central region Transmittance setting The display image is generated. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram showing a state of surgery using a medical display system according to an embodiment of the present technology; [Diagram 2] 1A and 1B are diagrams illustrating an example of an area configuration of a display image displayed on a display device. [Diagram 3] 13 is an enlarged view of a presentation information area in which monitoring information and the like are displayed. FIG. [Figure 4] 13 is an enlarged view of an operation menu area in which an operation menu is displayed; FIG. [Diagram 5] 13A and 13B are diagrams illustrating examples of other area configurations of a display image. [Figure 6] FIG. 13 is a diagram showing an example of still another area configuration of a display image. [Figure 7] FIG. 2 is a block diagram showing an example of a hardware configuration of a control device. [Figure 8] 2 is a block diagram showing an example of a functional configuration of a control device; [Figure 9] 10 is a flowchart illustrating a presentation process of a control device. [Figure 10] 11A and 11B are diagrams illustrating an example of a method for displaying a presentation information area. [Figure 11]13A and 13B are diagrams illustrating an example of a display method of a presentation information area in which presentation information is displayed. [Figure 12] 13A and 13B are diagrams illustrating examples of other display methods of a presentation information area in which presentation information is displayed. [Figure 13] 13 is a flowchart illustrating a highlighting process of a control device. [Figure 14] FIG. 13 is a diagram showing an example of an area configuration of a display image displayed on the display device when "Reference" is selected. [Figure 15] FIG. 11 is a diagram showing an example of a display image in which information representing a reference information group is displayed. [Figure 16] FIG. 13 is an enlarged view of a main reference information group area in which thumbnail images are displayed. [Figure 17] FIG. 13 is a diagram showing an example of a display image in which reference images are displayed in a list. [Figure 18] FIG. 13 is a diagram showing an example of a display image in which a selected reference image is displayed. [Figure 19] FIG. 13 is an enlarged view of a display image in which two reference images selected by a user are displayed. [Figure 20] 13A and 13B are diagrams illustrating examples of other area configurations of a display image in which information representing a reference information group is displayed. [Figure 21] FIG. 13 is a diagram showing another example of a display image in which reference images are displayed in a list. [Figure 22] FIG. 13 is a diagram showing yet another example of a display image in which reference images are displayed in a list. [Figure 23] 10 is a flowchart illustrating a reference information presentation process of a control device. [Figure 24] FIG. 13 is a diagram showing an example of a display image displayed on the display device 14 when "Move" is selected. [Diagram 25] FIG. 13 is a diagram showing an example of a displayed direction indicator. [Figure 26] 13 is a diagram showing an example of an area configuration of a display image presenting a direction in which the display range of the operative field image cannot be changed. FIG. [Figure 27] 10 is a flowchart illustrating a display range change process of the control device. [Figure 28] 13 is a diagram showing an example of an area configuration of a display image in which an alert is displayed. FIG. [Figure 29] 13A and 13B are diagrams showing examples of other area configurations of a display image in which a selected reference image is displayed. [Diagram 30] 13A and 13B are diagrams illustrating an example of an area configuration of a display image in which buttons are displayed. [Diagram 31] FIG. 13 is a diagram showing an example of a display image displayed on the display device when "Zoom" is selected. [Diagram 32] FIG. 13 is a diagram showing an example of an operation by a combination of gaze and voice. [Diagram 33] FIG. 13 is a diagram showing an example of a display image in which a thumbnail image is focused on based on an area where a user's gaze is directed. [Diagram 34] 13A and 13B are diagrams illustrating another example of an operation by a combination of eye gaze and voice. [Diagram 35] FIG. 13 is an enlarged view of a presented information region. [Diagram 36] FIG. 2 is an enlarged view of an operation menu area. [Figure 37] FIG. 2 is an enlarged view of an operation menu area. [Figure 38] 13 is a diagram showing an example of a lower region of a selected reference image region in which operation items are displayed. FIG. [Figure 39] FIG. 13 is a diagram showing an example of a display image in which an image is displayed in each region obtained by dividing the operative field image region. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present technology will be described in the following order. 1. First embodiment 2. Second embodiment 3. Third embodiment 4. Fourth embodiment 5. Variations

[0015] <1. First embodiment> -Outline of medical display system First, an overview of a system to which the present technology can be applied will be described. Fig. 1 is a diagram showing a state of surgery using a medical display system according to an embodiment of the present technology.

[0016] The medical display system 1 in FIG. 1 comprises a microscope device 11, a control device 12, a monitoring device 13, and a display device 14.

[0017] 1 shows a medical display system 1 including such devices, in which a surgeon 21 is performing surgery on a patient 23 on a patient bed 22. In the following description, the term "user" refers to any medical staff member who uses the medical display system 1, such as the surgeon or an assistant.

[0018] The microscope device 11 is an electronic imaging type microscope device (a so-called video type microscope device). The microscope device 11 captures an image of the operative part of a patient, and outputs a signal representing an operative field image showing the operative field to the control device 12. Note that instead of the microscope device 11, a medical imaging device such as an endoscope may be used as a device for capturing an operative field image.

[0019] The control device 12 is a server device, a personal computer, etc. The control device 12 generates a display image based on a signal of an operative field image (operative field image signal) supplied from the microscope device 11 and a signal of presentation information (presentation information signal) supplied from devices including an external device.

[0020] Here, the monitoring device 13 is included in the external device. Therefore, a signal of the monitoring information (monitoring information signal) supplied from the monitoring device 13 is included in the presented information signal. A signal representing the monitoring information obtained by monitoring the biological information (bioreaction) of the patient 23 is supplied from the monitoring device 13.

[0021] The control device 12 causes the display device 14 to display a display image including the operative field image and the presented information.

[0022] The communication between the control device 12 and each of the other devices may be performed by wired communication or wireless communication.

[0023] The control device 12 and the external server 16 are connected via a predetermined network 15, such as the Internet or a LAN (Local Area Network). It is also possible to superimpose medical information acquired from the external server 16 on the display image and display it on the display device 14.

[0024] The monitoring device 13 monitors the biological information of the patient 23 and generates monitoring information representing the monitoring results of the biological information. The biological information of the patient 23 includes an electrocardiogram, a heart rate, blood oxygen saturation, arterial pressure, etc. The monitoring device 13 outputs a signal representing the monitoring information to the control device 12.

[0025] The display device 14 is provided near the user, such as on a wall of an operating room. The display device 14 displays a display image generated by the control device 12 according to the control by the control device 12.

[0026] On the display device 14, various information related to the surgery, such as biological information, physical information, and surgical procedure information of the patient 23, are displayed together with the surgical field image captured by the microscope device 11. The display on the display device 14 is appropriately switched by a user's operation. As the display device 14, various display devices such as a liquid crystal display device or an EL (Electro Luminescence) display device are used.

[0027] A camera for capturing an image of the user performing surgery, a microphone for detecting the voice emitted by the user, various sensors, and the like are installed near the display device 14. As will be described later, the line of sight of the user is detected based on the captured image, and the display on the display device 14 is switched. Also, based on the voice of the user detected by the microphone, the display on the display device 14 is switched, or an operation is performed using the display on the display device 14.

[0028] The external server 16 is, for example, a server device installed in a hospital. The external server 16 stores medical information and the like.

[0029] In this way, during surgery using the medical display system 1, the display image generated by the control device 12 is displayed on the display device 14. The user performs various procedures such as resection of the affected area while observing the state of the surgical site by looking at the display image displayed on the display device 14.

[0030] FIG. 2 is a diagram showing an example of the area configuration of a display image displayed on the display device 14. As shown in FIG.

[0031] As shown in FIG. 2, the display image is configured by arranging a presentation information area A1 and an operation menu area A3, which are narrow strip-shaped areas, above and below an operative field image area A2, in which the operative field image is displayed.

[0032] 2, the presented information area A1, the surgical field image area A2, and the operation menu area A3 have the same horizontal length. The surgical field image area A2 has a larger vertical width (height) than the presented information area A1 and the operation menu area A3.

[0033] The presented information area A1 is a display area for presented information including monitoring information showing the monitoring results by the monitoring device 13. In the presented information area A1, the presented information is displayed in a predetermined display form including text, graphs, and the like.

[0034] FIG. 3 is an enlarged view of the presented information area A1 in which presented information including monitoring information and the like is displayed.

[0035] In the example of Fig. 3, from the left, electrocardiogram waveform, heart rate, pulse wave, blood oxygen saturation, and arterial pressure are displayed as monitoring information. To the right of these monitoring information, information on surgical instruments such as information showing the output of the electric scalpel used by the user and information on time such as a timer are displayed.

[0036] In this way, the presentation information is information generated from a presentation information signal acquired from an apparatus including an external apparatus (such as the monitoring apparatus 13) excluding a medical imaging apparatus such as the microscope apparatus 11. Therefore, in addition to the monitoring information, various kinds of information that need to be presented to the user, such as information related to surgical instruments and time, are displayed as presentation information in the presentation information area A1.

[0037] That is, the presentation information can include at least the monitoring information. Therefore, the presentation information can be said to be modality information including the monitoring information. The presentation information may also include information related to surgical instruments and time. Furthermore, the monitoring information, the information related to surgical instruments, and the information related to time can be information that changes in real time.

[0038] Returning to the explanation of Fig. 2, the operative field image area A2 formed in a wide area in the center of the display image is a display area for the operative field image captured by the microscope device 11. The operative field image displayed in the operative field image area A2 is an image that changes in real time.

[0039] The operation menu area A3 is a display area for an operation menu (operation panel) consisting of various optional items. By using the display in the operation menu area A3, the user can perform various operations such as selecting an image to be displayed on the display device 14 and causing the control device 12 to communicate with the external server 16. In this way, the operation menu area A3 displays user interface information used for user operations.

[0040] FIG. 4 is an enlarged view of the operation menu area A3 in which the operation menu is displayed.

[0041] In the example of Fig. 4, the texts "Extension", "PB control", "Zoom", "Move", "Pivot", "Timer", and "Reference" are displayed, which represent optional items selectable by the user. By selecting an optional item, the user can operate each device such as the control device 12. Each optional item may be displayed by an icon.

[0042] "Extension" is an item for allowing the control device 12 to communicate with the external server 16. For example, by selecting the "Extension" item, the user can make a voice call with a user outside the operating room.

[0043] "PB control" is an item for causing the display device 14 to display an image of the surgical field immediately after the start of bleeding in a predetermined area within the displayed image.

[0044] "Zoom" is an item for controlling the display magnification of the surgical field image.

[0045] "Move" is an item for causing the display device 14 to display information for controlling the display range of the operative field image.

[0046] "Pivot" is an item for controlling the imaging angle of the microscope device 11 with respect to the surgical site.

[0047] "Timer" is an item for causing the display device 14 to display a timer in the presentation information area A1. For example, the user can cause the presentation information area A1 to display an elapsed time together with a preset time.

[0048] "Reference" is an item for causing the control device 12 to perform a reference information presentation process, which will be described later.

[0049] An operation menu consisting of a plurality of option items as described above is displayed in the operation menu area A3.

[0050] FIG. 5 is a diagram showing an example of another area configuration of a display image.

[0051] 5A to 5C show display images in which the operation menu is arranged at different positions. Each of the display images in Fig. 5A to 5C is similar to the display image described with reference to Fig. 2, except that the operation menu is arranged at different positions. Duplicate descriptions will be omitted as appropriate.

[0052] In the example of A in FIG. 5, an operation menu area A3-1, which is a display area of ​​the operation menu, is arranged as a vertically long area at the left end of the display image. A predetermined gap is formed between the presented information area A1 and the operation menu area A3-1. In the operation menu area A3-1, for example, each option item is displayed aligned vertically. In this case, it is possible to secure a wider vertical width of the surgical field image area A2.

[0053] In the example of FIG. 5B, an operation menu area A3-2, which is a display area for the operation menu, is disposed symmetrically on the right side with respect to the operation menu area A3-1 in the display image of FIG.

[0054] In the example of C in Fig. 5, an operation menu area A3-1 is arranged on the left edge of the display image, and an operation menu area A3-2 is arranged on the right edge. In this manner, the operation menu may be displayed in two areas, the left edge and the right edge.

[0055] In the example of FIG. 2, the horizontal length of the operation menu area A3 is assumed to be equal to the horizontal length of the presented information area A1 (upper area) and the surgical field image area A2 (middle area), and the user interface information is arranged in the lower area below the area. On the other hand, as shown in the example of FIG. 5, the horizontal length of the operation menu area A3 may be different from the horizontal length of the presented information area A1 (upper area) and the surgical field image area A2 (middle area), and the user interface information may be arranged in the left and right areas corresponding to at least one of the left and right ends of the middle area. Note that the upper area is, for example, an area including the upper side in the case of a rectangular image. Also, the middle area is an area including the right side or the left side, and is different from the upper area. Also, the lower area is an area including the lower side.

[0056] FIG. 6 is a diagram showing an example of still another area configuration of a display image.

[0057] As shown on the left side of Fig. 6, the presented information area A1 and the operation menu area A3 may be arranged to be superimposed on the operative field image area A2. In this case, a predetermined transparency is set for the display of the presented information area A1 and the display of the operation menu area A3. When the presented information in the presented information area A1 and the operation menu in the operation menu area A3 are arranged in the upper layer, a part of the operative field image in the operative field image area A2 is seen through the lower layer.

[0058] The area configuration on the right side of Fig. 6 is the same as the area configuration of the display image described with reference to Fig. 2. That is, the presented information area A1 is disposed adjacent to the upper part of the surgical field image area A2. The operation menu area A3 is disposed adjacent to the lower part of the surgical field image area A2.

[0059] As described above, according to the control device 12, it is possible to display the presentation information together with the operative field image on one image in a form arranged on top of the operative field image.

[0060] Here, even if a surgeon is performing surgery while looking at the image displayed on the display device, he or she often looks at what is in front of them, and it is also expected that they may avert their gaze to someone next to them (such as an assistant) or to other equipment. In other words, the surgeon's gaze tends to go left and right and downward. On the other hand, if the surgeon's gaze goes upward, it is highly likely that they are intentionally directing their gaze.

[0061] In addition, papers and other sources have disclosed that there is an asymmetry of visual attention between the upper and lower visual fields of humans. In other words, it is known that humans pay more attention downward than upward.

[0062] Here, the presented information is displayed above the surgical field image, so that the surgeon must consciously turn his or her gaze to see the presented information. This makes it possible to present the presented information without disturbing the concentration of the surgeon who is performing surgery while looking at the surgical field image.

[0063] Control device configuration FIG. 7 is a block diagram showing an example of the hardware configuration of the control device 12. As shown in FIG.

[0064] In the control device 12, a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, and a RAM (Random Access Memory) 53 are interconnected by a bus 54. An input / output interface 55 is further connected to the bus 54. An input unit 56, an output unit 57, a recording unit 58, a communication unit 59, and a drive 60 are connected to the input / output interface 55.

[0065] The input unit 56 includes a microphone, a keyboard, a mouse, etc. The output unit 57 includes a speaker, a display, etc. The recording unit 58 includes a hard disk, a non-volatile memory, etc.

[0066] The communication unit 59 is composed of a network interface, etc. The communication unit 59 communicates with external devices such as the microscope device 11, the monitoring device 13, and the external server 16 connected via the network 15.

[0067] The drive 60 drives a removable recording medium 61 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0068] In the computer configured as described above, the CPU 51 loads the programs recorded in the ROM 52 or the recording unit 58 into the RAM 53 via the input / output interface 55 and the bus 54 and executes them, thereby causing the above-described screen display.

[0069] 8 is a block diagram showing an example of a functional configuration of the control device 12. At least a part of the functional units shown in FIG. 8 is realized by the CPU 51 in FIG.

[0070] As shown in FIG. 8, in the control device 12, an acquisition unit 81, a control unit 82, and a display control unit 83 are realized.

[0071] 7 to acquire signals of the operative field image output from the microscope device 11 and signals of the presentation information output from external devices including the monitoring device 13. The signals acquired by the acquisition unit 81 are supplied to the control unit 82. Note that the signals of the presentation information may be generated within the control device 12 and acquired by the acquisition unit 81.

[0072] The control unit 82 generates presentation information and an operative field image based on the signal supplied from the acquisition unit 81. The control unit 82 generates a display image having the area configuration of Fig. 2 by arranging the presentation information above the operative field image and arranging an operation menu below the operative field image. The control unit 82 outputs the display image to the display control unit 83.

[0073] The display control unit 83 outputs the display image supplied from the control unit 82 to the display device 14, causing the display image to be displayed.

[0074] -Control device operation Next, the presentation process of the control device 12 will be described with reference to the flowchart of FIG.

[0075] In step S1, the acquisition unit 81 acquires a signal of the surgical field image output from the microscope device 11 and a signal of the presentation information output from an external device or a signal of the presentation information generated by the control device 12, and outputs them to the control unit 82.

[0076] In step S2, the control unit 82 generates a display image based on the signal supplied from the acquisition unit 81, and outputs the display image to the display control unit 83.

[0077] In step S3, the display control unit 83 outputs the display image to the display device 14, and causes the display image to be displayed.

[0078] In step S4, the control unit 82 determines whether or not to end the process.

[0079] If it is determined in step S4 that the process should not be terminated, the process returns to step S1, and the subsequent processes are performed. That is, the operative field image and the presentation information, which change in real time, are updated and reflected in the displayed image.

[0080] On the other hand, if it is determined in step S4 that the process is to be ended, the presentation process is ended.

[0081] Through the above processing, the control device 12 can present presentation information including monitoring information without disturbing the concentration of the user who is performing surgery while viewing the operative field image.

[0082] <2. Second embodiment> In this example, the line of sight of the user with respect to the display device 14 is detected. A gaze point, which is a position on the display image at which the user is looking, is obtained from the detected line of sight, and the display method of the presentation information area A1 in FIG. 2 is switched based on the obtained gaze point.

[0083] FIG. 10 is a diagram showing an example of a display method of the presentation information area A1.

[0084] In the upper part of FIG. 10, a presented information area A1-1, which is a display area for presented information, and an operation menu area A3 are arranged so as to be superimposed on a surgical field image area A2.

[0085] After the display image is displayed on the display device 14, the control device 12 determines the user's gaze point on the display image. In the example in the upper part of Fig. 10, the gaze point P1 on the display image is determined based on the user's line of sight. When there is no gaze point on the presentation information area A1-1, the control device 12 sets a predetermined transparency for the display of the presentation information area A1-1.

[0086] 10, a fixation point P2 on the presentation information area A1-2 is obtained. When the fixation point is on the presentation information area A1-2, the control device 12 generates a display image without setting a transparency for the display of the presentation information area A1-2.

[0087] The user's line of sight with respect to the display device 14 is detected in real time. The control device 12 controls the transmittance of the presented information area A1 based on the detected line of sight to generate a display image.

[0088] Specifically, FIG. 11 shows an example of a display method for the presented information area A1 in which presented information is displayed.

[0089] The upper part of Fig. 11 shows an example of a steady state display when there is no gaze point on the presentation information area A1-1. Presentation information such as monitoring information similar to that in the presentation information area A1 described with reference to Fig. 3 is displayed in the presentation information area A1-1 with a predetermined transmittance.

[0090] By displaying the presented information in this steady state, the display of the presented information area A1-1 blends in with the outside of the field of view, allowing the user to concentrate on the procedure without being distracted by the presented information.

[0091] 11 shows an example of highlighting when the gaze point is on the presentation information area A1-2. Presentation information such as monitoring information similar to that in the presentation information area A1 described with reference to FIG. 3 is displayed in the presentation information area A1-2 without setting the transparency.

[0092] Only when the user needs to see the presented information, the user can clearly see the highlighted presented information by directing his / her gaze toward the presented information area A1-2.

[0093] Note that here, switching between the steady state and the highlighted display may be performed based on information about the user, such as the user's voice, touch operation, gesture (motion), and the like, in addition to the user's line of sight.

[0094] Although an example has been described in which the steady state and the highlighted display are switched by setting the transmittance, the steady state and the highlighted display may be switched by other settings of the display of the presentation information area A1. For example, the steady state and the highlighted display may be switched by changing the luminance, the display color, or the inversion of the display color of the presentation information area A1.

[0095] FIG. 12 is a diagram showing an example of another display method for the presented information area A1 in which presented information is displayed.

[0096] When the user's line of sight is not directed toward the presented information area A1, in the example in the upper part of FIG. 12, a display image is displayed that is configured of the surgical field image area A2 and the operation menu area A3, which are areas other than the presented information area A1.

[0097] When the user's gaze is directed toward a predetermined area, the display of the presented information area A1 is gradually displayed from the bottom to the top over time in the examples in the middle and bottom rows of Fig. 12. For example, when the user's gaze is detected upward on the surgical field image area A2, the presented information is displayed. In addition, when the user's gaze is directed toward a predetermined area, the presented information may be displayed by fading in, for example.

[0098] Next, the highlighting process of the control device 12 will be described with reference to the flowchart of FIG.

[0099] The highlighting process of FIG. 13 begins after a steady-state display image is displayed on the display device 14. In step S101, the control unit 82 detects the user's line of sight based on the captured image captured by the camera. The control unit 82 obtains a gaze point on the display image based on the detected line of sight of the user, and determines whether or not a gaze point of the user has been detected for a predetermined period or longer on the presentation information area A1 based on the obtained gaze point.

[0100] If it is determined in step S101 that a line of sight toward the presented information area A1 has been detected for a period of time equal to or longer than the predetermined period, the process proceeds to step S102.

[0101] In step S102, the control unit 82 generates a display image by arranging the presentation information so that it is displayed in a highlighted manner, and outputs the display image to the display control unit 83. The display control unit 83 outputs the display image supplied from the control unit 82 to the display device 14, and causes the display image to be displayed.

[0102] In step S103, the control unit 82 determines whether or not to end the highlighting process. If it is determined in step S101 that a line of sight to the presentation information area A1 has not been detected for a predetermined period or longer, step S102 is skipped and the process proceeds to step S103, where it is determined whether or not to end the highlighting process.

[0103] If it is determined in step S103 that the highlighting process should not be ended, the process returns to step S101, and the subsequent processes are carried out.

[0104] On the other hand, if it is determined in step S103 that the highlighting process is to be ended, the highlighting process is ended.

[0105] The above process makes it possible to present the presented information without disturbing the concentration of the user who is performing surgery while viewing the operative field image. Also, it becomes possible to clearly present the presented information only when the user needs the presented information.

[0106] <3. Third embodiment> When the item "Reference" is selected from the operation menu, the control device 12 performs a reference image presentation process. The case where "Reference" is selected will be described below.

[0107] FIG. 14 is a diagram showing an example of the area configuration of the display image displayed on the display device 14 when "Reference" is selected.

[0108] As shown in A and B of Figure 14, when "Reference" is selected, the display image is composed of a presentation information area A1, which is a narrow band-shaped area, placed on top of the surgical field image area A2, in which the surgical field image is displayed, and reference information group areas A11 to A14, which are rectangular areas, superimposed on the surgical field image area A2.

[0109] Each of the display images A and B in Figures 14 is similar to the display image described with reference to Figure 2, except that there is no operation menu area A3 and there are reference information group areas A11 to A14, so duplicate explanations will be omitted as appropriate.

[0110] In the example of A in Fig. 14, the reference information group regions A11 and A12 are arranged vertically side by side on the left side of the display image. A predetermined gap is formed between the reference information group regions A11, A12 and the left edge of the display image. A predetermined gap is also formed between the reference information group region A12 and the bottom edge of the display image, and between the reference information group region A11 and the presentation information region A1.

[0111] In the example of A in FIG. 14, furthermore, reference information group regions A13 and A14 are arranged on the right side of the displayed image symmetrically with respect to reference information group regions A11 and A12.

[0112] The reference information group regions A11 to A14 are display regions for reference images. The reference images include monitoring information representing the monitoring results by the monitoring device 13 and medical information acquired from the external server 16. The medical information includes pathological images, ultrasound images, ICG (Indocyanine green) images, MRI (Magnetic Resonance Imaging) images, CT (Computed Tomography) images, angio images (angiography images), and X-ray images.

[0113] The reference images are divided into predetermined reference information groups. In the reference information group areas A11 to A14, information representing the reference information groups is displayed for each reference information group.

[0114] The positions of the reference information group regions A11 to A14 on the display image may be determined based on the category of the reference information group. In this case, the information of each reference information group having the same reference information group category can be displayed side by side on the same side of the display image, either the left or right side. For example, the reference information group regions A11 and A12 display reference images that are updated in real time, such as monitoring information and ultrasound images, and the reference information group regions A13 and A14 display reference images that are prepared before the start of surgery, such as MRI images and CT images.

[0115] As shown in B of Fig. 14, the reference information group regions A11 to A14 may be arranged vertically on the left side of the display image, or may be arranged symmetrically on the right side of the display image with respect to the reference information group regions A11 to A14 in the display image in B of Fig. 14.

[0116] FIG. 15 is a diagram showing an example of a display image in which information indicating a reference information group is displayed.

[0117] In the example of FIG. 15, thumbnail images corresponding to the reference information groups are displayed in the reference information group areas A11 to A14 as information representing the reference information groups.

[0118] In the example of FIG. 15, the reference images are classified into four groups, namely, monitoring information, ultrasound images, ICG images, and MRI images, as reference information groups. The thumbnail images corresponding to the reference information groups displayed in the reference information group areas A11 to A14 use images representing monitoring information, ultrasound images, ICG images, and MRI images, respectively. Any group may be set as the reference information group, and any image may be used as the thumbnail image. For example, a surgical navigation image showing surgical navigation may be set as the reference information group, and the surgical navigation image may be displayed in the reference information group area. One thumbnail image may be selected from the reference images associated with each group and used as the thumbnail image.

[0119] FIG. 16 is an enlarged view of a reference information group region A11 in which thumbnail images of monitoring information are displayed and a reference information group region A12 in which thumbnail images of ultrasound images are displayed.

[0120] 16A, monitoring information showing the monitoring results by the monitoring device 13 is displayed as thumbnail images showing the monitoring information arranged vertically in the reference information group area A11. The monitoring information displayed in the reference information group area A11 may be information including the monitoring information displayed in the presentation information area A1 described with reference to FIG.

[0121] The name of the reference information group to be displayed and a number or symbol representing the reference information group are displayed at the top of the reference information group area A11. In the example of A in Fig. 16, the text "Monitor" is displayed at the top of the reference information group area A11 as the name of the monitoring information, which is the reference information group. In addition, to the right of the text "Monitor", the text "L1" is displayed as the symbol representing the monitoring information.

[0122] In the example of FIG. 16B, an ultrasound image acquired from the external server 16 is displayed in the reference information group area A12 as a thumbnail image representing the ultrasound image.

[0123] In the upper part of the reference information group region A12, as in the reference information group region A11, the text "Ultrasound" is displayed as the name of the ultrasound image, which is the reference information group. In addition, to the right of the text "Ultrasound", the text "L2" is displayed as the symbol representing the ultrasound image.

[0124] The user can display a list of reference images included in the selected reference information group by selecting a thumbnail image from the displayed list. At this time, the user can select a thumbnail image of the reference information group by, for example, line of sight or voice.

[0125] When voice input is performed, the user can select a thumbnail image of a reference information group by vocalizing the name of the reference information group or a symbol representing the reference information group, whereas when gaze input is performed, the user can reliably select a required reference information group by looking at the name of the reference information group.

[0126] FIG. 17 is a diagram showing an example of a display image in which reference images are displayed in a list.

[0127] The display image in the upper part of Fig. 17 is the same display image as the display image in which the thumbnail images of the reference information group are displayed as described with reference to Fig. 15. A dashed rectangle R1 on the diagonally lower right side of the display image indicates that the thumbnail image of the MRI image displayed in the reference information group area A14 has been selected.

[0128] When an MRI image is selected, the control device 12 generates, for example, a display image shown in the lower part of Fig. 17. In the example of the display image in the lower part of Fig. 17, a presentation information area A1 is arranged on an operative field image area A2, and rectangular reference image areas A31-1 to A31-7 are arranged so as to be superimposed on the operative field image area A2, thereby forming a display image.

[0129] That is, in the example display image in the lower part of Figure 17, after the images displayed in reference information group areas A11 to A14 disappear, reference image areas A31-1 to A31-7 are arranged in that order moving from right to left, and are lined up horizontally below the display image.

[0130] The reference image areas A31-1 to A31-7 are display areas of reference images included in a reference information group selected by a user. In the example of the display image in the lower part of Fig. 17, the reference image areas A31-1 to A31-7 display reference images that are different MRI images. Note that, although the example in the lower part of Fig. 17 shows an example in which there are seven reference image areas, the number of reference image areas constituting the display image is arbitrary.

[0131] When the number of reference images included in the reference information group selected by the user is greater than the number of reference image areas constituting the display image, the reference images are displayed in the display image by the number of reference image areas. In this case, the control device 12 switches the reference images to be displayed in the reference image areas by accepting a predetermined operation by the user. In the example of the display image in the lower part of FIG. 17, information regarding the number of reference images included in the selected reference information group is displayed in the operation menu. At this time, operation items such as "Go Back" for returning to the reference information group selection screen and "Cancel" for canceling the selection of the reference image and ceasing the display of the reference image may be displayed in the operation menu.

[0132] If the reference information group selected by the user contains one reference image, a display image including a reference image area is not generated, and a display image in which the reference image is displayed in a selected reference image area (described later) is generated.

[0133] Moreover, above the reference image areas A31-1 to A31-7 in which the reference images are displayed, numbers and symbols indicating the respective reference images to be displayed are displayed.

[0134] The user can display only the selected reference image among the reference images included in the reference information group by selecting the reference image to be displayed. At this time, the user can select the reference image by, for example, line of sight or voice.

[0135] When voice input is used, the user can select a reference image by vocalizing the number or symbol representing the respective reference image.

[0136] A dashed rectangle R2 shown below the display image in the lower part of FIG. 17 indicates that the reference image displayed in the reference image area A31-3 has been selected by the user.

[0137] FIG. 18 is a diagram showing an example of a display image in which a selected reference image is displayed.

[0138] The display image at the top of Fig. 18 is a display image that is displayed after the display image displayed in the reference image area A31-3 in Fig. 17 is selected. In the example of the display image at the top of Fig. 18, a rectangular selected reference image area A41 is arranged to be superimposed on the display image described with reference to Fig. 2. In the example of the display image at the top of Fig. 18, the selected reference image area A41 is arranged at the right end of the display image. A predetermined gap is formed between the selected reference image area A41 and the presentation information area A1, and between the selected reference image area A41 and the operation menu area A3.

[0139] The selected reference image area A41 is a display area of ​​a reference image selected by a user from a list of reference images included in a reference information group. In the example of the display image in the upper part of Fig. 18, an MRI image selected by a user is displayed in the selected reference image area A41. Note that the surgical field image area A2 may be shifted in the opposite direction to the selected reference image area A41 to make it easier to see the target site, etc.

[0140] When a reference image different from the reference image displayed in selected reference image area A41 is selected after a reference image is displayed in selected reference image area A41, the display image may be configured by superimposing rectangular selected reference image areas A41 and A42 on the display image described with reference to Figure 2, as shown in the lower part of Figure 18.

[0141] In the example of the display image in the lower part of FIG. 18, the selected reference image areas A41 and A42 are arranged vertically side by side at the right end of the display image. The selected reference image area A42 is a display area of ​​a reference image selected by the user from a list of reference images included in a reference information group, similar to the selected reference image area A41. In the example of the display image in the lower part of FIG. 18, an ICG image is displayed in the selected reference image area A42. When the user selects a third reference image while two reference images are displayed, the first reference image may be displayed in the display area in which the second reference image was displayed, and the third reference image may be displayed in the display area in which the second reference image was displayed. In addition, when the display of the reference image is erased by the user selecting the "Cancel" item, etc., and "Reference" is selected again, the control device 12 may display the reference image selected when "Reference" was selected last time as a reference image again or present it to the user as a candidate image. The candidate image is an image to which a visual effect different from other images in the list of reference images is applied so that it can be recognized as the reference image selected last time.

[0142] FIG. 19 is a diagram showing an example in which a display image in which two reference images selected by a user are displayed is enlarged.

[0143] The display image in Figure 19 is similar to the display image shown and described in the lower part of Figure 18, except that text is superimposed on the surgical field image, so duplicated explanations will be omitted as appropriate.

[0144] In the example of Fig. 19, numbers and names representing the reference images displayed in the selected reference image areas A41 and A42 are displayed on the left side of the selected reference image areas A41 and A42. The user can select a displayed reference image by line of sight and display the selected reference image in an enlarged form on the display image. The user may also select a displayed reference image by uttering the number or name representing the reference image as voice.

[0145] As shown in Fig. 19, the operation indicator region A51 may be arranged so as to be superimposed on the operative field image region A2. In the example of Fig. 19, the operation indicator region A51 is arranged above the operative field image region A2. Based on the line of sight of the user detected by the control device 12, the position on the display image where the operation indicator region A51 is arranged may be determined by the control device 12.

[0146] The operation indicator area A51 includes a display area for a number and a name representing the operative field image displayed in the operative field image area A2. In the example of Fig. 19, the text "CAMERA" is displayed in the operation indicator area A51 as a name representing the operative field image together with the number representing the operative field image.

[0147] By directing the user's gaze toward the operation indicator area A51, the user can select the operative field image and display the initial state, that is, the display image composed of the presentation information area A1, the operative field image area A2, and the operation menu area A3 described with reference to Fig. 2. The operative field image may also be selected by the user uttering a number or name representing the operative field image as voice.

[0148] As described above, the control device 12 makes it possible to display the presented information and the operative field image on a single image with the presented information placed on top of the operative field image, and also makes it possible to display reference images necessary for the surgery from among the reference images obtained from the external server 16 located outside the operating room.

[0149] In addition, when the user selects the "Reference" item from the operation menu, other information other than the thumbnail image of the reference information group may be displayed in the reference information group area as information representing the reference information group.

[0150] FIG. 20 is a diagram showing an example of another area configuration of a display image in which information representing a reference information group is displayed.

[0151] Fig. 20 shows an area configuration of a display image in which an index is displayed in text form as information representing a reference information group. The display image of Fig. 20 is similar to the display image described with reference to Fig. 14, except for the number and arrangement of reference information group areas, and therefore duplicated explanations will be omitted as appropriate.

[0152] In the example of Fig. 20, a reference information group area A15, which is a display area for information representing a reference information group, is disposed at the left edge of the display image. Note that the reference information group area A15 may be disposed at the right edge of the display image.

[0153] 20, an index that is a list of reference information groups is displayed in text form in the reference information group area A15. The index includes, for example, the name of the reference information group and a number that indicates the reference information group.

[0154] When an index is displayed as information representing a reference information group, just as when a thumbnail image is displayed as information representing a reference information group, the user can select a reference information group by a specified gaze or voice and display a list of reference images included in the selected reference information group.

[0155] Also, with reference to Figure 17, an example has been described where a list of reference images included in a selected reference information group is displayed below the display image, but the position where the list of reference images is displayed may be determined by the control device 12 based on the user's line of sight, etc.

[0156] FIG. 21 is a diagram showing another example of a display image in which reference images are displayed in a list.

[0157] Figures 21A and 21B show display images in which the position of the reference image is different from that of the display image described with reference to Figure 17. Each of the display images in Figures 21A and 21B is similar to the display image described with reference to Figure 17 except for the position of the reference image, and therefore overlapping descriptions will be omitted as appropriate.

[0158] In the example of A in Fig. 21, reference image areas A31-1 to A31-4, which are display areas of reference images included in a reference information group selected by a user, are arranged vertically as rectangular areas on the left edge of the display image. Different reference images are displayed in the reference image areas A31-1 to A31-4.

[0159] In the example of A in Fig. 21, an example in which the number of reference image areas constituting the display image is four has been described, but the number of reference image areas is arbitrary. If the number of reference images included in the reference information group selected by the user is greater than the number of reference image areas, the reference images are displayed in the display image as many as the number of reference image areas. In this case, the control device 12 switches the reference images displayed in the reference image areas by accepting a predetermined operation by the user, and generates a display image as shown in B in Fig. 21.

[0160] The display image in B of Fig. 21 is a display image in which the reference image areas A31-1 to A31-4 of the display image in A of Fig. 21 are replaced with reference image areas A31-5 to A31-8 which are display areas of reference images included in a reference information group selected by a user. The reference image areas A31-5 to A31-8 display reference images different from the reference images displayed in the reference image areas A31-1 to A31-4, respectively.

[0161] FIG. 22 is a diagram showing an example of yet another display image in which reference images are displayed in a list.

[0162] In the examples of A and B in FIGS. 22, reference image regions A31-1 to A31-8 are arranged on the right edge of the display image, symmetrically to reference image regions A31-1 to A31-8 in each of the display images in A and B in FIGS.

[0163] In this way, the position where the list of reference images included in the reference information group selected by the user is arranged is not limited to the bottom edge of the display image, but may be the left edge or right edge of the display image.

[0164] Here, the arrangement position of the list of reference images can be determined according to information on the reference images. For example, if the reference images are images acquired in time series, the reference images may be arranged in time series at the bottom edge of the display image in the horizontal direction. Also, for example, if the reference images are slice images (such as MRI images), the reference images may be arranged in slice order at the left or right edge of the display image in the vertical direction.

[0165] Next, the reference information presentation process of the control device 12 will be described with reference to the flowchart of FIG.

[0166] This reference information presentation process is started when the user selects the "Reference" item in the operation menu.

[0167] In step S201, the acquisition unit 81 acquires a reference image from the external server 16 or the like. The acquired reference image is supplied to the control unit .

[0168] The control unit 82 generates a display image having the area configuration of A in Fig. 14 by arranging the presentation information above the operative field image and, for example, arranging thumbnail images of the reference information group on the left and right sides of the display image. The control unit 82 outputs the display image to the display control unit 83. The display control unit 83 outputs the display image supplied from the control unit 82 to the display device 14, and causes the reference information group to be presented.

[0169] In step S202, the control unit 82 accepts the selection of a thumbnail image of a reference information group by the user. For example, the control unit 82 detects the user's line of sight based on a captured image provided by a camera installed near the display device 14. The control unit 82 identifies the reference information group selected by the user based on the detected line of sight of the user.

[0170] In step S203, the control unit 82 generates a display image as shown in the lower part of Fig. 17 by arranging the reference images included in the reference information group selected by the user below the display image. The control unit 82 outputs the display image to the display control unit 83. The display control unit 83 outputs the display image supplied from the control unit 82 to the display device 14, and causes a list of reference images included in the reference information group selected by the user to be presented.

[0171] In step S204, the control unit 82 accepts the selection of a reference image by the user. For example, the control unit 82 detects the user's line of sight based on a captured image provided by a camera installed near the display device 14. The control unit 82 identifies the reference image selected by the user based on the detected line of sight of the user.

[0172] In step S205, the control unit 82 places the reference image selected by the user at the right edge of the display image to generate a display image as shown in the upper part of Fig. 18. The control unit 82 outputs the display image to the display control unit 83. The display control unit 83 outputs the display image supplied from the control unit 82 to the display device 14, and causes the display device 14 to present the reference image selected by the user.

[0173] This is the end of the reference information presentation process. Note that if the number of reference images included in the reference information group selected by the user is one, the processes of steps S203 and S204 are skipped.

[0174] Through the above process, the control device 12 can present the presentation information and display the operative field image and the reference image without disturbing the user's concentration.

[0175] <4. Fourth embodiment> When the "Move" item is selected from the operation menu, the control device 12 performs a display range change process. The following describes the case where "Move" is selected.

[0176] FIG. 24 is a diagram showing an example of a display image displayed on the display device 14 when "Move" is selected.

[0177] As shown in Figure 24, when "Move" is selected, the display image is composed of a presentation information area A1, which is a narrow band-shaped area, placed on the surgical field image area A2, in which the surgical field image is displayed, and direction indicator areas A61-0 to A61-7 superimposed on the surgical field image area A2.

[0178] The display image in FIG. 24 is similar to the display image described with reference to FIG. 2, except that there is no operation menu area A3 and there are direction indicator areas A61-0 to A61-7, and therefore a duplicated description will be omitted where appropriate.

[0179] In the example of Fig. 24, direction indicator regions A61-1, A61-3, A61-5, and A61-7 are arranged at the four corners of the surgical field image region A2. Direction indicator region A61-0 is arranged at the center of direction indicator regions A61-7 and A61-1, and direction indicator region A61-2 is arranged at the center of direction indicator regions A61-1 and A61-3. Direction indicator region A61-4 is arranged at the center of direction indicator regions A61-3 and A61-5, and direction indicator region A61-6 is arranged at the center of direction indicator regions A61-5 and A61-7.

[0180] The direction indicator areas A61-0 to A61-7 include display areas of indicators that indicate the direction in which the display range of the displayed surgical field image can be changed. In the example of Fig. 24, the name of the direction and the number representing the direction are displayed as text as the direction indicator. Furthermore, an icon representing the direction is displayed as the direction indicator. These displays allow the user to intuitively recognize what operation to perform next.

[0181] The user can change the display range of the operative field image toward the selected direction by selecting a direction indicator displayed in the direction indicator areas A61-0 to A61-7 by line of sight or voice. That is, the direction indicator displayed in the direction indicator areas A61-0 to A61-7 indicates the direction in which the angle of view of the operative field image can be moved. When changing the angle of view, the camera in the microscope device 11 may be driven, or image processing of the captured image captured by the camera may be performed. In addition, when a predetermined voice (voice command) is input, the display image may be configured to display a predetermined mark at the user's line of sight. Furthermore, an operation may be performed to follow the predetermined mark so that it becomes the center of the display image. For example, when the user selects "Move" by voice input, the control device 12 displays a predetermined mark indicating the user's line of sight on the display image, and changes the display of the operative field image so that the predetermined mark becomes the center position of the display image.

[0182] If there is a direction in which the display range of the surgical field image cannot be changed, the control device 12 generates a display image in which only the direction indicators indicating the directions in which the display range can be changed are displayed among the direction indicators displayed in the direction indicator areas A61-0 to A61-7.

[0183] FIG. 25 is a diagram showing an example of a displayed direction indicator.

[0184] As shown in the upper part of FIG. 25, when there is no direction in which the display range of the operative field image cannot be changed, the display image has direction indicators displayed in direction indicator areas A61-0 to A61-7.

[0185] For example, when a user selects a direction indicator displayed in the direction indicator area A61-2, a surgical field image with a display range shifted to the right is displayed in the surgical field image area A2. The display range may be changed by performing a predetermined image processing on the surgical field image supplied from the microscope device 11, or the control device 12 may control the microscope device 11 to change the display range of the surgical field image.

[0186] Furthermore, for example, when it is not possible to change the display range of the surgical field image to the right, to the upper right, or to the lower right, the control device 12 generates a display image in which direction indicators are displayed in direction indicator area A61-0 and direction indicator areas A61-4 to A61-7, as shown in the lower part of the arrow tip in Figure 25.

[0187] The three dashed circles on the display image indicate that the direction indicators displayed in the direction indicator areas A61-1 to A61-3 are not displayed. The direction indicators displayed in the direction indicator areas A61-1 to A61-3 are direction indicators representing the rightward, upper right, and lower right directions, whose display ranges cannot be changed.

[0188] As described above, the control device 12 presents to the user the directions in which the display range of the displayed operative field image cannot be changed by hiding the direction indicators indicating the directions. Here, the directions in which the display range of the operative field image cannot be changed may be indicated by a display different from the direction indicators.

[0189] FIG. 26 is a diagram showing an example of an area configuration of a display image presenting a direction in which the display range of the operative field image cannot be changed.

[0190] A and B of Fig. 26 show display images with different positions of change limit information, which is information indicating the direction in which the display range of the operative field image cannot be changed. Each display image of A and B of Fig. 26 is similar to the display image described with reference to Fig. 2 except that there is no operation menu area A3 and there is a display area for the change limit information, so duplicated explanations will be omitted as appropriate. Direction indicator areas A61-0 to A61-7 may be arranged on the display images of A and B of Fig. 26, and direction indicators may be displayed in each area.

[0191] In the example of A in Fig. 26, a change limit information area A71-1, which is a display area for change limit information, is disposed as a vertically long area at the right end of the display image. In the change limit information area A71-1, for example, a monochrome image with a predetermined brightness is displayed.

[0192] The display image A in Figure 26 is generated by the control device 12, for example, when the display range of the operative field image continues to change toward the right, as indicated by arrow D1, and it becomes impossible to change the display range of the operative field image to the right, in the upper right direction, or in the lower right direction.

[0193] In the example of B in Fig. 26, a change limit information area A71-2, which is a display area of ​​the change limit information, is arranged as a long and narrow strip area at the bottom end of the display image. The display image of B in Fig. 26 is generated by the control device 12 when, for example, as shown by the arrow D2, the display range of the operative field image continues to change downward and it becomes impossible to change the display range of the operative field image downward, diagonally downward to the left, or diagonally downward to the right.

[0194] Next, the display range change process of the control device 12 will be described with reference to the flowchart of FIG.

[0195] This display range change process is started when the user selects the "Move" item in the operation menu.

[0196] In step S301, the control unit 82 generates the display image of Fig. 24 by arranging presentation information above the operative field image and, for example, arranging direction indicators at the four corners of the operative field image area A2 and at the center positions of the corners. The control unit 82 outputs the display image to the display control unit 83. The display control unit 83 outputs the display image provided by the control unit 82 to the display device 14, and causes it to present a direction in which the display range of the operative field image can be changed.

[0197] In step S302, the control unit 82 accepts the selection of a direction indicator by the user. For example, the control unit 82 detects the user's line of sight based on a captured image provided by a camera installed near the display device 14. The control unit 82 identifies the direction indicator selected by the user based on the detected line of sight of the user.

[0198] The control unit 82 generates a display image in which the display range of the operative field image is changed according to the direction indicator selected by the user. The control unit 82 outputs the display image to the display control unit 83. The display control unit 83 outputs the display image supplied from the control unit 82 to the display device 14, and causes the display image in which the display range of the operative field image is changed to be displayed.

[0199] In step S303, the control unit 82 determines whether or not the display range has reached its limit at which it can be changed.

[0200] If it is determined in step S303 that the display range has reached its limit of change, the process proceeds to step S304.

[0201] In step S304, the control unit 82 generates a display image in which, for example, among the direction indicators, only direction indicators in the directions in which the display range of the operative field image can be changed are arranged. The control unit 82 outputs the display image to the display control unit 83. The display control unit 83 outputs the display image supplied from the control unit 82 to the display device 14, and causes it to present the limit at which the display range of the operative field image can be changed, as shown in B of Fig. 25. In addition, as shown in Fig. 26, change limit information may be displayed in a direction in which the display range of the operative field image cannot be changed.

[0202] In step S305, the control unit 82 determines whether or not to end the display range change process based on the user's selection. If it is determined in step S303 that the display range of the operative field image is not at the limit for which it can be changed, step S304 is skipped and the process proceeds to step S305, where it is determined whether or not to end the display range change process based on the user's selection.

[0203] If it is determined in step S305 that the display range change process should not be ended, the process returns to step S301, and the subsequent processes are carried out.

[0204] On the other hand, if it is determined in step S305 that the display range change process is to be ended, the display range change process is ended.

[0205] Through the above process, the control device 12 can present the presentation information and change the display range of the surgical field image without disturbing the user's concentration.

[0206] <5. Modifications> Example of adjusting the display position based on the work concentration area The control device 12 determines the gaze point based on the user's line of sight detected based on the captured image provided by a camera installed near the display device 14. At this time, the control device 12 may determine a predetermined work concentration area based on the distribution of the gaze points.

[0207] The work concentration area is an area on the display image where the user is expected to focus his / her gaze. Here, an area included in the surgical field image that changes more than a predetermined amount within a predetermined time may be determined as the work concentration area, which is a location with large movement.

[0208] Also, an alert may be displayed at a predetermined position on the display image based on the determined work concentration area.

[0209] FIG. 28 is a diagram showing an example of an area configuration of a display image in which an alert is displayed.

[0210] Each of the display images A and B in FIG. 28 is similar to the display image described with reference to FIG. 2, except that an alert is displayed, and therefore overlapping descriptions will be omitted where appropriate.

[0211] In the example of A in Fig. 28, the work concentration area A81 is obtained by the control device 12 as a circular area in the approximate center of the surgical field image area A2. In the example of A in Fig. 28, an alert #1 is placed near the work concentration area A81. In addition, the alert #1 is placed so as to be superimposed on the surgical field image area A2.

[0212] Alert #1 is displayed based on information such as monitoring information. Alert #1 can be displayed by setting a predetermined transparency. Note that the work concentration area A81 is displayed for convenience of explanation, but is not actually displayed.

[0213] In the example of B in Fig. 28, alert #1 is arranged in the presented information area A1, unlike alert #1 in A in Fig. 28. In this case, alert #1 is displayed in the presented information area A1 similarly to the presented information such as monitoring information displayed in the presented information area A1.

[0214] In this way, the position of alert #1 on the display image can be set to any position. Also, other information based on monitoring information, etc. may be displayed in the same manner as alert #1.

[0215] Also, the arrangement position of the information to be superimposed on the operative field image area A2 may be determined based on the obtained work concentration area. For example, the arrangement position of the selected reference image area described in the third embodiment may be determined based on the work concentration area.

[0216] FIG. 29 is a diagram showing another example of an area configuration of a display image in which a selected reference image is displayed.

[0217] The display image in A of FIG. 29 is the same as the display image described with reference to the upper part of FIG. 18, except for the arrangement position of the selected image region, and therefore overlapping descriptions will be omitted as appropriate.

[0218] In the example of A in Fig. 29, the circular area on the upper right side of the display image is the work concentration area A81. The control device 12 places the selected reference image area A41 at either the left or right end of the display image farther away from the work concentration area A81. In the example of A in Fig. 29, the reference image area A41-1, which is the display area of ​​the reference image selected by the user, is placed at the left end of the display image.

[0219] In the example of B in Fig. 29, the circular area on the lower left side of the display image is the work concentration area A81. In the example of B in Fig. 29, the selected reference image area A41-2, which is the display area of ​​the reference image selected by the user, is located on the right edge of the display image, symmetrically to the selected reference image area A41-1 in the display image of A in Fig. 29.

[0220] In this way, it is also possible to arrange the information that is arranged so as to be superimposed on the operative field image area A2 at a position determined based on the work concentration area.

[0221] Example of all clear operation The user can control the area configuration of the display image by performing a predetermined operation. For example, the user can display the display image described with reference to Fig. 2 by performing a line of sight, voice, gesture, touch operation, etc. Also, the user can display a display image composed of only the surgical field image area A2 by performing a similar operation.

[0222] Here, a display image made up of only the operative field image area A2 may be displayed by pressing a button displayed on the display image.

[0223] FIG. 30 is a diagram showing an example of an area configuration of a display image in which buttons are displayed.

[0224] The display image in the upper part of FIG. 30 is similar to the display image described with reference to FIG. 2, except for the placement of buttons, and therefore duplicated description will be omitted where appropriate.

[0225] In the example in the upper part of Fig. 30, a button area A91, which is a display area for buttons, is arranged in the lower right corner of the display image as a substantially square area superimposed on the operation menu area A3. By pressing a button displayed in the button area A91, the user can display the display image in the lower part shown by the arrow in Fig. 30.

[0226] In the display image at the bottom of FIG. 30, the operative field image area A2 is disposed over the entire screen, and only the operative field image is displayed.

[0227] As described above, by performing a specified operation, the user can switch from a display image in a certain state to a display image in an initial state (such as the display image in Figure 2 or the display image in B of Figure 30) with a single action.

[0228] Example when "Zoom" is selected When the "Zoom" item is selected from the operation menu, the control unit 82 generates a display image in which the operative field image area A2 is divided.

[0229] FIG. 31 is a diagram showing an example of a display image displayed on the display device 14 when "Zoom" is selected.

[0230] As shown in the upper part of Fig. 31, when "Zoom" is selected, the display image is configured by arranging a presentation information area A1, which is a narrow band-like area, on an operative field image area A2. The operative field image area A2 is configured by rectangular operative field image division areas A101-0 to A101-8.

[0231] The operative field image division area A101-0 is the central area among the nine 3×3 (vertical×horizontal) areas obtained by dividing the operative field image area A2. The operative field image division areas A101-1 and A101-5 are rectangular areas with the same width and height, respectively, located above and below the operative field image division area A101-0. The operative field image division areas A101-2, A101-4, A101-6, and A101-8 are rectangular areas with the same width and height, respectively, located at the corners of the operative field image area A2. The operative field image division areas A101-3 and A101-7 are rectangular areas with the same width and height, respectively, located to the right and left of the operative field image division area A101-0.

[0232] In the example in the upper part of FIG. 31, the sizes of the respective operative field image division regions are not the same. However, it is also possible that the sizes of all the operative field image division regions are the same.

[0233] The operative field image displayed in the operative field image area A2 is divided and displayed in the operative field image divided areas A101-0 to A101-8. Also, in a predetermined position of each of the operative field image divided areas A101-0 to A101-8, a number representing each area and a name of each area are displayed.

[0234] The user can select the divided operative field images displayed in the operative field image division areas A101-0 to A101-8 by line of sight or voice. For example, when the divided operative field image displayed in the operative field image division area A101-0 is selected, the control device 12 generates the display image in the lower part shown by the arrow in FIG.

[0235] In the display image at the bottom of Fig. 31, the operative field image displayed in the operative field image division area A101-0 is enlarged by four times and displayed in the operative field image area A2. The dashed rectangle indicates the area corresponding to the operative field image division area A101-0.

[0236] When a surgical field image to be displayed in the surgical field image division area is selected by the user, the control unit 82 enlarges the surgical field image displayed in the area selected by the user to a predetermined magnification and places it in the surgical field image area A2. The magnification for enlarging the surgical field image may be specified by the user's line of sight or voice after the user selects the surgical field image to be displayed in the surgical field image division area.

[0237] Example of combined gaze and voice control The user can select information to be displayed by combining multiple inputs such as gaze and voice. For example, the user can select a thumbnail image of the reference information group described in the third embodiment by combining gaze and voice.

[0238] FIG. 32 is a diagram showing an example of an operation by a combination of eye gaze and voice.

[0239] 32, the user's line of sight is directed toward the thumbnail image displayed in the reference information group area A11. This line of sight is detected by the control device 12 based on the captured image provided by a camera installed near the display device 14.

[0240] The thumbnail image displayed in the reference information group area A11 at which the gaze is directed is similar to the display image described with reference to Fig. 15. In the example of Fig. 32, the hatched thumbnail images displayed in the reference information group areas A12 to A14 indicate that the thumbnail images are displayed with a predetermined transparency set.

[0241] In this way, the control unit 82 sets the transparency of thumbnail images other than the thumbnail image on which the detected user's gaze is directed among the thumbnail images displayed in the reference information group areas A11 to A14, and generates a display image. In other words, the transparency is not set for the thumbnail image on which the user's gaze is directed and focused, while the transparency is set for thumbnail images other than the thumbnail image.

[0242] After the user confirms that the focus is on the thumbnail image that he or she is looking at, the user can select the thumbnail image that he or she is looking at by saying a specified voice (voice command) such as "select this."

[0243] Also, the focus may be set on a thumbnail image based on the area to which the user's gaze is directed.

[0244] FIG. 33 is a diagram showing an example of a display image in which a thumbnail image is focused on based on the area where the user's gaze is directed.

[0245] In the example at the top of Fig. 33, the user's gaze is directed toward the vicinity of the thumbnail image of the reference information group region A11. At this time, the control unit 82 determines, for example, a rectangular region R3 surrounded by a dashed line as the region toward which the user's gaze is directed, based on the user's gaze. Since the rectangular region R3 surrounded by a dashed line includes only a part of the reference information group region A11 among the reference information group regions A11 to A14, the control unit 82 generates a display image shown at the bottom of the arrow in Fig. 33.

[0246] The display image in the lower part of Fig. 33 is the same as the display image described with reference to Fig. 32. That is, the control unit 82 generates a display image by not setting a transparency to thumbnail images included in the area where the user's gaze is directed, and setting a transparency to thumbnail images other than the thumbnail image. Then, after checking the thumbnail image in focus, the user can select the thumbnail image by using a predetermined voice.

[0247] It should be noted that the user can also select a thumbnail image that is in focus by hearing a predetermined sound without directing his or her gaze at the thumbnail image.

[0248] FIG. 34 is a diagram showing another example of an operation performed by a combination of eye gaze and voice.

[0249] In the example of Fig. 34, the user's gaze is directed to a position approximately in the center of the operative field image area A2. In the display image of Fig. 34, a marking M1, which is a flag icon, is displayed at the position on the operative field image area A2 where the user's gaze is directed. Here, when the user utters a predetermined sound such as "marking" while directing his / her gaze, the control device 12 generates a display image in which the marking M1 is superimposed and placed at the position where the user's gaze is directed.

[0250] Furthermore, the display image displayed after a predetermined voice (voice command) is input may be configured to display the user's gaze position on the screen with a predetermined mark. For example, when the user selects "Move" shown in Fig. 4 by voice input, the control device 12 displays a mark on the screen so as to follow the detected user's gaze position. At this time, the control device 12 may change the display range of the surgical field image so as to follow the position of the mark so that the position of the mark overlaps with the center position of the image.

[0251] Furthermore, the presented information displayed in the presented information area A1 (header area) may include time information indicating the total surgery time counted from the start of the surgery, as shown in Fig. 35. The presented information may also include alert information, a set value of the insufflation device, a set value of the electric scalpel, and information indicating a timer. As information indicating the timer, for example, it is preferable to display the set time and the count-up time, and to change the color of the text when the count-up time reaches the set time. Furthermore, the control device 12 may display the content of the alert as alert information.

[0252] The operation menu displayed in the operation menu area A3 (footer area) may be the operation menu shown in FIG. 36. In the example of FIG. 36, the texts "Camera", "Reference", "Timer Start", "Screen" and "Others" are displayed, which indicate optional items selectable by the user. "Camera" is an item for causing the control device 12 to control the medical imaging device capturing the operative field image or control the display of the operative field image. "Timer Start" is an item for causing the control device 12 to start counting the count-up timer. "Reference" is an item similar to "Reference" in the above embodiment. "Screen" is an item for causing the control device 12 to switch the display state. The switching of the display state is, for example, switching between display and non-display of the user interface, and switching between ON and OFF of the full-screen display of the reference image. "Others" is an item for causing the control device 12 to perform other operations, such as displaying a submenu. The control device 12 may display "Cancel", which is an item for causing the control device 12 to cancel the content instructed by the user to the control device 12.

[0253] Also, when "Camera" is selected, the items of the operation menu displayed in the operation menu area A3 are changed to items for controlling the medical imaging device or the display of the operative field image. In the example of FIG. 37, the texts "Zoom In", "Zoom Out", "Zoom Reset", "Center", "Move" and "Cancel" are displayed, which represent option items selectable by the user. "Zoom In" is an item for causing the control device 12 to control the enlargement magnification of the operative field image to be increased. "Zoom Out" is an item for causing the control device 12 to control the enlargement magnification of the operative field image to be decreased. "Zoom Reset" is an item for causing the control device 12 to control the enlargement magnification of the operative field image to be returned to a specified value, and "Center" is an item for causing the control device 12 to control the center of the operative field image to be aligned with the center of the display device. "Move" is an item similar to "Move" in the above-mentioned embodiment. "Cancel" is an item for causing the control device 12 to cancel the contents of the previous instruction given to the control device 12 by the user.

[0254] When "Zoom In" or "Zoom Out" is selected by the user, it is preferable to detect the user's line of sight at the time of selection and change the display area of ​​the surgical field image so that the line of sight position is at the center.

[0255] When displaying the reference images, the control device 12 may display operation items in the lower regions of the selected reference image regions A41 and A42 for causing the display device 12 to control the display of the reference images. As shown in A of FIG. 38, when a plurality of reference images are displayed, it is preferable to display operation items in the lower regions of the selected reference image regions in which the respective reference images are displayed. As shown in B of FIG. 38, the operation items are, for example, "Enlarge" for controlling the display of an image to be enlarged, and "Close" for canceling the display of the reference image. Note that the control device 12 may display, as operation items, "Full Size" for displaying the reference image in the entire display region, and "Half Size" for displaying the reference image in half the display region, in the lower regions of the selected reference image regions. Also, when a plurality of reference images are displayed, a reference image for controlling the display may be selected based on the user's line of sight. Also, the control device 12 may display the lower region of the reference image (selected reference image) whose display region overlaps with the detected user's line of sight position darker than the lower regions of the other reference images (non-selected reference images). This allows the control device 12 to present the reference image to be operated by voice control to the user.

[0256] 39, the control device 12 may be configured to divide the operative field image area A2 into two areas in the horizontal direction in response to a user's instruction, and display the operative field image in one area and a reference image or an operative field image enlarged at a predetermined magnification in the other area. Note that a "Half size" item may be added to the operation menu items as an item for the user to instruct the control device 12 to divide the display area. Also, when the user selects "Half Size," a "Cancel Half size" item may be added to the operation menu items as an item for causing the control device 12 to cancel the instruction to divide the area.

[0257] It is possible to arrange any image at the position where the user's gaze is directed. The image to be arranged may be determined based on the user's voice. Here, as the gaze area corresponding process related to the gaze area according to the user's gaze, a process of selecting a selection target according to the user's voice when a selection target exists within the gaze area, a process of marking a position indicated by the gaze area according to the user's voice, or a process of focusing on a selection target within the gaze area are exemplified, but other processes may also be performed.

[0258] ·others The program executed by the control device 12 may be a program that is processed chronologically in the order described in this specification, or it may be a program that is processed in parallel or at the required timing, such as when called.

[0259] The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0260] The program to be installed is provided by being recorded on a removable recording medium 61 shown in FIG. 7, which is made up of an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. The program may also be provided via a wired or wireless transmission medium, such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in ROM 52 or recording unit 58.

[0261] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0262] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0263] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0264] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0265] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0266] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0267] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0268] <Examples of configuration combinations> The present technology can also be configured as follows.

[0269] (1) An acquisition unit that acquires an operative field image signal generated by a medical imaging device that images an operative field of a patient and a presentation information signal generated by an apparatus including an external device other than the medical imaging device; a control unit that generates a display image based on the surgical field image signal and the presentation information signal; A control device having The control unit generates the display image so as to arrange, in an upper region of a display region of the display image, presentation information generated from the presentation information signal, in a middle region having a length equal to that of the upper region in the horizontal direction, an operative field image generated based on the operative field image signal, and in a region below the upper region, user interface information for controlling the control device. Medical display system. (2) The presented information includes at least monitoring information generated from a monitoring information signal acquired from a monitoring device that monitors biological information of the patient. The medical display system according to (1) above. (3) The control unit arranges the presented information in a predetermined display form including text or a graph. The medical display system according to (2) above. (4) The presented information includes information regarding a surgical instrument. The medical display system according to (2) or (3). (5) The presentation information includes information related to time. A medical display system according to any one of (2) to (4). (6) The presentation information includes information that changes in real time. A medical display system according to any one of (2) to (5). (7) The operative field image is an image that changes in real time. A medical display system according to any one of (1) to (6). (8) a region below the upper region including a lower region having a length equal to that of the upper region in a horizontal direction; The control unit arranges the user interface information in the lower area. A medical display system according to any one of (1) to (7). (9) a region below the upper region has a horizontal length different from that of the upper region, and includes left and right regions corresponding to at least one of the left and right ends of the central region; The control unit arranges the user interface information in the left and right regions. A medical display system according to any one of (1) to (7). (10) The control unit changes a display method of the presentation information arranged in the upper area based on information about a user. A medical display system according to any one of (1) to (9). (11) The information about the user includes information about the user's line of sight, or information according to an operation or action by the user. The medical display system according to (10) above. (12) The control unit arranges one or more captured images generated by a medical imaging device other than the medical imaging device in a region below the upper region. A medical display system according to any one of (1) to (11) above. (13) The control unit is A thumbnail image corresponding to a reference image, which is a photographed image for reference, is arranged; When the thumbnail image is selected, a plurality of reference images corresponding to the selected thumbnail image are arranged in a predetermined direction; When a desired reference image is selected from the plurality of reference images, an enlarged version of the selected reference image is arranged. The medical display system according to (12) above. (14) The control unit arranges a direction indicator for changing a display range of the operative field image in a region below the upper region. A medical display system according to any one of (1) to (11) above. (15) The control unit is Identifying a task concentration area in the operative field image where the user focuses his or her gaze; A position for placing information to be superimposed on the surgical field image is determined based on the identified work concentration area. A medical display system according to any one of (1) to (14). (16) The control unit determines the placement position so that an alert or reference information is placed in an area other than the work concentration area. The medical display system according to (15) above. (17) The control unit is Detecting a user's gaze toward the central region; Execute gaze area corresponding processing for the gaze area corresponding to the detected gaze A medical display system according to any one of (1) to (16). (18) The gaze area corresponding process includes a process of selecting a selection target in response to a voice of the user when the selection target is present within the gaze area, a process of marking a position indicated by the gaze area in response to the voice of the user, or a process of focusing on the selection target within the gaze area. The medical display system according to (17) above. (19) An acquisition unit that acquires an operative field image signal generated by a medical imaging device that images an operative field of a patient and a presentation information signal generated by an apparatus including an external device other than the medical imaging device; a control unit that generates a display image based on the operative field image signal and the presentation information signal, The control unit generates the display image so as to arrange, in an upper region of a display region of the display image, presentation information generated from the presentation information signal, arrange an operative field image generated based on the operative field image signal in a middle region having a length equal to that of the upper region in the horizontal direction, and arrange user interface information for controlling a control device in a region below the upper region. Control device. (20) The control device, Acquire an operative field image signal generated by a medical imaging device that images an operative field of a patient, and a presentation information signal generated by an apparatus including an external device excluding the medical imaging device; generating a display image based on the operative field image signal and the presentation information signal; The display image is generated so that the presentation information generated from the presentation information signal is arranged in an upper region of the display region of the display image, the surgical field image generated based on the surgical field image signal is arranged in a middle region having a length equal to that of the upper region in the horizontal direction, and user interface information for controlling a control device is arranged in a region below the upper region. Control methods. [Explanation of symbols]

[0270] 1 Medical display system, 11 Microscope device, 12 Control device, 13 Monitoring device, 14 Display device, 15 Network, 16 External server, 81 Acquisition unit, 82 Control unit, 83 Display control unit

Claims

1. An acquisition unit that acquires an operative field image signal generated by a medical imaging device that images an operative field of a patient and a presentation information signal generated by an apparatus including an external device other than the medical imaging device; a control unit that generates a display image based on the surgical field image signal and the presentation information signal; A control device having The control unit is presentation information generated from the presentation information signal is arranged in an upper region of a display region of the display image; A surgical field image generated based on the surgical field image signal is arranged in a middle region having a length equal to that of the upper region in a horizontal direction and overlapping the upper region; user interface information for controlling the control device is arranged in a region below the upper region; When the user's gaze position is not in the upper region, a predetermined transmittance is set for the presentation information arranged in the upper region with respect to the surgical field image arranged in the middle region, and the display image is generated. Medical display system.

2. The presented information includes at least monitoring information generated from a monitoring information signal acquired from a monitoring device that monitors biological information of the patient. The medical display system of claim 1 .

3. The control unit arranges the presented information in a predetermined display form including text or a graph. The medical display system of claim 2 .

4. The presented information includes information regarding a surgical instrument. The medical display system of claim 2 .

5. The presentation information includes information related to time. The medical display system of claim 2 .

6. The presentation information includes information that changes in real time. The medical display system of claim 2 .

7. The operative field image is an image that changes in real time. The medical display system of claim 1 .

8. a region below the upper region including a lower region having a length equal to that of the upper region in a horizontal direction; The control unit arranges the user interface information in the lower area. The medical display system of claim 1 .

9. a region below the upper region has a horizontal length different from that of the upper region, and includes left and right regions corresponding to at least one of the left and right ends of the central region; The control unit arranges the user interface information in the left and right regions. The medical display system of claim 1 .

10. The control unit changes a display method of the presentation information arranged in the upper area based on information corresponding to an operation or action by the user. The medical display system of claim 1 .

11. The control unit arranges one or more captured images generated by a medical imaging device other than the medical imaging device in a region below the upper region. The medical display system of claim 1 .

12. The control unit is A thumbnail image corresponding to a reference image, which is a photographed image for reference, is arranged; When the thumbnail image is selected, a plurality of reference images corresponding to the selected thumbnail image are arranged in a predetermined direction; When a desired reference image is selected from the plurality of reference images, an enlarged version of the selected reference image is arranged. The medical display system of claim 11.

13. The control unit arranges a direction indicator for changing a display range of the operative field image in a region below the upper region. The medical display system of claim 1 .

14. The control unit is Identifying a task concentration area in the operative field image where the user focuses his / her gaze; A position for placing information to be superimposed on the surgical field image is determined based on the identified work concentration area. The medical display system of claim 1 .

15. The control unit determines the placement position so that an alert or reference information is placed in an area other than the work concentration area.

15. The medical display system of claim 14.

16. The control unit is Detecting a line of sight of the user with respect to the central region; Execute gaze area corresponding processing for the gaze area corresponding to the detected gaze The medical display system of claim 1 .

17. The gaze area corresponding process includes a process of selecting a selection target in response to a voice of the user when the selection target is present within the gaze area, a process of marking a position indicated by the gaze area in response to the voice of the user, or a process of focusing on the selection target within the gaze area.

17. The medical display system of claim 16.

18. An acquisition unit that acquires an operative field image signal generated by a medical imaging device that images an operative field of a patient and a presentation information signal generated by an apparatus including an external device other than the medical imaging device; a control unit that generates a display image based on the operative field image signal and the presentation information signal, The control unit is presentation information generated from the presentation information signal is arranged in an upper region of a display region of the display image; A surgical field image generated based on the surgical field image signal is arranged in a middle region having a length equal to that of the upper region in a horizontal direction and overlapping the upper region; user interface information for controlling the device is arranged in an area below the upper area; When the user's gaze position is not in the upper region, a predetermined transmittance is set for the presentation information arranged in the upper region with respect to the surgical field image arranged in the middle region, and the display image is generated. Control device.

19. The control device, Acquiring an operative field image signal generated by a medical imaging device that images an operative field of a patient and a presentation information signal generated by an apparatus including an external device other than the medical imaging device; generating a display image based on the surgical field image signal and the presentation information signal; Including, generating the display image Arranging presentation information generated from the presentation information signal in an upper region of a display region of the display image; Arranging a surgical field image generated based on the surgical field image signal in a middle region having a length equal to that of the upper region in a horizontal direction and overlapping the upper region; Arranging user interface information for controlling the control device in an area below the upper area; When the gaze position of the user is not in the upper region, a predetermined transmittance is set for the presented information arranged in the upper region with respect to the surgical field image arranged in the middle region. Includes Control methods.

Images