Image processing device, image processing system, image display method, and image processing program

Abstract

To smooth communication between an operator and a manipulator.SOLUTION: An image processing device displays, in a first display, a manipulator screen including a first region for displaying a surgery support image including a biological tissue image that has put a structure of a biological tissue in the body of a surgery object person into an image on the basis of information acquired by sensing, and a second region for displaying at least one operation element for controlling the display of the surgery support image in the first region; displays, in a second display different from the first display, a browser screen for displaying the surgery support image matched with the display of the surgery support image in the first region; receives user operation performed on the manipulator screen using input equipment; and reflects the display of the user operation on the browser screen when the user operation is performed in the first region, and does not reflect the display of the user operation on the browser screen when the user operation is performed in the second region.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an image processing system, an image display method, and an image processing program.

Background Art

[0002] Patent Document 1 discloses an image diagnostic apparatus that displays image information of a subject on a monitor for an operator and a monitor for a technician, and displays image information independent of the image information displayed on the monitor for the operator on the monitor for the technician.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Monitor screen sizes come in various types such as HD, NTSC, WXGA, and CinemaScope. "HD" is an abbreviation for high definition. "NTSC" is an abbreviation for National Television System Committee. "WXGA" is an abbreviation for Wide Extended Graphics Array. The screen ratio of HD is 16 in width and 9 in height, that is, 16:9. The screen ratio of NTSC is 4 in width and 3 in height, that is, 4:3. The screen ratio of WXGA is 16 in width and 10 in height, that is, 16:10. The screen ratio of CinemaScope is 2.35 in width and 1 in height, that is, 2.35:1.

[0005] In rooms such as angiography rooms where operators perform catheter treatments, a 4:3 screen size is often adopted. On the other hand, in rooms where operators perform software operations, it is desirable to adopt a wider screen size such as 16:9, which is suitable for menu operations and detailed settings. However, when a wider screen size than the operator's monitor is adopted for the operator's monitor, since the drawing size changes, when the operator's screen is displayed on the operator's monitor, there is a problem that the vertically compressed screen is displayed. As a countermeasure against such a problem, it is conceivable to display only important information for the operator on the operator's monitor.

[0006] When the operator's screen is displayed on the operator's monitor, the operator can smoothly communicate with the operator while checking what operations are being performed on the operator's screen. On the other hand, when only clinically important information is displayed on the operator's monitor, the operator cannot check what operations are being performed on the operator's screen, and there is a risk that it will be difficult to smoothly communicate with the operator.

[0007] An object of the present disclosure is to smooth communication between the operator and the operator.

Means for Solving the Problem

[0008] An image processing apparatus according to one aspect of the present disclosure includes an operator screen including a first region for displaying a surgical support image including a biological tissue image obtained by imaging based on information obtained by sensing the structure of a biological tissue inside a surgical subject, and a second region for displaying at least one operation element for controlling the display of the surgical support image in the first region on a first display, and a viewer screen for displaying the surgical support image in accordance with the display of the surgical support image in the first region on a second display different from the first display, the image processing apparatus receiving a user operation performed on the operator screen using an input device, and when the user operation is performed in the first region, reflecting the display of the user operation on the viewer screen, and when the user operation is performed in the second region, including a control unit that does not reflect the display of the user operation on the viewer screen.

[0009] As one embodiment, when the user operation is performed in the second region, the control unit displays notification information indicating that the user operation is performed in the second region on the viewer screen.

[0010] As one embodiment, the control unit displays a setting menu including two or more setting items on the operator screen, and when receiving an operation of selecting any one of the two or more setting items from the setting menu using the input device, displays one or more operation elements corresponding to the selected setting item in the second region, and when receiving an operation of the one or more operation elements as the user operation, displays information indicating the selected setting item as the notification information on the viewer screen.

[0011] As one embodiment, as the at least one operation element, the control unit displays two or more operation elements in the second region, and when receiving an operation of any one of the two or more operation elements as the user operation, displays information indicating the operation element being operated as the notification information on the viewer screen.

[0012] As an embodiment, the input device is a pointing device, and the control unit displays a first pointer movable by the user operation on the operator screen. When the first pointer is in the first area, the control unit reflects the display of the user operation on the viewer screen by displaying a second pointer that moves in accordance with the movement of the first pointer on the viewer screen. When the first pointer is in the second area, the control unit stops reflecting the display of the user operation on the viewer screen by making the second pointer invisible on the viewer screen.

[0013] As an embodiment, even when the first pointer is in the first area, if the control unit determines that the position of the first pointer has not changed for a certain period of time, the control unit makes the second pointer invisible on the viewer screen.

[0014] As an embodiment, the input device is a pointing device, and the control unit displays a first pointer movable by the user operation on the operator screen. When the first pointer is in the first area, the control unit reflects the display of the user operation on the viewer screen by displaying a second pointer that moves in accordance with the movement of the first pointer on the viewer screen. When the first pointer is in the second area, the control unit stops reflecting the display of the user operation on the viewer screen by displaying the second pointer on the viewer screen with its position fixed.

[0015] As an embodiment, even when the first pointer is in the first area, if the control unit determines that the position of the first pointer has not changed for a certain period of time, the control unit makes the second pointer invisible on the viewer screen.

[0016] As one embodiment, the second region includes a floating region that can be moved using the input device and is allowed to overlap the first region, and when the floating region overlaps at least a part of the surgical support image on the operator screen, the control unit hides at least the part of the surgical support image on the operator screen and continues to display at least the part of the surgical support image on the viewer screen.

[0017] As one embodiment, the control unit separately renders the image displayed in the first region as the surgical support image and the image displayed on the viewer screen based on the same information obtained by the sensing.

[0018] An image processing system according to one aspect of the present disclosure includes the image processing device, the first display, and the second display.

[0019] As one embodiment, the second display is installed in the radiation room where the surgery is performed, and the first display is installed in a separate room where operations using the input device are performed.

[0020] An image display method according to one aspect of the present disclosure is an image display method in which an operator screen including a first region for displaying a surgical support image including a biological tissue image obtained by imaging information obtained by sensing the structure of a biological tissue in a subject's body and a second region for displaying at least one operation element for controlling the display of the surgical support image in the first region is displayed on a first display, and a viewer screen for displaying the surgical support image in accordance with the display of the surgical support image in the first region is displayed on a second display different from the first display, and a user operation performed on the operator screen using an input device is received. When the user operation is performed in the first region, the display of the user operation is reflected on the viewer screen, and when the user operation is performed in the second region, the display of the user operation is not reflected on the viewer screen.

[0021] As one embodiment, when the user operation is performed in the second region, the image display method displays notification information indicating that the user operation is being performed in the second region on the viewer screen.

[0022] An image processing program according to an aspect of the present disclosure causes a first display to display an operator screen including a first region for displaying a surgical support image including a biological tissue image imaged based on information obtained by sensing the structure of a biological tissue in a subject to be operated on, and a second region for displaying at least one operation element for controlling the display of the surgical support image in the first region, and causes a computer different from the first display to display a viewer screen for displaying the surgical support image in accordance with the display of the surgical support image in the first region. The computer receives a user operation performed on the operator screen using an input device, and when the user operation is performed in the first region, reflects the display of the user operation on the viewer screen, and when the user operation is performed in the second region, does not reflect the display of the user operation on the viewer screen.

Advantages of the Invention

[0023] According to the present disclosure, communication between a surgeon and an operator can be made smooth.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0025] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0026] In each figure, the same or corresponding parts are denoted by the same reference numerals. In the description of the present embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0027] With reference to FIG. 1, the configuration of the image processing system 10 according to the present embodiment will be described.

[0028] The image processing system 10 includes an image processing apparatus 20, a first display 30, a second display 40 different from the first display 30, and an input device 50. The image processing apparatus 20 is connected to the first display 30, the second display 40, and the input device 50 via a cable or a network, or wirelessly.

[0029] The image processing apparatus 20 is, for example, a general-purpose computer such as a PC, a server computer such as a cloud server, or a dedicated computer. "PC" is an abbreviation for personal computer. The image processing apparatus 20 may be installed in a medical facility such as a hospital, or may be installed in a facility other than a medical facility, such as a data center.

[0030] The first display 30 and the second display 40 are, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescent. The screen size of the first display 30 may be any screen size, but in this embodiment it is NTSC. That is, the screen ratio of the first display 30 may be any screen ratio, but in this embodiment it is 4:3. The screen size of the second display 40 may be any screen size, but in this embodiment it is HD. That is, the screen ratio of the second display 40 may be any screen ratio, but in this embodiment it is 16:9. The input device 50 is, for example, a pointing device such as a mouse, a keyboard, or a touch screen provided integrally with the first display 30. As shown in FIG. 2, the second display 40 is installed in the radiation room 11 of the medical facility. The first display 30 and the input device 50 are installed in a separate room 12 of the medical facility. The radiation room 11 is visible from the separate room 12 through the window 13. In the radiation room 11, an operation such as a surgery is performed by an operator 14 such as a doctor. In the separate room 12, an operation using the input device 50 is performed by an operator 15 such as a clinical engineer. Surgical support images 16 such as IVUS images, OFDI images, OCT images, CT images, MRI images, echo images, or X-ray images are displayed on the first display 30 and the second display 40. "IVUS" is an abbreviation for intravascular ultrasound. "OFDI" is an abbreviation for optical frequency domain imaging. "OCT" is an abbreviation for optical coherence tomography. "CT" is an abbreviation for computed tomography. "MRI" is an abbreviation for magnetic resonance imaging.

[0031] With reference to FIGS. 3 to 10, the outline of this embodiment will be described.

[0032] The image processing apparatus 20 causes the first display 30 to display an operator screen 31 as shown in FIGS. 3, 5, 7, and 9, and causes the second display 40 to display a viewer screen 41 as shown in FIGS. 4, 6, 8, and 10. The operator screen 31 includes a first area 32 that displays the surgical support image 16, and a second area 33 that displays at least one operation element for controlling the display of the surgical support image 16 in the first area 32. The surgical support image 16 includes a biological tissue image 17 formed by imaging information obtained by sensing the structure of the biological tissue in the body of the surgical subject. The viewer screen 41 displays the surgical support image 16 in accordance with the display of the surgical support image 16 in the first area 32. The image processing apparatus 20 receives a user operation performed on the operator screen 31 using the input device 50. When the user operation is performed in the first area 32, the image processing apparatus 20 reflects the display of the user operation on the viewer screen 41. When the user operation is performed in the second area 33, the image processing apparatus 20 does not reflect the display of the user operation on the viewer screen 41.

[0033] In this embodiment, while the surgical support image 16 displayed in the first area 32 of the operator 15 side screen is displayed on the monitor on the surgeon 14 side, the operation elements displayed in the second area 33 of the operator 15 side screen can be made non-displayed. Here, the surgical support image 16 is important information for the surgeon 14, but the operation elements are unnecessary information for the surgeon 14. That is, according to this embodiment, only clinically important information can be displayed on the monitor on the surgeon 14 side.

[0034] In this embodiment, when the user operation is performed in the first area 32, the display of the user operation is reflected on the monitor on the surgeon 14 side, but when the user operation is performed in the second area 33, the display of the user operation is not reflected on the monitor on the surgeon 14 side. Therefore, the surgeon 14 can confirm at least what operations are being performed in the first area 32 of the operator 15 side screen. As a result, the surgeon 14 can communicate smoothly with the operator 15. That is, according to this embodiment, the communication between the surgeon 14 and the operator 15 can be made smooth.

[0035] When the input device 50 is a pointing device, the image processing apparatus 20 displays a first pointer 51 that can be moved by a user operation on the operator screen 31. When the first pointer 51 is in the first area 32, the image processing apparatus 20 reflects the display of the user operation on the viewer screen 41 by displaying a second pointer 52 that moves in accordance with the movement of the first pointer 51 on the viewer screen 41. When the first pointer 51 is in the second area 33, the image processing apparatus 20 stops reflecting the display of the user operation on the viewer screen 41 by making the second pointer 52 invisible on the viewer screen 41. In the example of FIG. 3, the first pointer 51 is in the second area 33. Therefore, in the example of FIG. 4 corresponding to the example of FIG. 3, the second pointer 52 is invisible on the viewer screen 41. On the other hand, in the example of FIG. 5, the first pointer 51 is in the first area 32. Therefore, in the example of FIG. 6 corresponding to the example of FIG. 5, the second pointer 52 is displayed on the viewer screen 41 so as to move in accordance with the movement of the first pointer 51.

[0036] When the first pointer 51 is in the second area 33, instead of making the second pointer 52 invisible, the image processing apparatus 20 may stop reflecting the display of the user operation on the viewer screen 41 by displaying the second pointer 52 on the viewer screen 41 with a fixed position. In the example of FIG. 7, the first pointer 51 is in the second area 33. Therefore, in the example of FIG. 8 corresponding to the example of FIG. 7, the second pointer 52 is displayed on the viewer screen 41 with a fixed position on the viewer screen 41. The second pointer 52 is fixed at a position corresponding to the position immediately before the first pointer 51 exits the first area 32 on the viewer screen 41. Thus, the operator 14 can confirm at least in which direction the first pointer 51 has been moved on the screen on the operator 15 side. For example, if the operator 14 knows that the second area 33 is adjacent to the right side of the first area 32 on the operator screen 31 and the setting menu 35 is arranged below the second area 33, the operator 14 can infer that the operator 15 is operating the setting menu 35 from the fact that the second pointer 52 is in the lower right corner of the viewer screen 41.

[0037] In the second area 33, any operation element may be displayed. In this embodiment, however, the 2D button 34a, the 2D / 3D button 34b, the enlargement button 34c, the reduction button 34d, and the rotation button 34e are displayed.

[0038] The 2D button 34a is an operation element for selecting a mode in which a two-dimensional image is displayed in the first area 32 as the surgical support image 16. For example, by clicking the 2D button 34a, an IVUS image can be displayed in the first area 32 as the surgical support image 16. The IVUS image is a two-dimensional image obtained by transmitting ultrasonic waves from an ultrasonic vibrator provided at the tip of a catheter inserted into the body of the surgical subject as a biological tissue image 17, receiving reflected waves, and processing the signals of the reflected waves, and imaging the cross-sectional structure of biological tissues such as the blood vessels or heart of the surgical subject.

[0039] The 2D / 3D button 34b is an operation element for selecting a mode in which a two-dimensional image and a three-dimensional image are displayed in the first area 32 as the surgical support image 16. For example, by clicking the 2D / 3D button 34b, the latest IVUS image and a three-dimensional image obtained by stacking a plurality of IVUS images can be displayed in the first area 32 as the surgical support image 16. This three-dimensional image is a three-dimensional image obtained by sequentially imaging the cross-sectional structure of the biological tissue of the surgical subject while moving the ultrasonic vibrator by a pullback operation to generate a plurality of two-dimensional images and stacking these plurality of two-dimensional images as a biological tissue image 17.

[0040] The enlargement button 34c is an operation element for enlarging and displaying the surgical support image 16 in the first area 32. For example, by clicking the enlargement button 34c once, the IVUS image displayed in the first area 32 as the surgical support image 16 can be enlarged and displayed.

[0041] The reduction button 34d is an operation element for reducing and displaying the surgical support image 16 in the first region 32. For example, by clicking the reduction button 34d once, the IVUS image displayed in the first region 32 as the surgical support image 16 can be reduced and displayed.

[0042] The rotation button 34e is an operation element for rotating the surgical support image 16 in the first region 32. For example, by clicking the rotation button 34e once, the IVUS image displayed in the first region 32 as the surgical support image 16 can be rotated 90 degrees clockwise.

[0043] When a user operation is being performed in the second region 33, the image processing apparatus 20 displays notification information 42 indicating that the user operation is being performed in the second region 33 on the viewer screen 41. In the example of FIG. 3, when the IVUS image is displayed in the first region 32 as the surgical support image 16, the rotation button 34e is clicked in the second region 33. Therefore, in the example of FIG. 4 corresponding to the example of FIG. 3, an icon indicating that the rotation button 34e is clicked in the second region 33 is displayed on the viewer screen 41 as the notification information 42. In the example of FIG. 9, when the IVUS image is displayed in the first region 32 as the surgical support image 16, a calibration operation is being performed in the floating region 37 included in the second region 33. Therefore, in the example of FIG. 10 corresponding to the example of FIG. 9, a label indicating that the calibration operation is being performed in the second region 33 is highlighted on the viewer screen 41 as the notification information 42.

[0044] The image processing apparatus 20 displays the setting menu 35 on the operator screen 31. In the present embodiment, the image processing apparatus 20 displays the setting menu 35 only in the second area 33 and does not display it on the viewer screen 41. As a modification of the present embodiment, the image processing apparatus 20 may display the setting menu 35 not only in the first area 32 but also on the viewer screen 41. The setting menu 35 includes two or more setting items 36. When the image processing apparatus 20 receives an operation of selecting any one of two or more setting items 36 from the setting menu 35 using the input device 50, one or more operation elements corresponding to the selected setting item are displayed in the second area 33. When the image processing apparatus 20 receives an operation of the one or more operation elements as a user operation, information indicating the selected setting item is displayed on the viewer screen 41 as notification information 42. In the example of FIG. 5, the setting menu 35 is closed. In the example of FIG. 7, when the setting menu 35 is opened and two or more setting items 36 are displayed, the setting item corresponding to the calibration operation is clicked. Therefore, in the example of FIG. 9, a window for performing the calibration operation is displayed in the floating area 37 included in the second area 33 as an operation element corresponding to the clicked setting item. In the example of FIG. 9, an operation of the window displayed in the floating area 37 is being performed. Therefore, in the example of FIG. 10 corresponding to the example of FIG. 9, a label indicating that the calibration operation has been selected is highlighted on the viewer screen 41 as notification information 42.

[0045] The image processing apparatus 20 displays two or more operation elements in the second area 33. When the image processing apparatus 20 receives an operation of any one of the two or more operation elements as a user operation, information indicating the operated operation element is displayed on the viewer screen 41 as notification information 42. In the example of FIG. 3, when the enlargement button 34c, the reduction button 34d, and the rotation button 34e are displayed in the second area 33, the rotation button 34e is clicked. Therefore, in the example of FIG. 4 corresponding to the example of FIG. 3, an icon indicating that the rotation button 34e has been clicked is displayed on the viewer screen 41 as notification information 42.

[0046] The floating area 37 is an area that can be moved using the input device 50 and is allowed to overlap with the first area 32. When the floating area 37 overlaps at least a part of the surgical support image 16 on the operator screen 31, the image processing apparatus 20 hides at least that part of the surgical support image 16 on the operator screen 31 and continues to display at least that part of the surgical support image 16 on the viewer screen 41. In the example of FIG. 9, the floating area 37 overlaps at least a part of the surgical support image 16 on the operator screen 31. Therefore, in the example of FIG. 9, at least that part of the surgical support image 16 is hidden on the operator screen 31, but in the example of FIG. 10 corresponding to the example of FIG. 9, at least that part of the surgical support image 16 is displayed on the viewer screen 41.

[0047] Referring to FIG. 11, the configuration of the image processing apparatus 20 according to the present embodiment will be described.

[0048] The image processing apparatus 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, and an output unit 25.

[0049] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. While controlling each part of the image processing system 10 including the image processing apparatus 20, the control unit 21 executes processing related to the operation of the image processing apparatus 20.

[0050] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. The magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 22 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used for the operation of the image processing apparatus 20 and data obtained by the operation of the image processing apparatus 20.

[0051] The communication unit 23 includes at least one communication interface. The communication interface is, for example, an interface corresponding to a wireless LAN communication standard such as IEEE802.11, or an interface corresponding to a wired LAN communication standard such as Ethernet (registered trademark). "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. The communication unit 23 receives data used for the operation of the image processing apparatus 20 and transmits data obtained by the operation of the image processing apparatus 20.

[0052] The input unit 24 includes at least one input interface. The input interface is an interface corresponding to a short-range wireless communication standard such as, for example, a USB interface, an HDMI (registered trademark) interface, or Bluetooth (registered trademark). "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface. The input unit 24 receives an operation of inputting data used for the operation of the image processing apparatus 20. The input unit 24 is connected to the input device 50.

[0053] The output unit 25 includes at least one output interface. The output interface is an interface corresponding to a short-range wireless communication standard such as, for example, a USB interface, an HDMI (registered trademark) interface, or Bluetooth (registered trademark). The output unit 25 outputs data obtained by the operation of the image processing apparatus 20. The output unit 25 is connected to the first display 30 and the second display 40.

[0054] The functions of the image processing apparatus 20 are realized by executing the image processing program according to the present embodiment on a processor as the control unit 21. That is, the functions of the image processing apparatus 20 are realized by software. The image processing program causes a computer to execute the operation of the image processing apparatus 20, thereby causing the computer to function as the image processing apparatus 20. That is, the computer functions as the image processing apparatus 20 by executing the operation of the image processing apparatus 20 according to the image processing program.

[0055] The program can be stored in a non-transitory computer-readable medium. The non-transitory computer-readable medium is, for example, a flash memory, a magnetic recording device, an optical disc, a magneto-optical recording medium, or a ROM. The distribution of the program is performed, for example, by selling, transferring, or lending a portable medium such as an SD card, a DVD, or a CD-ROM that stores the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be stored in the server's storage and distributed by transferring the program from the server to other computers. The program may be provided as a program product.

[0056] The computer temporarily stores, for example, a program stored in a portable medium or a program transferred from a server in the main memory device. Then, the computer reads the program stored in the main memory device with the processor and executes the processing according to the read program with the processor. The computer may directly read the program from the portable medium and execute the processing according to the program. Each time a program is transferred from the server to the computer, the computer may sequentially execute the processing according to the received program. The processing may be executed by a so-called ASP-type service that realizes the function only by execution instructions and result acquisition without transferring the program from the server to the computer. "ASP" is an abbreviation for application service provider. The program includes information for use in processing by an electronic computer that conforms to the program. For example, data that is not a direct instruction to the computer but has the property of defining the processing of the computer corresponds to "what conforms to the program".

[0057] Some or all of the functions of the image processing apparatus 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. That is, some or all of the functions of the image processing apparatus 20 may be realized by hardware.

[0058] Referring to FIG. 12, the operation of the image processing apparatus 20 according to the present embodiment will be described. This operation corresponds to the image display method according to the present embodiment.

[0059] In step S1, the control unit 21 receives a user operation performed on the operator screen 31 using the input device 50. The operator screen 31 is displayed on the first display 30. In the first area 32 of the operator screen 31, a surgical support image 16 including a biological tissue image 17 is displayed. In the second area 33 of the operator screen 31, at least one operation element for controlling the display of the surgical support image 16 in the first area 32 is displayed.

[0060] Specifically, the control unit 21 receives a user operation via the input unit 24. In the example of FIG. 3, as a user operation, the control unit 21 uses a mouse to move the first pointer 51 to the rotation button 34e among two or more operation elements displayed in the second area 33, and receives an operation of clicking the rotation button 34e with the mouse. In the example of FIG. 5, as a user operation, the control unit 21 uses a mouse to move the first pointer 51 on the surgical support image 16 displayed in the first area 32, and receives an operation of clicking an arbitrary position of the surgical support image 16 with the mouse for marking. In the example of FIG. 7, the control unit 21 displays a setting menu 35 including two or more setting items 36 on the operator screen 31 via the output unit 25. As an operation of selecting any one of two or more setting items 36 from the setting menu 35 using the input device 50, the control unit 21 uses a mouse to move the first pointer 51 to the setting menu 35 displayed in the second area 33 to open the setting menu 35, and receives an operation of clicking with the mouse on the setting item corresponding to the calibration operation among two or more setting items 36. The control unit 21 displays, in the floating area 37 of the second area 33, a window including a GUI component group for performing a calibration operation as one or more operation elements corresponding to the selected setting item. "GUI" is an abbreviation for graphical user interface. In the example of FIG. 9, as a user operation, the control unit 21 uses a mouse to move the first pointer 51 to the floating area 37 of the second area 33, and receives a calibration operation performed by operating the GUI component group in the window displayed in the floating area 37 with the mouse. Since the floating area 37 overlaps at least a part of the surgical support image 16 on the operator screen 31, the control unit 21 makes the at least a part of the surgical support image 16 non-displayed on the operator screen 31.

[0061] When the user operation is performed in the first area 32, the process of step S2 is executed. When the user operation is performed in the second area 33, the processes of step S3 and step S4 are executed.

[0062] In step S2, the control unit 21 reflects the display of the user operation on the viewer screen 41. The viewer screen 41 is displayed on the second display 40. On the viewer screen 41, the surgical support image 16 is displayed in accordance with the display of the surgical support image 16 in the first region 32.

[0063] Specifically, the control unit 21 reflects the display of the user operation on the viewer screen 41 via the output unit 25. In the example of FIG. 6 corresponding to the example of FIG. 5, the control unit 21 moves the second pointer 52 on the surgical support image 16 displayed on the viewer screen 41 so as to follow the same movement trajectory as the first pointer 51, and stops the second pointer 52 at a position corresponding to the position clicked in the first region 32 on the viewer screen 41. That is, the control unit 21 reflects the display of the user operation on the viewer screen 41 by displaying the second pointer 52 on the viewer screen 41 so as to move in accordance with the movement of the first pointer 51.

[0064] In step S3, the control unit 21 stops reflecting the display of the user operation on the viewer screen 41.

[0065] Specifically, the control unit 21 stops reflecting the display of the user operation on the viewer screen 41 via the output unit 25. In the example of FIG. 4 corresponding to the example of FIG. 3, the control unit 21 stops reflecting the display of the user operation on the viewer screen 41 by making the second pointer 52 invisible on the viewer screen 41. In the example of FIG. 8 corresponding to the example of FIG. 7, the control unit 21 stops reflecting the display of the user operation on the viewer screen 41 by displaying the second pointer 52 on the viewer screen 41 with a fixed position.

[0066] In step S4, the control unit 21 displays the notification information 42 on the viewer screen 41.

[0067] Specifically, the control unit 21 displays the notification information 42 on the viewer screen 41 via the output unit 25. In the example of FIG. 4 corresponding to the example of FIG. 3, the control unit 21 displays, as the notification information 42, an icon indicating that the rotation button 34e has been clicked on the viewer screen 41. That is, the control unit 21 displays, as the notification information 42, information indicating the operation element being operated on the viewer screen 41. In the example of FIG. 10 corresponding to the example of FIG. 9, the control unit 21 highlights, on the viewer screen 41, as the notification information 42, a label indicating that the calibration operation has been selected. For example, the control unit 21 displays a group of labels corresponding to two or more setting items 36 included in the setting menu 35 at the upper part of the viewer screen 41, and displays the label corresponding to the setting item selected on the operator screen 31 in a color tone different from that of the other labels. That is, the control unit 21 displays, as the notification information 42, information indicating the selected setting item on the viewer screen 41. Although the floating region 37 overlaps at least a part of the surgical support image 16 on the operator screen 31, the control unit 21 continues to display the at least a part of the surgical support image 16 on the viewer screen 41.

[0068] As a method of rendering the surgical support image 16 on the operator screen 31 and the viewer screen 41, for example, information obtained by sensing is stored in the storage unit 22 as 2D memory information. Based on this 2D memory information, after setting the positions of the light source and the camera, rendering is performed on the operator screen 31, and the two-dimensional image generated as the surgical support image 16 is drawn in the first area 32 of the operator screen 31. It is also conceivable to draw it on the viewer screen 41 after adjusting the scale. However, in this embodiment, the screen ratios of the first area 32 of the operator screen 31 and the viewer screen 41 are different. Therefore, it is desirable to perform rendering separately for the operator screen 31 and the viewer screen 41. For example, as a more desirable method, after setting the positions of the light source and the camera based on the 2D memory information, rendering according to the screen ratio is performed on the operator screen 31, and the two-dimensional image generated as the surgical support image 16 is drawn in the first area 32 of the operator screen 31. At the same time, separate rendering according to the screen ratio is performed on the viewer screen 41, and another two-dimensional image generated as the surgical support image 16 is drawn on the viewer screen 41. That is, the control unit 21 of the image processing apparatus 20 may perform separate rendering of the image displayed in the first area 32 as the surgical support image 16 and the image displayed on the viewer screen 41 based on the same information obtained by sensing. Since each rendering is repeated at regular intervals, when the operator 15 changes the position of the light source or the camera, the change is automatically reflected in both the operator screen 31 and the viewer screen 41.

[0069] Even when displaying a three-dimensional image as the surgical support image 16, it is the same as when displaying a two-dimensional image. That is, as a method of drawing the surgical support image 16 on the operator screen 31 and the viewer screen 41, for example, information on a 3D space is stored in the storage unit 22, and based on this information on the 3D space, after setting the positions of the light source and the camera, volume rendering is performed on the operator screen 31, and the three-dimensional image generated as the surgical support image 16 is drawn in the first area 32 of the operator screen 31, and it is also conceivable to draw it on the viewer screen 41 after adjusting the scale. However, in this embodiment, the screen ratios of the first area 32 of the operator screen 31 and the viewer screen 41 are different. Therefore, it is desirable to use a method of performing volume rendering twice rather than a method of performing volume rendering only once. For example, as a more desirable method, based on the information on the 3D space, after setting the positions of the light source and the camera, volume rendering according to the screen ratio is performed on the operator screen 31, and the three-dimensional image generated as the surgical support image 16 is drawn in the first area 32 of the operator screen 31, and at the same time, separate volume rendering according to the screen ratio is performed on the viewer screen 41, and another three-dimensional image generated as the surgical support image 16 is drawn on the viewer screen 41. Since each volume rendering is repeated at regular intervals, when the operator 15 changes the position of the light source or the camera, the change is automatically reflected on both the operator screen 31 and the viewer screen 41.

[0070] As described above, in this embodiment, information unnecessary for the surgeon 14, such as operation elements for controlling the display of the surgical support image 16, is displayed only on the operator screen 31, and such unnecessary information can be omitted on the viewer screen 41. Therefore, the visibility of the viewer screen 41 is improved.

[0071] In this embodiment, instead of displaying a copy of the operator screen 31 on the viewer screen 41, clinically necessary information displayed on the operator screen 31 can be displayed on the viewer screen 41. Therefore, even if the floating area 37 on the operator screen 31 overlaps at least a part of the surgical support image 16, only the operator screen 31 has the part hidden, and the entire surgical support image 16 can continue to be displayed on the viewer screen 41.

[0072] In this embodiment, when the mouse cursor position on the operator screen 31 is within the first area 32 that displays clinically relevant information, its position can be identified and the mouse cursor can be displayed at the corresponding position on the viewer screen 41. Therefore, the operation status of the operator 15 can be conveyed to the surgeon 14.

[0073] In this embodiment, while minimizing the number of exposed personnel in the radiation room 11, communication between the surgeon 14 in the radiation room 11 and the operator 15 in the separate room 12 can be smoothed through the notification information 42.

[0074] As a modification of this embodiment, when the control unit 21 of the image processing apparatus 20 determines that the position of the first pointer 51 has not changed for a certain period even when the first pointer 51 is in the first area 32, the second pointer 52 may be made invisible on the viewer screen 41. For example, when the first pointer 51 does not move in the first area 32 for 3 seconds, the control unit 21 may make the second pointer 52 invisible on the viewer screen 41. According to this modification, the second pointer 52 can be prevented from interfering with the surgeon 14 when confirming the structure of the biological tissue. As a further modification, when the control unit 21 determines that the position of the first pointer 51 has not changed for a certain period, not only the second pointer 52 is made invisible on the viewer screen 41, but also the first pointer 51 may be made invisible on the operator screen 31.

[0075] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks described in the block diagram may be integrated, or one block may be divided. Instead of executing two or more steps described in the flowchart in chronological order according to the description, they may be executed in parallel or in a different order according to the processing capabilities of the device that executes each step, or as necessary. In addition, changes can be made without departing from the spirit of the present disclosure.

Explanation of Signs

[0076] 10 Image processing system 11 Radiation room 12 Separate room 13 Window 14 Surgeon 15 Operator 16 Surgical support image 17 Biopsy tissue image 20 Image processing apparatus 21 Control unit 22 Memory unit 23 Communication unit 24 Input unit 25 Output unit 30 First display 31 Operator screen 32 First area 33 Second area 34a 2D button 34b 2D / 3D button 34c Zoom-in button 34d Zoom-out button 34e Rotation button 35 Setting menu 36 Setting item 37 Floating area 40 Second display 41 Viewer screen 42 Notification information 50 Input device 51 First pointer 52 Second pointer

Claims

Claim 1 An image processing apparatus that causes a first display to display an operator screen including a first region that displays a surgical support image including a biological tissue image obtained by imaging based on information obtained by sensing the structure of a biological tissue in a subject's body, and a second region that displays at least one operation element for controlling the display of the surgical support image in the first region, and causes a second display different from the first display to display a viewer screen that displays the surgical support image in accordance with the display of the surgical support image in the first region, wherein: The image processing apparatus includes a control unit that receives a user operation performed on the operator screen using an input device, and when the user operation is performed in the first region, reflects the display of the user operation on the viewer screen, and when the user operation is performed in the second region, does not reflect the display of the user operation on the viewer screen. Claim 2 The image processing apparatus according to claim 1, wherein when the user operation is performed in the second region, the control unit displays notification information indicating that the user operation is being performed in the second region on the viewer screen. Claim 3 The control unit displays a setting menu including two or more setting items on the operator screen, and when receiving an operation of selecting any one of the two or more setting items from the setting menu using the input device, displays one or more operation elements corresponding to the selected setting item in the second region. When receiving an operation of the one or more operation elements as the user operation, the control unit displays information indicating the selected setting item as the notification information on the viewer screen. The image processing apparatus according to claim 2. Claim 4 The control unit displays two or more operation elements in the second region as the at least one operation element, and when receiving an operation of any one of the two or more operation elements as the user operation, displays information indicating the operation element being operated as the notification information on the viewer screen. The image processing apparatus according to claim 2. Claim 5 The input device is a pointing device. The control unit displays a first pointer movable by the user operation on the operator screen, and when the first pointer is in the first area, it displays a second pointer that moves in accordance with the movement of the first pointer on the viewer screen, thereby reflecting the display of the user operation on the viewer screen. When the first pointer is in the second area, it stops reflecting the display of the user operation on the viewer screen by making the second pointer non-displayed on the viewer screen. The image processing apparatus according to any one of claims 1 to 4.

6. The control unit, when it determines that the position of the first pointer has not changed for a certain period even when the first pointer is in the first area, makes the second pointer non-displayed on the viewer screen. The image processing apparatus according to claim 5.

7. The input device is a pointing device, The control unit displays a first pointer movable by the user operation on the operator screen, and when the first pointer is in the first area, it displays a second pointer that moves in accordance with the movement of the first pointer on the viewer screen, thereby reflecting the display of the user operation on the viewer screen. When the first pointer is in the second area, it stops reflecting the display of the user operation on the viewer screen by displaying the second pointer on the viewer screen with its position fixed. The image processing apparatus according to any one of claims 1 to 4.

8. The control unit, when it determines that the position of the first pointer has not changed for a certain period even when the first pointer is in the first area, makes the second pointer non-displayed on the viewer screen. The image processing apparatus according to claim 7.

9. The second area includes a floating area that can be moved using the input device and is allowed to overlap the first area, When the floating area overlaps at least a part of the surgical support image on the operator screen, the control unit makes the at least a part of the surgical support image non-displayed on the operator screen and continues to display the at least a part of the surgical support image on the viewer screen. The image processing apparatus according to any one of claims 1 to 8.

10. The control unit renders separately, based on the same information obtained by the sensing, the image displayed in the first region as the surgical support image and the image displayed on the viewer screen, for the image processing apparatus according to any one of claims 1 to 9.

11. An image processing apparatus according to any one of claims 1 to 10, the first display, and the second display comprising an image processing system.

12. The second display is installed in the radiation room where the surgery is performed, and the first display is installed in a separate room where operations using the input device are performed, for the image processing system according to claim 11.

13. A first region for displaying a surgical support image including a biological tissue image formed based on information obtained by sensing the structure of a biological tissue inside a surgical subject, and a second region for displaying at least one operation element for controlling the display of the surgical support image in the first region, are displayed on a first display, and a viewer screen for displaying the surgical support image in accordance with the display of the surgical support image in the first region is displayed on a second display different from the first display. A method for displaying an image, comprising: receiving a user operation performed on the operator screen using an input device, when the user operation is performed in the first region, reflecting the display of the user operation on the viewer screen, and when the user operation is performed in the second region, not reflecting the display of the user operation on the viewer screen.

14. When the user operation is performed in the second region, display notification information indicating that the user operation is performed in the second region on the viewer screen, for the image display method according to claim 13.

15. Causing a computer to display an operator screen including a first region for displaying a surgical support image including a biological tissue image formed based on information obtained by sensing the structure of a biological tissue inside a surgical subject, and a second region for displaying at least one operation element for controlling the display of the surgical support image in the first region, on a first display, and causing the computer to display a viewer screen for displaying the surgical support image in accordance with the display of the surgical support image in the first region on a second display different from the first display. A process of receiving a user operation performed on the operator screen using an input device, When the user operation is performed in the first area, reflecting the display of the user operation on the viewer screen, and when the user operation is performed in the second area, a process of not reflecting the display of the user operation on the viewer screen An image processing program that causes the above to be executed.

Images