Medical image processing device and medical image processing program

Abstract

To provide a medical image processing device and the like which can display a three-dimensional medical image and perform various display processing while having the excellent workability and keeping cleanliness of an operator.SOLUTION: An image processing device (301) is equipped with: a display (361); a control unit (350) which is connected to the display; a speech input device (370); and a motion sensor (380). The control unit (350) has: an image display part (355a) for displaying a three-dimensional medical image on the display; a mode selection part (355d) for switching, upon recognizing speech input through the speech input device (370), the mode of display related to the three-dimensional medical image in response to the speech; and a display processing part (355c) for changing, upon recognizing via the motion sensor a motion input by an operator, the display of the three-dimensional medical image in response to the motion.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a medical image processing apparatus and a medical image processing program. In particular, the present invention relates to a medical image processing apparatus and the like that can display a three-dimensional medical image obtained by imaging a patient and perform various display processes with good workability while maintaining the cleanliness of the operator.

Background Art

[0002] Currently, as medical image diagnostic apparatuses, CT (Computed Tomography) apparatuses, MRI (Magnetic Resonance Imaging) apparatuses, PET (Positron Emission Tomography) apparatuses, ultrasonic diagnostic apparatuses, angiography imaging apparatuses, etc. are known.

[0003] In recent years, for example, a simulation may be performed before performing an operation on an organ such as the liver where blood vessels are intricately intertwined. The simulation is performed, for example, by performing a contrast CT examination, preparing a fluoroscopic imaging image of the site to be treated, and confirming it on a display. Such a simulation is useful for examining a treatment plan. For example, Patent Document 1 also discloses a technique for simulating a surgical operation by displaying an organ such as the liver on a tablet terminal.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Since the technology of Patent Document 1 uses a tablet terminal, it is useful in that preoperative simulation can be carried out anywhere without choosing a location. In addition, since it can be brought into the operating room, it is also useful in that some kind of confirmation can be made during the operation.

[0006] However, if it is inconvenient for the operator who uses the terminal, the provided functions may not be utilized. Also, it is desirable that the operation can be performed while maintaining the cleanliness of the operator. This is because in a medical image processing apparatus, doctors and the like may perform operations during the operation.

[0007] The present invention has been made paying attention to such problems. The object is to provide a medical image processing apparatus, an image processing program, etc. that can display three-dimensional medical images obtained by imaging a patient and perform various display processes with good workability while maintaining the cleanliness of the operator.

Means for Solving the Problems

[0008] One form of the medical image processing apparatus of the present invention for solving the above problems is as follows: A display, A control unit (processor) connected to the display, A voice input device, A motion sensor, A medical image processing apparatus comprising: The control unit (processor) a: An image display unit that causes the display to display a three-dimensional medical image, b: A mode selection unit that recognizes the voice input using the voice recognition device and switches the mode related to the display of the three-dimensional medical image accordingly, c: A display processing unit that recognizes the motion input of the operator via the motion sensor and changes the display of the three-dimensional medical image accordingly, A medical image processing apparatus having

[0009] In other words, in the device according to one embodiment of the present invention, the control unit (processor) - causes the display to display a three-dimensional medical image, - recognizes the voice input using the voice recognition device, and accordingly switches the mode related to the display of the three-dimensional medical image, - recognizes the motion input of the operator via the motion sensor, and accordingly changes the display of the three-dimensional medical image.

[0010] (Explanation of Terms) · "Anatomical structure" refers to an object recognizable within the subject (e.g., organs, bones, blood vessels, etc.), and includes lesions such as fat and tumors. · "Terminal" refers to an information processing device that is connected to a network or used stand-alone and performs data processing. Any peripheral device may be connected to the information processing device to configure it. Basically, those in which various functions are provided within one device, such as a tablet terminal or a laptop computer, are preferred, but in some cases, some of its functions can be functionally or physically distributed in arbitrary units, for example, according to the load. · "Connection" includes not only the case where two elements are directly connected, but also the case where one element is indirectly connected to another element via some intermediate element without departing from the gist of the present invention. Also, both wired connection and wireless connection are included.

[0011] In this specification, a component expressed as "function" + "unit" corresponds to a functional block that performs a predetermined function. A functional block does not necessarily indicate a division between hardware circuits. Therefore, for example, one or a plurality of functional blocks can be implemented by a single hardware, but can also be implemented by a plurality of hardwares.

Advantages of the Invention

[0012] According to the present invention, it is possible to provide a medical image processing apparatus or the like that can display a three-dimensional medical image obtained by imaging a patient and perform various display processes with good workability while maintaining the cleanliness of the operator.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

Figure 8

Figure 9A

Figure 9B

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0014] Embodiments of the present invention will be described below with reference to the drawings. 〔Section A: Medical Image Processing Apparatus Capable of Performing Various Display Processes While Maintaining Workability and Operator Cleanliness〕 1. Configuration The medical image processing apparatus 301 of the present embodiment is, as an example, a portable computer device such as a tablet terminal. Alternatively, it may be a laptop PC (notebook PC) having a touch panel display. FIG. 1 shows an example of a tablet terminal, and it may be configured by installing an image processing program according to an aspect of the present invention on a commercially available tablet terminal. Hereinafter, the medical image processing apparatus will be simply referred to as an image processing apparatus for explanation. Regarding the tablet terminal and the notebook PC, although not particularly limited, in one aspect, it is preferable that the screen size is 9 inches or more, or 10 inches or more. The thickness is preferably 20 mm or less or 15 mm or less in one aspect. The mass is preferably within 2 kg or within 1.5 kg in one aspect.

[0015] As shown in FIG. 1, in this example, the image processing apparatus 301 has a thin housing 301a, and a touch panel display 360 is provided on one of its surfaces. The touch panel display 360 is composed of a display 361 (see FIG. 2) and a touch panel 363 (see FIG. 2).

[0016] Such an image processing apparatus 301 can be connected to the network of a hospital system, for example, as shown in FIG. 3. The hospital system in this example includes the following devices connected to the network 30: an imaging device 1, a chemical solution injection device 10, a HIS (Hospital Information System) 21 which is a hospital information system, a RIS (Radiology Information System) 22 which is a radiology department information system, a PACS (Picture Archiving and Communication Systems) 23 which is an image storage and communication system, a workstation 24, a printer 25, etc. Note that not all of these are essential components, and some can be omitted. Each of the above elements may be one or a plurality. The connection to the network may of course be a wired connection or a wireless connection.

[0017] Examples of the imaging device 1 include imaging devices such as a CT device, an MR device, and an angiography device. Other types of imaging devices may be used, or one or more imaging devices of the same type or different types may be used. Three-dimensional medical images described later may be created using a plurality of modality images, such as combining an image taken by a CT device and an image taken by an MR device.

[0018] The chemical solution injection device 10 may be at least a contrast agent injection device for injecting a contrast agent. Specifically, it may include a drive mechanism for extruding the chemical solution from a container (a syringe in one example) filled with the chemical solution, and a control circuit for controlling its operation. As an example, a contrast agent injection device including an injection head and a console can be used. The drive mechanism may be a piston drive mechanism or a roller pump, etc.

[0019] Refer to the block diagram of FIG. 2. The image processing apparatus 301 includes a display 361, a touch panel 363, an input device 365, a communication unit 367, an interface 368, a slot 369, a control unit 350, a storage unit 359, etc. Note that not all of these are essential, and some may be omitted.

[0020] Examples of the display 361 include devices such as a liquid crystal panel and an organic EL panel. A touch panel type display in which the touch panel 363 is integrally provided can also be used. As the touch panel, those of methods such as a resistive film, capacitance, electromagnetic induction, surface acoustic wave, and infrared can be used. As a specific example, it may be capable of detecting multi-touch, which is a touch at a plurality of positions such as the capacitance method. The touch operation can be performed using the user's finger, a touch pen, or the like. The touch panel may detect the start of a touch operation on the touch panel, the movement of the touch position, the end of the touch operation, etc., and output the type of the detected touch and the coordinate information.

[0021] Note that, as will be described later, the image processing apparatus 301 of the present embodiment can perform operations related to image display using voice input and motion input. Therefore, in some cases, the touch panel 363 may be omitted.

[0022] Examples of the input device 365 include general devices such as a keyboard and a mouse.

[0023] The storage unit 359 may be configured by a hard disk drive (HDD), a solid state drive (SDD), and / or a memory, etc., and may store an OS (Operating System) program and a medical image processing program (including algorithm data, graphical user interface data, etc.) according to an aspect of the present invention.

[0024] Also, other programs used for various processes, tables, databases, etc. are stored as necessary. The computer program is executed by being loaded into the memory of the control unit, and may cooperate with hardware such as a CPU to constitute a control unit having functions as in this embodiment.

[0025] The computer program may be such that all or part of it is downloaded from an external device when necessary via an arbitrary network. The computer program may be stored in a computer-readable recording medium, and the "recording medium" includes any "portable physical medium" such as a memory card, a USB memory, an SD card (registered trademark), a flexible disk, a magneto-optical disk, a ROM, an EPROM, an EEPROM, a CD-ROM, an MO, a DVD, and a Blu-ray (registered trademark) Disc. The medical image processing apparatus of this embodiment may be provided with a slot 369 for reading the above-described storage medium.

[0026] The communication unit 367 is a unit for enabling communication with an external network or device in a wired or wireless manner. The communication unit 367 may include a transmitter for sending data externally and a receiver for receiving data from the outside. The interface 368 is for connecting various external devices, etc., and although only one is shown in the figure, a plurality may of course be provided.

[0027] The slot 369 is a part for reading data from a computer-readable medium. The interface 368 is a part for connecting external devices, etc.

[0028] The image processing apparatus 301 of this embodiment includes a microphone 370 as a voice input device. It may be a microphone built into the housing, or an external microphone that is separate from the housing and is connected to the terminal by wire or wirelessly. It is also possible to use a device in which the following motion sensor 380 and the microphone are integrated.

[0029] In order to perform voice recognition, voice recognition software is installed in the image processing apparatus 301, and thereby, the voice recognition unit 351 is configured.

[0030] (Motion Sensor) The motion sensor 380 is a sensor that non - contact three - dimensionally detects the movement of at least a part of the operator's body. Motion recognition software is installed in the image processing apparatus 301, and thereby, the motion recognition unit 353 is configured.

[0031] As the motion sensor 380, for example, a Leap Motion Controller (manufactured by Leap Motion, "Leap Motion" is a registered trademark) etc. can be used. This Leap Motion Controller is an input device that can recognize the position, shape, and movement of the operator's fingers and / or the position and movement of the palm in real time without contact. The Leap Motion Controller is configured as a sensor unit incorporating an infrared irradiation unit and a CCD camera etc. The upper region of the sensor unit is the recognition area. This sensor unit is connected to a tablet terminal or a laptop PC by wire or wirelessly and used.

[0032] As the motion sensor 380, for example, Kinect (registered trademark, manufactured by Microsoft) can also be used. The motion sensor 380 may include one or more cameras and one or more distance sensors, or may include only one of them. Or, it may be such that a microphone is built into the unit of the motion sensor 380. As the accuracy of the detection target (e.g., hand) of the motion sensor 380, it is preferably possible to detect with an accuracy of 5 mm or less in one example, and more preferably possible to detect with an accuracy of 1 mm or less.

[0033] The detection principle of the motion sensor 380 is not limited to a specific one. As one aspect, a method called Light Coding can be used. In this method, a large number of dot patterns are irradiated from an infrared light emitting unit, and the amount of change (deformation) when the dot pattern hits the detection target (person) is read by a camera. As another aspect, a method called TOF (Time Of Flight) can be used. This is suitable for use in a situation where the recognition range is relatively short and fine movements of fingers are sensed. In the TOF method, the time until the irradiated infrared light hits the object and returns is analyzed to measure the distance. Generally, compared with the above Light Coding method, the recognition accuracy is high and the decrease in accuracy due to distance is small. As yet another aspect, a method may be used in which the reflection of the light irradiated from an infrared LED to the object is photographed by two cameras to recognize the movement.

[0034] (Control unit) Referring to FIG. 2 again. The control unit 350 has hardware such as a central processing unit (CPU) and a memory, and a computer program is installed to perform various arithmetic processes. The control unit 350 conceptually includes an image display unit 355a, an operation determination unit 355b, a display processing unit 355c, and a mode selection unit 355d. Also, as described above, it has an audio recognition unit 351 and a motion recognition unit 353.

[0035] The image display unit 355a displays a three-dimensional medical image for medical use on the display 361. As an example, the image display unit 355a displays each anatomical structure part, such as the liver or blood vessels, as an independent object. Also, it may be configured to display each anatomical structure part in a different color. Whether to display each anatomical structure part in which color may be manually input and set by the operator, but it is not necessarily limited to that. As will be described later, when color assignment or the like is made in advance on the side of a predetermined data server (by a table or the like), it may be configured to perform display according to that.

[0036] The operation determination unit 355b receives an input operation on the input device 365, the touch panel 363, etc.

[0037] The display processing unit 355c performs various image processes. For example, - Rotation of the three-dimensionally displayed image, - Translation of the three-dimensionally displayed image, - Enlargement / reduction of the three-dimensionally displayed image, - Change of the transparency of the three-dimensionally displayed image, - Switching between display and non-display of a predetermined object, - Cutting (dividing) function of a predetermined object, - Region designating function of a predetermined object, etc. It is as described above. The specific contents of these functions will be described in detail in a series of operations to be described later.

[0038] The voice recognition unit 351 performs various voice recognitions. For example, it recognizes the following words. - "Rotation", which is a command for changing the display mode - "Movement", which is a command for changing the display mode - "Multi", which is a command for changing the display mode - "Stop", which is a command for changing the display mode - "Cut", which is a command for processing - "Box", which is a command for processing - The name of an anatomical structure (for example, the name of an organ such as "liver", or the name of a blood vessel such as "portal vein", "hepatic artery").

[0039] The motion recognition unit 353 performs various motion recognition processes. For example, assuming that a motion sensor detects a hand, it detects the position and movement of the hand (finger) in the detection space.

[0040] Regarding an invention having main features in image processing and other data processing, the hardware configuration is not limited to the specific ones disclosed in the above-described embodiments, and various aspects can be utilized. Therefore, for example, it should be noted that not only tablet terminals and notebook PCs, but also processes performed by other computer means can be the subject of one form of the present invention. In addition, hereinafter, the invention mainly disclosed as an explanation of "operation" can be understood by those skilled in the art as an invention of an object or a computer program with a changed category expression. Therefore, this specification also discloses such an invention.

[0041] 2. Operation Subsequently, an operation example of image display in the image processing apparatus 301 of the present embodiment will be described. Hereinafter, as an example, an example of displaying and operating a three-dimensional medical image as illustrated in FIG. 5 will be described. This three-dimensional medical image includes a liver 371 and a blood vessel 375.

[0042] First, as shown in the flowchart of FIG. 4, acquisition of three-dimensional medical image data is performed as step S11. The "three-dimensional medical image data" may be created based on data obtained by tomographically imaging a patient with an imaging device. In particular, it may be volume data by volume rendering. Note that the data format of the three-dimensional image is not particularly limited, and various formats can be used. For example, the STL (Standard Triangulated Language) file format can also be used.

[0043] The image data may be stored in a predetermined data storage area such as a predetermined database server, a PACS, a DICOM server, or a workstation. As an example, the image processing apparatus 301 reads the data from a predetermined data storage area on the network and stores it in the storage unit 359 within the apparatus.

[0044] Next, the image processing apparatus 301 causes the 3D medical image to be displayed on the display 361 (step S12). The creation of the 3D medical image can basically be performed using a known method. The image creation flow according to one embodiment of the present invention will be described later with reference to the drawings. The data of the created 3D medical image may be stored within the image processing apparatus 301 and / or may be stored in an external server (for example, a server on the cloud).

[0045] Here, various display modes are prepared for the display of medical images. For example, - Displaying a predetermined anatomical structure in a semi-transparent state, - Displaying a predetermined anatomical structure in an opaque state, - Displaying a predetermined anatomical structure with color separation, - Displaying a predetermined anatomical structure with a shadow, - Displaying an image of a 3D coordinate axis (or something equivalent thereto. For example, a cube.) on the screen, etc., and there is at least one of these.

[0046] Regarding transparency, for example, the liver is displayed in a semi-transparent state, and the blood vessels are displayed in an opaque state. When there is a tumor, the tumor may also be displayed in an opaque state. According to such a display mode, by displaying the liver in a semi-transparent manner, it becomes possible to confirm the position and running state of the internal blood vessels that are originally hidden by the liver and cannot be visually recognized. The fact that such confirmation can be performed is very useful, for example, in a surgical operation such as removing a part of the liver in a laparoscopic operation, in that the positional relationship between the liver, blood vessels, tumor, etc. can be confirmed well. The image processing apparatus 301 of the present embodiment is portable, and therefore, the apparatus can be operated in the operating room to confirm the 3D medical image.

[0047] Regarding color separation, for example, the liver and blood vessels may be displayed in different colors. If there is a tumor, the tumor may be further displayed in a different color. More specifically regarding blood vessels, the liver, portal vein, and hepatic artery may each be displayed in a different color. When blood vessels are grouped, they may be displayed together in the same color.

[0048] Next, in step S13, the image processing apparatus 301 detects the position of the hand so that the distance between the motion sensor 380 and the operator's hand is appropriate. Specifically, the operator positions the hand above the motion sensor 380. The appropriate distance (height from the sensor to the hand) between the sensor and the operator's hand is preset, for example, in the range of h1 (mm) to h2 (mm). The reason for presetting such an appropriate range is that if the operator's hand is too close to or too far from the sensor, there is a possibility that the movement of the hand cannot be recognized well.

[0049] Regarding the detection of this hand position, there is no particular limitation on what kind of image is displayed on the screen, but it may be, for example, like that in FIG. 6. In this example, a reference circle (first circle) 391 of a predetermined size is displayed approximately in the center of the screen. The first circle 391 is always fixedly displayed with a constant size regardless of the position of the operator's hand. On the other hand, a second circle 393 is also displayed on the screen.

[0050] The center of the second circle 393 corresponds to the position of the operator's hand. That is, when the operator's hand is directly above (an example) the motion sensor 380, the center of the second circle 393 is the same as the center of the circle 391 at the second position. That is, the two circles 391 and 393 are displayed as concentric circles.

[0051] When the operator's hand is shifted in a predetermined direction (right side as an example) from the directly above position of the motion sensor 380, the second circle 393 is also shifted in the same direction (right side as an example) corresponding to this and is displayed in real time.

[0052] With such a display mode, the operator can check whether their hand is in an appropriate position (horizontal position) relative to the motion sensor 380 while observing the positional relationship between the two circles 391 and 393 on the screen.

[0053] Regarding the position in the height direction, it can be checked as follows. That is, the diameter of the second circle 393 corresponds to the height of the operator's hand. For example, when the height of the hand is the reference height (in one example, (h1 + h2) / 2), the second circle 393 is displayed so that its diameter is the same as that of the first circle 391. As the position of the hand becomes higher, the size of the second circle 393 also becomes smaller accordingly, and conversely, as the position of the hand becomes lower, the size of the second circle 393 also becomes larger accordingly. With such a display mode, the operator can check whether the height of their hand is appropriate while observing the size relationship between the two circles 391 and 393 on the screen.

[0054] To make it easier to confirm that it is in the appropriate position, the following display may be used. That is, when the horizontal position, height position, or a combination thereof of the hand enters a predetermined appropriate position, the second circle 393 may be displayed in a special display. For example, in one embodiment, it is preferable to display the second circle 393 with a different color when it is outside the appropriate range as shown in Fig. 6(a) and when it is inside the appropriate range as shown in Fig. 6(b), or to switch between a blinking display and a lit display.

[0055] In the above manner, the steps for making the distance between the motion sensor and the operator's hand appropriate are completed (step S13).

[0056] Next, in step S14, voice input for selecting a display mode is received. As an example of the voice input, it may recognize the following words: - "Move" - "Stop" - "Rotate" - "Multi"

[0057] Note that the voice recognition function may be configured to turn on when the position of the operator's hand enters a predetermined appropriate range in step S3. In this configuration, when the position of the operator's hand is not within the predetermined appropriate range, the voice recognition function is off, and it turns on only when it is within the appropriate range. Thus, according to the configuration in which the voice recognition function turns on only under a predetermined condition, it is possible to prevent voice input due to misrecognition unintended by the operator.

[0058] As schematically shown in FIG. 1, in order to notify the operator that the voice recognition function is on, it is also preferable that a display such as "Voice Recognition In Progress" is displayed on the screen.

[0059] Note that in one aspect of the present invention, the step of detecting the hand position in S13 may be omitted.

[0060] <Zoom (Enlargement and Reduction) / Pan> When it is desired to change the size or position of the displayed three-dimensional medical image, the following procedure is performed.

[0061] First, the operator vocalizes "Move" with the voice recognition function on. The image processing apparatus 301 analyzes the voice input from the microphone 370 by the voice recognition unit 351 and recognizes the word "Move". In response, it transitions to the "Zoom / Pan" mode (step S15).

[0062] In the "Zoom / Pan" mode, the image processing apparatus 301 then waits for input of the operator's motion by hand. The image processing apparatus 301 uses the motion sensor 380 and the motion recognition unit 353 to recognize the position and movement of the operator's hand in real time. And when the operator moves the hand upward (that is, when the hand moves from the initial height h0 to a higher h h to move), the image is gradually reduced in accordance with the movement. On the other hand, when the operator moves the hand downward (from the initial height h0 to a lower h LWhen moving to [destination], the medical image is gradually enlarged according to the movement.

[0063] Also, when the hand is moved horizontally, the three-dimensional medical image is panned (translated parallel) corresponding to the moving direction and the moving amount.

[0064] As described above, in this mode, the image is reduced or enlarged by moving the hand up and down, and the image is translated parallel by moving the hand horizontally. According to the configuration of the present embodiment, compared with input methods such as voice recognition and numerical input, zoom / pan of the image (and further rotation etc. described below) can be performed using motion input that enables intuitive analog input. Therefore, it is intuitive and excellent in operability for the operator, and is also practical.

[0065] Also, since the motion sensor 380 can perform input without contacting the device, input can be performed while maintaining the cleanliness of the operator's hand. According to such a configuration, for example, it is very advantageous in that a doctor during surgery can use the image processing device in the operating room to check the image.

[0066] Particularly in the case of the liver, there are a plurality of blood vessels branching in the liver. Therefore, when removing a part of the liver parenchyma so as not to damage the blood vessels more than necessary, it is necessary to sufficiently check the positional relationship between the blood vessels and the tumor. In this regard, according to the image processing device of the present embodiment, the positional relationship of blood vessels etc. can be checked while observing the three-dimensional medical image during surgery. Also, in a three-dimensional medical image, for example, there may be a case where a blood vessel exists on the front side and a tumor is hidden behind it (the tumor is not shown, see FIG. 5 for reference). Even in such a case, with the image processing device of the present embodiment, it is also possible to rotate the image in the "rotation" mode to check the tumor. In particular, in the configuration of the present embodiment, it is not rotated at predetermined angles, but can be freely (steplessly) rotated by an arbitrary angle by motion input, so good observation is possible.

[0067] In addition, in one aspect, it is preferable that during this "zoom / pan" mode, the "rotation" of the three-dimensional medical image (described in detail below) is configured not to be performed. When using this mode, it is often the case that only zooming in / out and translation are desired. Therefore, it is more user-friendly for the operator that rotation is prohibited and zooming in / out and translation are performed while maintaining the desired orientation.

[0068] In the above description, a mode in which both zooming and panning can be performed has been described, but it is not limited thereto. It is also possible to have a mode in which only one of them can be performed.

[0069] <Rotation> When it is desired to rotate the displayed three-dimensional medical image, the following procedure is performed: First, the operator vocalizes "stop" to cancel the above "zoom / pan" mode. The image processing apparatus 301 receives this by means of a voice recognition function, and cancels the "zoom / pan" mode and transitions to a state where other modes are accepted.

[0070] In this state, the operator vocalizes "rotation". The image processing apparatus 301 receives this by means of a voice recognition function and transitions to the "rotation" mode. The image processing apparatus 301 then waits for input of a motion by the operator's hand.

[0071] In the "rotation" mode, the image processing apparatus 301 recognizes the position and movement of the operator's hand in real time. Then, the image processing apparatus 301 rotates the three-dimensional medical image around a predetermined rotation axis (X-axis, Y-axis, Z-axis) in accordance with the movement of the operator's hand. Specifically, it recognizes the horizontal movement of the operator's hand or a movement such that the hand is moved along the surface of a virtual sphere. Then, the three-dimensional medical image is rotated by a predetermined angle corresponding to the movement direction, movement speed, and movement amount.

[0072] Also in this rotation mode, it is preferable in one embodiment that only rotation is allowed and panning (translation) and zooming (magnification / reduction) are prohibited. Thereby, for example, it becomes possible to rotate an image in a desired direction while maintaining a predetermined image size, and then perform predetermined image processing and observation.

[0073] Also when canceling the "rotation mode", as described above, the operator voices "stop".

[0074] In the description so far, a mode in which when one of "movement" and "rotation" is performed, the other is not performed has been described. However, a "multi" mode may be prepared so that both inputs can be performed simultaneously. In this mode, all of "zoom", "pan", and "rotation" are performed according to the movement of the operator's hand.

[0075] Specifically, when the operator voices "multi", the image processing apparatus 301 recognizes this and transitions to the "multi" mode. Rotation, movement, and magnification / reduction of the three-dimensional medical image are performed according to the input of the operator's hand motion.

[0076] Note that a function may be implemented that allows rotation of the three-dimensional medical image only by voice input instead of motion input. For example, by voicing "rotate", "left", "15°", the image processing apparatus 301 recognizes this. Then, the image is rotated by 15° around a predetermined rotation axis (for example, the Z axis extending in the vertical direction of the screen). When it is desired to rotate 15° upward around an axis extending in the horizontal direction, for example, voice input such as "rotate", "up", "15°" may be performed.

[0077] <Other voice inputs> The image processing apparatus 301 of the present embodiment can also select an anatomical structure part or change the transmittance of the selected one by voice input. Note that this will be described again after the description of the operations through the touch panel.

[0078] (Regarding various functions through touch panel input) The image processing device 301 displays anatomical structures in a three-dimensional medical image as independent objects. As a result, each can be individually selected or the display can be toggled on and off. For example, blood vessels such as the hepatic artery, portal vein, and hepatic vein may be grouped so that they can be selected collectively, or they may be individually selectable.

[0079] (Rotation function) The rotation of the three-dimensional medical image can also be performed by an operation on the touch panel. When the operator touches the touch panel and moves their finger, the image processing device 301 rotates the three-dimensional medical image accordingly.

[0080] (Zoom in / Zoom out function) The image processing device 301 may also perform image enlargement or display when the operator touches two points on the screen and performs an operation to separate or bring closer the distance between the two points (pinch-out operation, pinch-in operation).

[0081] (Display transparency switching function) The image processing device 301 may change the display density when the operator touches any anatomical structure (e.g., the liver). Specifically, it may be switched between two states: a normal opaque display state and a semi-transparent state. That is, as an example, it may become semi-transparent when touched once, and return to the normal display state when touched again.

[0082] As another aspect, for example, it may be set in multiple stages such as transparency 0%, 30%, 70%, 100% (non-display), and the display density may be sequentially switched in a loop each time it is touched. In this case, transparency 100% (i.e., non-display state) may be excluded from this loop. Naturally, the specific numerical values of the transparency can be changed as appropriate. In short, it suffices that the transparency is set in at least multiple stages and they can be switched.

[0083] According to such a display transparency switching function, it is possible to exert the display transparency switching function only by touching an arbitrary anatomical structure. Therefore, compared with a method in which it is necessary to separately select some icon or the like or select a command in order to switch the transparency, the operation can be performed simply and intuitively.

[0084] Also, when the display switches in a loop as described above, it is preferable in that it is not necessary to separately select an icon or the like, for example, to return to the original display state. Further, such loop-like display switching is also preferable in that it can be realized only by changing the display color of the selected object, so that image processing can be simplified and arithmetic processing with less memory becomes possible.

[0085] The gesture for changing the transparency is not limited to those described above. For example, when a finger (an example) is swiped in the vertical or horizontal direction, the image processing apparatus may receive the input and change the transparency accordingly. In this case, the transparency may be set in several steps, for example, 0%, 30%, 70%, 100% (non-display), or alternatively, the transparency may be changed without interruption (continuously).

[0086] (Display / non-display switching function) When the operator touches a predetermined anatomical structure for a certain period of time or more (an example), the image processing apparatus 301 sets the anatomical structure to the "selected state". In order to indicate that it is in the "selected state", the anatomical structure (for example, the liver) may be displayed in a color different from the initial state or may be blinked.

[0087] When the operator moves the fingertip (an example) to the edge of the screen while touching the anatomical structure in the selected state (for example, the liver) (swipe operation, drag operation, etc.), the image processing apparatus 301 makes the anatomical structure non-displayed. In the case of this example, the liver becomes non-displayed, and only the three-dimensional image of blood vessels and the like remains.

[0088] Such a function is useful when the operator wants to view only the desired anatomical structure. Also, according to the method of the present embodiment, in which the anatomical structure can be made invisible simply by directly selecting and moving it, it is possible to perform operations more simply and intuitively compared to a mode in which the display is switched on and off only by selecting some icon or the like.

[0089] (Cut function) The cut function is performed as follows. An example of cutting the liver and then making a part of it invisible will be described below. FIG. 8 is a flowchart of a series of operations.

[0090] The image processing apparatus 301 first displays a three-dimensional medical image as shown in FIG. 7A as step S1. Then, when the operator touches two points on an arbitrary anatomical structure (here, the liver 71) as shown in FIG. 7B, the image processing apparatus 301 determines that state, that is, that the two points are touched (step S2). Note that as for the timing, the two points may be touched simultaneously or substantially simultaneously.

[0091] Next, in order to cause a function to be exhibited in the case of a so-called long press, the image processing apparatus 301 determines whether or not the state in which the two points are touched continues for a certain period of time or more (step S3).

[0092] When the image processing apparatus 301 determines in step S3 that the continuation has been for a certain period of time or longer, it displays on the screen in a predetermined display mode that the two points P1 and P2 that have been touched have been specified. The "predetermined display mode" may be any one, but for example, (i) both the points P1 and P2 and the line L1 connecting them are displayed, or (ii) only the points P1 and P2 or only the line L1 may be displayed. Regarding the points P1 and P2, in order to clearly show the specified positions, instead of merely small dots, a display such as a slightly larger graphical image as shown in FIG. 7B (for example, any shape such as a circle, rectangle, polygon, star, etc., and a circle is exemplified here) may be used.

[0093] The image processing apparatus 301 may be configured to keep displaying the specified points P1 and P2 as they are even after the operator removes their hand from the screen, as shown in FIG. 7C. Also, it may be configured to accept fine adjustment of the positions of the points P1 and P2. So that it can be understood that it is in the mode of accepting such fine adjustment, for example, the circular graphical images of P1 and P2 and / or the line L1 may be made to blink. In FIG. 7C, as an example, a state where the point P2 is slightly moved and finely adjusted to the point P2' is exemplified.

[0094] This fine adjustment may be performed, for example, by the operator moving the graphical images of the points P1 and P2 with a finger (operation on the touch panel). As another aspect, motion input may be utilized to non - contactingly perform fine adjustment of the positions of the points P1 and P2 on the device. Regarding displaying the cut reference line L1 by voice input, it will be described again later. Here, first, assuming touch panel operation, the cut function and the like of this embodiment will be described.

[0095] After the specification of the points P1 and P2 is thus completed, in step S4, for example, the operator touches a predetermined icon (for example, an icon for "OK" input) on the screen. Then, by the cut function, as shown in FIG. 7D, the liver is cut by the line L1 connecting the points P1 and P2 (step S5).

[0096] The first part 71-1 and the second part 71-2, which are divided into two parts with the line L1 in between, can be operated as independent anatomical structures. As a method other than the above operation, for example, (i) instead of touching an icon, by touching a predetermined area on the screen, or (ii) performing an unusual operation such as touching multiple times (double-tap in one example), the above function may be executed. It may also be by voice input.

[0097] Since it can be operated as an independent anatomical structure, for example, when the first part 71-1 is touched (step S6), only the relevant part is selected as shown in FIG. 7E by the above-described function. And the display density thereof is switched. Specifically, only the first part 71-1 becomes semi-transparent. Touching it again returns to the original display.

[0098] Also, for example, when the first part 71-1 is long-pressed and a swipe operation or a drag operation is performed toward the peripheral part of the screen, that part becomes non-displayed, and only the second part 71-2 and the blood vessels 73, 75 remain. What becomes non-displayed may be displayed as a thumbnail image 66 as illustrated in FIG. 7F.

[0099] (Region specification) The image processing apparatus 301 designates a part of an anatomical structure as a region by the following operation of the operator. FIG. 9A shows a state where two points P1 and P2 are touched as in the operation described with reference to FIG. 7B (note that the operator's finger remains touching two points on the screen, but the illustration is omitted).

[0100] From this state, next, for example, as shown in FIG. 9B, when the operator moves two fingers (simultaneous movement of two fingers, although not limited), the image processing apparatus 301 specifies the positions of the two points P1' and P2' after the movement, and designates a substantially rectangular region based on that. Specifically, a rectangle surrounded by the four points of the two points P1 and P2 before the movement and the two points P1' and P2' after the movement is designated as the region.

[0101] Here too, as described above, it is also preferable that the image processing apparatus 301 is configured such that, even when the operator releases their hand, the approximate rectangle of the designated area remains on the screen, and the positions of the respective points P1, P2, P1', and P2' can be individually moved for fine adjustment of the position. As a method for determining the designated area, for example, the operator may touch a predetermined icon (e.g., an icon for "OK" input) on the screen.

[0102] When the positions of the four points P1, P2, P1', and P2' are designated as shown in FIG. 9B, the circular (an example) graphical images of the points P1, P2, P1', and P2' or the lines connecting them may be blinked so that each position can be finely adjusted.

[0103] The designated area Sa1 (see FIG. 9B) is divided from other parts and can be operated as an independent object. Therefore, it is possible to change only the display density of the area or switch the display on and off. With such a function, for example, by making only the area Sa1 non-displayed, it is possible to observe the inner blood vessels 73 and 75 and confirm the relationship between the blood vessels 73 and 75 and the liver 71.

[0104] Note that the area designation does not necessarily have to be performed in a rectangular shape, and a triangular or polygonal shape with five or more sides may be designated as the area.

[0105] In the above description, medical images of the liver and its surroundings were given as an example. Of course, in the present invention, anatomical structures are not limited to specific ones. For example, it may be possible to display medical images of the examiner's head and perform various image processes on them.

[0106] 〔Other functions by voice input〕 (Display / Emergency switching) It is also preferable that the display on / off switching function and the cut function as described so far can be executed only by voice input or the like without performing input on the touch panel.

[0107] First, regarding the selection of a predetermined anatomical structure, instead of touch, when the object is vocalized and recognized by voice, the selection is made. For example, when the operator says "liver", the image processing device 301 recognizes it by voice and sets the liver to the selected state. In order to indicate that it is in the "selected state", the anatomical structure (for example, the liver) may be displayed in a color different from the initial state or may be blinked.

[0108] And when it is desired to change the transparency of the liver, for example, the operator vocalizes "transparency". The image processing device 301 recognizes it by voice and switches the display of the liver to a semi-transparent state. In this case, other anatomical structures (in one example, blood vessels and tumors) continue to be displayed as opaque. With such a configuration, it becomes possible to confirm blood vessels and the like that would otherwise be hidden by the liver.

[0109] As one embodiment, for example, the transparency may be changed according to the distance from the motion sensor to the operator's hand. That is, when the hand is brought closer to the motion sensor, the transparency gradually increases (or decreases), and conversely, when the hand is moved away from the motion sensor, the transparency gradually decreases (or increases). Specifically, when the operator vocalizes "transparency" and the image processing device 301 recognizes it by voice, it enters a mode for receiving the above-described motion input. Then, the device detects the distance from the motion sensor to the operator's hand and changes the transparency accordingly.

[0110] (Line cut / Box cut) When performing a line cut, for example, the operator vocalizes "line cut". The image processing device 301 recognizes it by voice and displays a cutting reference line on the screen. This reference line may be like the line L1 in FIG. 7B.

[0111] Then, the image processing apparatus 301 waits for a motion input. The operator can change the position, length, orientation, etc. of the cutting reference line by the motion input. Thereby, the reference line can be set at a predetermined position without contact.

[0112] Next, for example, by saying "right cut", the area on the right side of the reference line is removed. Instead, by saying "left cut", the area on the left side of the reference line is removed. It may be a semi-transparent display instead of removal. FIG. 5(b) shows an example, where the part with the symbol 371-2 on the right side of the reference line remains opaque, and the part with the symbol 371-1 on the left side is semi-transparent.

[0113] When performing a box cut, for example, the operator voices "box cut". Although detailed illustration is omitted, the image processing apparatus 301 recognizes it by voice and displays a rectangle (an example) serving as a resection reference on the screen. The size of the rectangle may be only a predetermined one size, or a plurality of sizes such as large, medium, and small may be prepared.

[0114] This box cut is to resect a target anatomical structure at a predetermined depth. In a state where the rectangle serving as the resection reference on the screen is displayed, the image processing apparatus 301 waits for a motion input. The size and shape of the rectangle may be fixed, or may be changeable. For example, since the position of the corner of the initially displayed default rectangle can be moved, the size and shape of the rectangle may be configured to be changeable. A motion input can be used to move the position of the corner.

[0115] In this state, when the operator brings the hand closer to the motion sensor 380, for example, correspondingly, a substantially rectangular parallelepiped-shaped hole with a predetermined depth corresponding to the moving distance of the hand is formed in the liver while using the above rectangle as a contour. Thereby, a medical observation image can be obtained in which a part of the liver is resected and the internal blood vessels are not resected.

[0116] In addition, with the resection part of a predetermined depth formed in this way, it is also possible to rotate the entire three-dimensional medical image by a predetermined angle. For example, when the operator vocalizes "up", "15°", etc. in the rotation mode, the image processing apparatus 301 recognizes this and rotates the medical image in the state where the hole is formed by 15°. According to such a configuration, since the internal configuration of the hole (for example, although a part of the liver has been resected, the blood vessels are displayed) can be observed from different angles, it is useful.

[0117] In the above description, it has been explained that the removal is performed with a rectangular contour. Of course, the contour may be defined by any geometric shape such as a triangle, polygon, circle, ellipse, or the like.

[0118] In the above, the area designated as a box was removed. Conversely, a configuration may be adopted in which only the area designated as a box remains and the other areas are made non-displayed.

[0119] 〔Section B: Collective Handling of Multiple Anatomical Structure Parts〕 1. Problem of the Invention in this Section As the three-dimensional medical image exemplified in FIG. 5, as described above, volume data or the like is used. By the way, in such a three-dimensional medical image (that is, one including several anatomical structure parts of different types), the CT values (signal values) of the data are different between the liver and the blood vessels. Also, even for the same blood vessel, for example, the CT values (signal values) are different for arteries, veins, portal veins, etc.

[0120] This is because in fluoroscopic imaging using a contrast agent, after the contrast agent is injected, fluoroscopic imaging is performed after a predetermined time to obtain image data. Due to differences in the time of arrival of the contrast agent, etc., differences occur in the CT values (signal values) of each part such as arteries, veins, portal veins, and liver parenchyma. In the conventional method, threshold setting and filter processing were performed for each blood vessel and organ, and volume data for each part was created.

[0121] However, even for the same blood vessel, if the artery, vein, and portal vein are registered separately based on differences in CT values (signal values), the following situations may require relatively laborious operations. That is, in the viewer function of 3D medical images, for example, there are some that can switch between displaying a blood vessel in an emphasized state and not emphasizing it. By being able to do this, for example, the peripheral part of the blood vessel (with a low CT value (signal value)) can be switched between being displayed or not as needed, enabling observation as required (see Fig. 10). However, in the case of 3D medical images containing multiple different types of blood vessels, for each blood vessel, the operator cannot uniformly change the display mode of all blood vessels without resetting the display threshold and filtering, and thus there is a problem that it is difficult to easily make such a display change.

[0122] On the other hand, in actual observation, for example, there may be cases where it is preferable to be able to handle the vascular system and the parenchymal system separately. Therefore, in the invention of this section, the image processing apparatus has the following functions.

[0123] 2. Functions and Operations The image processing apparatus of this embodiment reads volume data based on information obtained by imaging a patient (see also step S1 in Fig. 3).

[0124] Then, it analyzes the signal value, CT value, and standard deviation (SD: Standard Deviation) within the volume. And, for example, if the average CT value is 300 HU or more, it automatically determines that it is an artery, and if the average CT value is 100 HU or less, it automatically determines that it is an organ. In addition to the CT value, the shape of the CT value histogram is also recognized. Generally, arteries tend to have a high peak and a narrow width (distribution), while portal veins and veins tend to have a low peak and a wide width (distribution). Therefore, based on such elements, automatic recognition of the type of blood vessel can be realized.

[0125] As described above, it becomes possible to automatically distinguish arteries, veins, portal veins, etc. with different CT values (signal values) by an image processing apparatus and register the data. Also, different colors may be automatically assigned to arteries, veins, portal veins, etc. and they may be displayed in different colors.

[0126] Also, histograms of blood vessels originally exist one by one for arteries, veins, portal veins, etc., but they may be integrated and normalized so that all blood vessels (or any two or more in other embodiments) can be represented by one histogram. Specifically, as an example, the average value, centroid, etc. of each histogram may be calculated, and the whole may be shifted so that the lower ones match the higher ones to create one histogram.

[0127] When a plurality of blood vessels (or other anatomical structures) are integrated and represented in one histogram in this way, instead of performing operations on the histograms of individual blood vessels, operations on only one histogram can be used to collectively change the display, etc. That is, for example, when the peripheral parts of blood vessels do not need to be displayed, the image processing apparatus receives a predetermined input from the operator and performs processing such as not displaying the parts below a certain reference value (or not displaying the parts below a certain reference value), so that the peripheral parts of arteries, veins, portal veins, etc. can be collectively made non-displayed (see, for example, FIG. 10(b)). On the other hand, when it is desired to emphasize and display up to the peripheral parts of blood vessels, the lower limit value of the CT value to be displayed may be set lower as shown in FIG. 10(a) (here the threshold value is 130 HU).

[0128] According to this configuration, it is not necessary to change the display modes of arteries, veins, portal veins, etc. respectively, so it is very easy to operate and useful in practice.

[0129] The above-mentioned "predetermined input from the operator" may be, for example, performed by operating an image button such as an icon, cursor, slider on the screen. Or it may be performed by recognizing a predetermined gesture of the operator's finger on the touch panel and based on that.

[0130] For example, in a mode where the display of blood vessels is changed, the operator touches the touch panel with several fingers and moves the fingers simultaneously in a predetermined direction, so that the display of the blood vessels can be changed. More specifically, when several fingers are simultaneously moved upward on the screen (the first direction), the peripheral part of the blood vessels will be displayed, and conversely, when the fingers are moved downward (the second direction), the peripheral part of the blood vessels (more precisely, near the outer edge of the thick part of the blood vessels) will disappear.

[0131] It is also preferable that the above operation can be performed by motion input via the motion sensor 380 instead of operation on the touch panel. That is, in this configuration, the display of blood vessels (an example. Other anatomical structures may also be used.) can be switched only by voice input and motion input, so there is no need to touch the touch panel, and the three-dimensional medical image can be observed while maintaining cleanliness.

[0132] 〔Section C: Other functions〕 (1) 3D-pointer As an example of the usage mode of three-dimensional medical images illustrated in FIG. 5, for example, a plurality of medical staff can check the running state of blood vessels (an example) before surgery and perform a simulation of the actual surgery.

[0133] In the case of two-dimensional medical images, for example, by making a pointer appear on the screen as an image and positioning it at a predetermined site, the relevant part can be focused on. However, in three-dimensional medical images, the pointer needs to be placed at an arbitrary site in three-dimensional space rather than in a plane. An operation of moving the pointer to an arbitrary site in such a dimensional space is relatively difficult to perform with an input interface such as a mouse or a touch panel.

[0134] Therefore, in this embodiment, the three-dimensional arrangement of the pointer may be performed using motion input. Specifically, the medical image processing apparatus 301 first accepts an input of "pointer" (an example) using the voice recognition function. Then, as an example, a three-dimensionally displayed pointer is displayed on the screen.

[0135] Regarding the vertical and horizontal directions on the screen, the pointer may be moved corresponding to the movement of the operator's hand within the horizontal plane. Regarding the depth direction, when the operator brings the hand closer to the motion sensor 380, the pointer may move in the depth direction of the three-dimensional medical image, and when moving it away, the pointer may move in the forward direction.

[0136] Regarding the display of the pointer, a display mode may be adopted in which the display size gradually decreases as it moves in the depth direction and gradually increases as it moves in the forward direction.

[0137] (2) Output of the schema image In diagnosis and examination, a two-dimensional drawing called a schema image representing a patient's body part may be used. Therefore, the image processing apparatus according to an aspect of the present invention may include a function of outputting a schema image.

[0138] For example, when there is a predetermined input by the operator, the image processing apparatus creates a corresponding schema image using the data of the three-dimensional medical image (see, for example, FIG. 5). The schema image may be any two-dimensional image such as a line drawing, a monochrome image, or a color image. Examples of the predetermined input by the operator include various inputs such as an input by touching an icon on the screen, a voice input, or a predetermined gesture input via a motion sensor.

[0139] As an example of creating a corresponding two-dimensional schema image using three-dimensional medical image data, for example, for a medical image displayed in an orientation (one example) as shown in FIG. 5, it may be such that the currently displayed image is two-dimensionally converted as it is and the data is written out. At this time, contour extraction processing may be performed to create a line drawing. The data format may be any type, but for example, the PDF (Portable Document Format) format or any other arbitrary image format such as GIF, PNG, or JPEG can be used. A doctor can write sketches, findings, etc. on the schema image created in this way, for example, with a touch pen or a finger.

[0140] The schema image created by the image processing apparatus can be sent out from the apparatus to the outside and stored in a predetermined storage area connected on a network (see FIG. 3). For example, it may be incorporated as part of an electronic medical record.

[0141] (3-1) Encryption The image processing apparatus of the present embodiment is a portable one such as a tablet terminal and can be taken out and used outside the hospital in some cases. Such a configuration may be useful, for example, when performing a surgical simulation while checking the three-dimensional medical image of a specific patient outside the hospital. However, from a security perspective, it is necessary that the internal information is encrypted when taken out of the hospital.

[0142] Therefore, it is preferable that the image processing apparatus according to one aspect of the present invention has the following functions: (a) a function of recognizing the current position of the apparatus, (b) a function of determining based on that whether the apparatus exists outside the hospital (or whether the apparatus does not exist inside the hospital), and (c) a function of automatically making non-display a predetermined information held by the apparatus when it is determined that it is outside the hospital (or when it is determined that it does not exist inside the hospital).

[0143] Examples of information to be hidden include information (identification information) that includes at least information that can identify a patient. Note that instead of simply displaying the information, the target information may be encrypted or access to the information may be prohibited.

[0144] As a method for determining whether it is inside or outside the hospital, for example, it may be based on whether it is within the range of the wireless network system within the hospital.

[0145] In this way, if the device is configured such that predetermined information is automatically anonymized when taken out of the hospital, it contributes to preventing leakage of patient information and the like, and is thus more preferable from the perspective of security.

[0146] Note that the effects of anonymization as described above are not necessarily limited to only inside or outside the "hospital". Such anonymization may be performed in a target specific area (which can be any facility or location).

[0147] (3-2) Regarding anonymization, furthermore, the following functions may be provided. FIG. 12 is a diagram schematically showing a state in which anonymization is being performed. On this screen, all information that can identify a patient is anonymized. Also, an icon 441 is displayed on the screen.

[0148] By the way, for a doctor or the like viewing a 3D medical image, for example, when they want to confirm which patient the image belongs to, it is also assumed that there may be a situation where they want to temporarily confirm the patient information.

[0149] Therefore, the medical image processing device in this example first determines that the icon 441 has been pressed (FIGS. 12 and 13), and then displays the patient information. The patient information may use data stored inside the device, or may be data obtained by accessing an external server (for example, a server within the hospital system).

[0150] More specifically, it is also preferable that the conditions under which such patient information can be displayed are limited under certain conditions. For example, when fingerprint authentication of the operator is performed on the device. Of course, it may also be the case where identity authentication is performed by other authentication methods.

[0151] Communication with an external server is preferably configured to be possible only under a secure communication situation such as a VPN (Virtual Private Network) in one example.

[0152] As the information to be displayed, for example, one, two, or three or more of the patient's initials, date of birth, gender, age, address, surgery date, attending physician, etc. may be used. Patient ID, examination ID, etc. may also be displayed.

[0153] In a case of confirming patient information under the above-described circumstances, in one form, it is preferable that only the minimum necessary information regarding the patient (for example, information that can identify the patient and can identify the surgery date or the physician) is displayed.

[0154] The displayed patient information may be automatically anonymized again after a certain period of time has elapsed. Or, once displayed, it may continue to be displayed during the operation (in one example, it continues to be displayed until the end of the work (logout) state).

[0155] As described above, by being able to confirm the minimum necessary patient information and / or surgery information as needed, it is possible to reduce the possibility of problems such as misidentification between the displayed three-dimensional medical image and the patient on whom the surgery is actually performed.

[0156] Regarding the above features, this specification discloses not only the invention of the device but also the inventions of the method and program corresponding to the above content.

[0157] (4) Stereo image A medical image processing apparatus according to an aspect of the present invention may have a function of displaying a stereo image as follows.

[0158] As shown in FIG. 11, an image including a first image 431L and a second image 431R may be displayed so that an operator can perform stereoscopic viewing. The first image 431L and the second image 431R display the same subject with a predetermined parallax. They may also be referred to as a left-eye image and a right-eye image, etc.

[0159] As shown in FIG. 11, an operation pad area 433 may be displayed in a screen including the images 431L and 431R. This operation pad area 433 is an area for changing the display angle of the subject. When the operator touches and moves a finger within this area 433, the medical image processing apparatus simultaneously changes the display angle of the subject (the two images) accordingly. That is, the orientation of the subject, etc. can be changed in conjunction with the movement of the finger.

[0160] Note that the above operation assumes an operation on a touch panel, but the input method is not limited to this and various methods can be used. For example, one or more of the non-contact input methods disclosed in this specification can be used.

[0161] In the case of an aspect where one operation pad area 433 is displayed together with the first image 431L and the second image 431R as shown in FIG. 11, the operator can intuitively understand that the orientation of the subject image can be changed, etc. by operating there.

[0162] The subject image is not particularly limited, but may be an angiogram image of blood vessels.

[0163] According to the configuration of displaying the first image 431L and the second image 431R as described above, a doctor can more accurately grasp the three-dimensional structure of the subject by stereoscopic viewing.

[0164] The device according to one embodiment of the present invention described above has a function of displaying a first image and a second image with different parallaxes from each other. More specifically, it further has a function of displaying an operation pad area for operating their display angles. For changes in the display of these stereoscopic images, for example, one or a combination of gesture input, voice input, motion input, etc. can be used. Note that this specification also discloses inventions of methods and programs corresponding to the above contents.

[0165] (5) Flow of image creation As a method for creating a medical image, for example, the following method may be adopted. In this method, first, fluoroscopic images of a patient are stored in a predetermined data server (for example, a DICOM server that stores data received from a modality in a predetermined format).

[0166] Then, in that predetermined data server (or another computer), individual parts of the subject are automatically recognized and managed as separate and independent objects. As shown in the table illustrated in FIG. 14, they may be divided and managed into categories such as arteries, veins, bones, organs, etc. (further subdivided categories). For each individual object, information (for example, if it is the aorta, the voice input "aorta (だいどうみゃく)") selected by the voice recognition function may be set.

[0167] Also, an automatic discrimination key (symbols, alphabets, numbers, combinations thereof) may be set to be convenient when selectively loading only necessary ones among the individual objects.

[0168] Also, an offset value may be set for each individual object. In this example, the offset value (refer to "CombineOfs" in the table) is set, for example, to "+50" for the aorta, "+100" for the abdominal artery, "+400" for the vein, etc.

[0169] The offset value is used as follows in one example. For example, assume that the central CT value of an artery is 350 HU and the central CT value of a vein is 200 HU. As such, the degree of contrast of blood vessels varies for arteries, veins (and even the portal vein), etc. respectively. Therefore, in order to align them, in the above example where the offset value is used, by setting the offset value of the vein to +150 HU and virtually increasing the contrast effect, it becomes possible to handle arteries and veins with the same threshold setting. Of course, the offset value may be set not only for arteries and veins but also for other blood vessels such as the portal vein.

[0170] Also, the following uses may be made for the offset value of the parenchymal system rather than the vascular system. Here, in one example, assume that the offset value of a parenchymal organ (such as the liver) is set to a large value such as +700 HU. By setting the parenchymal organ to such a large value, there is an advantage that the organ shape is likely to be kept as follows.

[0171] That is, for example, when the display of artery elimination is not required, an operation is assumed to increase the threshold so that the elimination is not displayed. In this case, however, the display of parenchymal organs such as the liver also changes accordingly and the shape collapses. In order to prevent this (that is, while changing the display mode of blood vessels, keeping the parenchymal organs in their original form), the parenchymal organs may be displayed with a large offset value such as +700 HU.

[0172] (Data conversion) When blood vessels, bones, organs, etc. are managed as separate objects in a predetermined data server (such as a DICOM server) (set in a table in one example), there are also the following advantages. That is, when constructing a medical image on a tablet terminal, etc., there is no need to manually input information of individual objects (text for voice recognition, offset value for image synthesis, etc.), and it becomes possible to easily create a medical image.

[0173] In this embodiment, basically, it is necessary to create medical images dedicated to a medical image processing apparatus. In this case, having the above-described setting table is preferable in that it can save some of the labor of production. However, depending on the surgical procedure (such as what kind of surgery is to be performed, etc.), it is also assumed that it may be preferable to prepare several tables. The reason for this is that in a certain surgical procedure, it may be preferable that blood vessel A (or organ A) can be clearly visualized, but in another surgical procedure, it may be preferable that blood vessel B (or organ B) can be more clearly visualized than blood vessel A (or organ A). That is, in one form, it is preferable that several tables with offset values set according to each surgical procedure are registered.

[0174] Also, in this case, the system or apparatus may be configured as follows. That is, it may be configured to (i) display a plurality of surgical procedures, (ii) accept that the operator has selected one of them, and (iii) call (and display if necessary) the corresponding table.

[0175] The display of the surgical procedure may be made, for example, according to a mode (site selection) for selecting the part of the body to be treated in a user interface for setting injection conditions. For example, it may be configured to display the surgical procedure corresponding to the selected predetermined site (such as the head, chest, abdomen, etc.) (see (i) above).

[0176] (5) Combined use with a physical switch or the like The above-described embodiment utilized voice input and motion input (input corresponding to human movements). In one form of the present invention, a predetermined physical switch may be further used in combination with these inputs.

[0177] Specifically, a switch for detecting physical contact (such as a foot switch) can be used. The "foot switch" is, for example, something placed on the floor and used, and has a switch housing where sensors and substrates are arranged, and a pressing part that is stepped on by a foot or the like. The pressing part is not limited, but may be a part configured to be movable so as to be pushed down when stepped on. The detection signal from the foot switch may be supplied externally by a cable (wired), or may be configured to be supplied externally wirelessly.

[0178] The foot switch may be electrically connected to the medical image processing apparatus (for example, the control unit 350, see FIG. 2) of the present invention, but is not limited thereto. The foot switch may be configured to be connected to other devices (for example, provided as a part of a chemical solution injection device).

[0179] FIG. 15 shows an example of the arrangement of the foot switch. In this example, the chemical solution injection device includes an injection head 475 arranged near the imaging device 470, a first control unit (power supply unit) 478 connected thereto, and a console (second control unit) 476 connected thereto. The foot switch 477d is connected to the power supply unit as an example.

[0180] (Example of using a physical switch) In the device according to one embodiment of the present invention, a plurality of reception conditions may be set for voice operation. The foot switch may be used as one of the reception conditions.

[0181] First, the following settings may be made as a plurality of reception conditions for voice operation (one or more). 1) Accept voice input only when there is motion input. 2) Accept voice input only when the foot switch is ON. 3) Accept voice input only when there is motion input and the foot switch is ON. 4) Always accept voice input.

[0182] In addition, the above input conditions may be registered in the voice operation command table of the device. In other words, the input in (1) means that when there is no motion input, the voice input is not accepted.

[0183] Regarding the voice input, it is also preferable that the following input method is adopted. That is, for example, when combination information of direction and angle such as "left 45°, up 30°" is input, the command input is accepted. This is because if it is not combination information, the probability of misrecognition may increase. This will be described in detail below.

[0184] When the voice input is ON and only words such as "up" or "front" can be recognized, in some cases, words in the conversation during the treatment may be recognized, and it is also assumed that unintended input may be made. Therefore, it may be configured to accept the command only when a combination of multiple words is recognized.

[0185] Such a voice input method for preventing misrecognition is not necessarily limited to the combination information of direction and angle as described above. As a specific example, consider the case of moving an image forward. In this case, when two words, the word "direction" (an example) for identifying that it is an operation mode to move and the word "front" (an example) which is the direction to move, are recognized, the command is accepted. From the perspective of preventing misrecognition, an input such as "direction" + "front" may be accepted, but an input such as "front" + "direction" may not be accepted (that is, the order of the combination is determined). In addition, regarding the command to enlarge the display of the image, instead of simply recognizing the word "enlarge", it may be configured to accept the command only when a combination such as "3D" + "enlarge" is recognized. With such a configuration, unintended voice input can be prevented, and it becomes more user-friendly.

[0186] As one aspect of voice input, in addition to or together with the above-described aspect, a configuration may be adopted in which voice input is accepted only for a certain period of time (a configuration in which voice input is not accepted outside of that certain period of time).

[0187] Regarding the input of a foot switch, the foot switch may be of a type that turns ON only while the pressing portion is being stepped on. As processing for the input of the foot switch, subsequent processing may be performed only when the foot switch has been stepped on for a certain period of time or longer (in the case of so-called long pressing). According to this, it is possible to prevent a situation in which the foot switch is accidentally pressed and unnecessary processing is performed. Regarding a process in which such processing is a concern, the above-described input may be performed. Further, instead of a foot switch, the above operation may be performed when another switch (physical switch) is ON.

[0188] Regarding voice input, the following functions may be provided. Here, a typical image processing apparatus according to one embodiment of the present invention is assumed to be configured such that input is performed by voice input only in a predetermined mode suitable for voice input, and input is performed by another input method otherwise. In such a case, when a predetermined input is made (for example, inputting "voice all ON" as voice input), a function may be provided that enables voice input for those for which voice input is not originally the default (at least some of them). However, such an expansion of voice input may, for example, automatically end after a certain period of time and return to the original state (timeout function). Although not limited, a timeout time of about 1 minute, 3 minutes, or 5 minutes set in advance may be used.

[0189] Although not limited thereto, with respect to the input for specifying an angle, it may always be configured to respond only to voice. Also, a table of each word of the voice input may be prepared, and for each word, it may be set what kind of situation permits the input (for example, for the word input of "aaa", it is not accepted unless the foot switch is ON, and the word input of "bbb" is always accepted, etc.).

[0190] (Home position setting function) By the way, the image processing apparatus of the present invention may basically be configured to always display a medical image viewed from a certain direction by default, regardless of the part. As an example, it may be configured to always display a default image viewed from the front of the body, regardless of any part and / or surgical procedure.

[0191] However, depending on the type of the part and / or surgical procedure, in one form, it may also be preferable to be configured to display a default image viewed from a preset angle suitable therefor. For example, in video-assisted thoracic surgery, the lateral decubitus position is basic. Therefore, in the case of video-assisted thoracic surgery, it may be configured to set the direction of the lateral decubitus position as the home position and display it by default.

[0192] That is, in one form of the present invention, depending on the type of the part and / or surgical procedure, not only the front view but also different display angles suitable for each may be preset and configured to display them by default.

[0193] The selection of the home position may be set manually or automatically according to at least one of the surgical procedure, the part, and the position of the tumor. As a specific example, depending on whether the lesion is in the left lung or the right lung, it is determined whether the left lateral decubitus position or the right lateral decubitus position should be taken. A configuration that automatically recognizes the position of the lesion (tumor) and automatically determines one of them accordingly is also useful.

[0194] (An example of screen display) The screen displayed when voice input is ON may be, for example, as shown in FIG. 16. Note that this screen is obtained by adding several images to the display example shown in FIG. 6, but those skilled in the art will easily understand that the images in FIG. 6 (see reference numerals 391 and 393) etc. may be omitted.

[0195] As shown in FIG. 16, a display 397b indicating that voice recognition (voice input) is ON may be provided on the screen so that it can be understood that voice recognition (voice input) is possible. Similarly, for motion input, a display 397a indicating that motion input is ON may be provided. It is also preferable to have a display unit 398 that displays the recognized voice as text. This makes it possible to visually confirm what voice input has been made.

[0196] Further, there may be a display unit 399 that indicates whether the recognized voice has been accepted as a command. When it is not accepted, a display such as "REJECTED" may appear, enabling the operator to visually confirm that it has not been accepted.

[0197] The embodiments of the present invention have been described above with reference to the drawings, but the present invention can be variously modified without departing from the spirit thereof and is not limited to the above examples.

[0198] Although a plurality of technical features have been described above, each of the technical features disclosed above can be used in appropriate combination, except in cases where they are mutually contradictory, and this specification also discloses such content. Also, for the sake of explanation, regarding the parts where several technical features are described as one embodiment, it is possible to omit one or more of those features. Those skilled in the art will understand that the technical content described without distinguishing between inventions of products, inventions of methods, or inventions of computer programs can be grasped as any of the inventions.

[0199] (Supplementary Note) This specification discloses the following invention (note that the reference numerals in parentheses do not limit the present invention in any way): 1. A medical image processing apparatus (301) comprising: a display (361); a control unit (350) connected to the display; a voice input device (370); a motion sensor (380); wherein the control unit (350) includes: a. an image display unit that causes the display to display a three-dimensional medical image; b. a mode selection unit that recognizes voice input using the voice recognition device (170) and switches the mode related to the display of the three-dimensional medical image accordingly; c. a display processing unit that recognizes an operator's motion input via the motion sensor and changes the display of the three-dimensional medical image accordingly; and has: a medical image processing apparatus. Or, an apparatus or system having at least a control unit having the above-described features.

[0200] 2. As the mode related to the display of the three-dimensional medical image, at least one of a zoom mode in which the three-dimensional medical image is enlarged or reduced by moving the operator's hand in a first direction, a pan mode in which the image is translated, a rotation mode in which the image is rotated, is included in the medical image processing apparatus described above.

[0201] 3. In the zoom mode, when the operator's hand is moved in a first direction, the three-dimensional medical image is enlarged, and when it is moved in a second direction opposite thereto, the three-dimensional medical image is reduced. The medical image processing apparatus described above.

[0202] 4. In the zoom mode, enlargement or reduction is performed without rotating the three-dimensional medical image.

[0203] 5. In the rotation mode, the three-dimensional medical image rotates in response to the movement of the operator's hand. The medical image processing apparatus according to the above description.

[0204] 6. In the rotation mode, the medical image processing apparatus according to the above description, wherein enlargement or reduction is performed without enlarging or reducing the three-dimensional image.

[0205] 7. It includes a housing (301a), A portable medical image processing apparatus in which at least the display and the control unit are integrally incorporated into the housing.

[0206] 8. The medical image processing apparatus according to the above description, wherein the motion sensor has one or more cameras.

[0207] 9. The medical image processing apparatus according to the above description, wherein the three-dimensional medical image includes at least image data of the liver and blood vessels.

[0208] 10. The medical image processing apparatus according to the above description, wherein the voice input device is a microphone.

[0209] 11. A medical image processing program that causes a computer to a: perform a process of displaying a three-dimensional medical image on a display, and b: recognize the voice input using the voice recognition device (170) and switch the mode related to the display of the three-dimensional medical image accordingly, and c: recognize the motion input of the operator via the motion sensor and change the display of the three-dimensional medical image accordingly.

[0210] 12. As the mode related to the display of the three-dimensional medical image, The zoom mode for enlarging and reducing the image, The pan mode for translating the image, The rotation mode for rotating the image, The program according to the above description having at least one of them.

[0211] 13. In the zoom mode, the computer is made to perform processing such that when the operator moves their hand in the first direction, the three-dimensional medical image is enlarged, and when the operator moves their hand in the second direction opposite thereto, the three-dimensional medical image is reduced. The program described above.

[0212] 14. In the zoom mode, the computer is made to perform processing such that enlargement or reduction is performed without rotating the three-dimensional image. The program described above.

[0213] 15. In the rotation mode, the computer is made to perform processing such that the three-dimensional medical image rotates in response to the movement of the operator's hand. The program described above.

[0214] 16. In the rotation mode, the computer is made to perform processing such that enlargement or reduction is performed without enlarging or reducing the three-dimensional image. The program described above.

[0215] 17. A step in which the computer displays a three-dimensional medical image on a display, A step in which the computer recognizes the voice input using the voice recognition device (170) and switches the mode related to the display of the three-dimensional medical image accordingly, A step in which the computer recognizes the motion input of the operator via a motion sensor and changes the display of the three-dimensional medical image accordingly, A method of operating a medical image processing device including the above steps.

[0216] 18. As the above mode related to the display of a three-dimensional medical image, A zoom mode for enlarging and reducing an image, A pan mode for translating an image, A rotation mode for rotating an image, The method described above having at least one of the above.

[0217] 19. In the zoom mode, when the operator moves their hand in the first direction, the three-dimensional medical image is enlarged, and when it is moved in the second direction opposite thereto, the three-dimensional medical image is reduced, according to the operation method described above.

[0218] 20. In the zoom mode, enlargement or reduction is performed without rotating the three-dimensional image, according to the operation method described above.

[0219] 21. In the rotation mode, the three-dimensional medical image rotates in response to the movement of the operator's hand, according to the operation method described above.

Explanation of Signs

[0220] 1 Imaging device 10 Chemical solution injection device 71, 371 Liver 73, 75, 375 Blood vessels 301 Medical image processing device 301a Housing 350 Control unit 351 Voice recognition unit 353 Gesture recognition unit 355a Image display unit 355b Operation determination unit 355c Display processing unit 359 Storage unit 360 Touch panel display 361 Display 363 Touch panel 365 Input device 367 Communication unit 368 Interface 369 Slot 370 Microphone 380 Motion sensor

Claims

1. The display and a control unit connected to the display; A medical image processing apparatus comprising: The control unit reading volume data including a plurality of anatomical structures based on information obtained by imaging a patient; analyzing CT values ​​in the volume data and determining the anatomical structure based on the analysis results; setting a CT value offset value for each of the plurality of anatomical structures so that the CT values ​​of the plurality of anatomical structures are aligned; receiving a predetermined input operation from an operator for setting a threshold value of the CT number, and setting a lower limit value of the CT number to be displayed on the display of the image of the anatomical structure based on the received input operation; 10. A medical image processing device configured to:

2. A medical image processing device as described in claim 1, wherein the multiple anatomical structures include blood vessels and solid organs, and the control unit sets the offset value of the solid organs to be greater than the offset value of the blood vessels.

3. A medical image processing device as described in claim 1 or 2, wherein the display is a touch panel display.

4. A medical image processing device as described in Claim 3, wherein the specified input operation from the operator is a specified movement of the operator's finger on the touch panel display.

5. A medical image processing device as described in Claim 4, wherein the predetermined action of the operator's fingers is the operator touching the touch panel display with multiple fingers and moving the multiple fingers simultaneously in a predetermined direction.

6. Equipped with a housing, 6. The medical image processing apparatus according to claim 1, wherein at least the display and the control unit are integrally incorporated into the housing, and the medical image processing apparatus is portable.

7. On the computer, reading volume data including a plurality of anatomical structures based on information obtained by imaging a patient; analyzing CT values ​​in the volume data and determining the anatomical structure based on the analysis results; setting a CT value offset value for each of the plurality of anatomical structures so that the CT values ​​of the plurality of anatomical structures are uniform; receiving a predetermined input operation from an operator for setting a threshold value of the CT number, and setting a lower limit value of the CT number to be displayed on a display of the image of the anatomical structure based on the received input operation; A medical image processing program that performs the following:

8. a computer reading volume data including a plurality of anatomical structures based on information obtained by imaging a patient; a computer analyzing CT values ​​in the volume data and determining the anatomical structure based on the analysis results; a computer setting an offset value of the CT number for each of the plurality of anatomical structures so that the CT numbers of the plurality of anatomical structures are aligned; a computer receiving a predetermined input operation from an operator for setting a threshold value of the CT number, and setting a lower limit value of the CT number to be displayed on a display of the image of the anatomical structure based on the received input operation; A method of operating a medical imaging device, comprising:

Images