Medical image processing device and medical image processing program

Abstract

To provide a medical image processing device and the like which can display a three-dimensional medical image and perform various display processing while having the excellent workability and keeping cleanliness of an operator.SOLUTION: An image processing device (301) is equipped with: a display (361); a control unit (350) which is connected to the display; a speech input device (370); and a motion sensor (380). The control unit (350) has: an image display part (355a) for displaying a three-dimensional medical image on the display; a mode selection part (355d) for switching, upon recognizing speech input through the speech input device (370), the mode of display related to the three-dimensional medical image in response to the speech; and a display processing part (355c) for changing, upon recognizing via the motion sensor a motion input by an operator, the display of the three-dimensional medical image in response to the motion.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a medical image processing apparatus and a medical image processing program. In particular, the present invention relates to a medical image processing apparatus and the like that can display a three-dimensional medical image obtained by imaging a patient and perform various display processes with good workability while maintaining the cleanliness of the operator.

Background Art

[0002] Currently, as medical image diagnostic apparatuses, CT (Computed Tomography) apparatuses, MRI (Magnetic Resonance Imaging) apparatuses, PET (Positron Emission Tomography) apparatuses, ultrasonic diagnostic apparatuses, angiography imaging apparatuses, etc. are known.

[0003] In recent years, for example, a simulation may be performed before performing surgery on an organ such as the liver where blood vessels are intricately intertwined. The simulation is performed, for example, by conducting a contrast CT examination, preparing a fluoroscopic imaging image of the site to be operated on, and confirming it on a display. Such a simulation is useful for considering a treatment plan. For example, Patent Document 1 also discloses a technique for simulating surgical operations by displaying an organ such as the liver on a tablet terminal.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology described in Patent Document 1 is useful because it utilizes a tablet device, allowing for pre-operative simulations to be performed anywhere, regardless of location. Furthermore, because it can be brought into the operating room, it is also useful for performing checks during surgery.

[0006] However, if the terminal is difficult for the user to operate, the features provided may not be used. Furthermore, it is desirable that the terminal can be operated while maintaining the user's hygiene. This is because, in the case of medical image processing equipment, doctors and other medical professionals may operate it during surgery.

[0007] This invention was made in view of these problems. Its purpose is to provide a medical image processing device and image processing program, etc., that can display three-dimensional medical images obtained by imaging a patient, and that can perform various display processing in an easy-to-use manner while maintaining the cleanliness of the operator. [Means for solving the problem]

[0008] A medical image processing apparatus according to one embodiment of the present invention for solving the above problems is as follows: a display and The control unit (processor) connected to the display, Voice input device and Motion sensor and, A medical image processing device comprising: The control unit (processor) is: a: An image display unit that displays a three-dimensional medical image on the aforementioned display, b: The aforementioned audio input A mode selection unit that recognizes the voice input using the device and switches the mode for displaying the three-dimensional medical image accordingly, c: A display processing unit that recognizes the operator's motion input via the motion sensor and changes the display of the 3D medical image accordingly, A medical image processing device having [a certain feature].

[0009] In other words, in one embodiment of the present invention, the control unit (processor) is - Display a 3D medical image on the aforementioned display, - The aforementioned audio input The device recognizes the input voice and switches the mode for displaying the 3D medical image accordingly. -The system is configured to recognize the operator's motion input via the motion sensor and to change the display of the 3D medical image accordingly.

[0010] (Explanation of terms) "Anatomical structures" refer to recognizable objects within the subject (e.g., organs, bones, blood vessels, etc.), and also include fat, tumors, and other lesions. • "Terminal" refers to an information processing device that is connected to a network or used standalone and performs data processing. This information processing device may be configured by connecting any peripheral devices. While devices with various functions integrated into a single unit, such as tablet terminals or laptop computers, are generally preferred, in some cases, some of these functions may be functionally or physically distributed in arbitrary units, for example, depending on the load. "Connection" includes not only cases where two elements are directly connected, but also, to the extent that it does not depart from the spirit of the present invention, cases where one element and another are indirectly connected through some intermediate element. It also includes both wired and wireless connections.

[0011] In this specification, a component expressed as "function" + "part" corresponds to a functional block that performs a predetermined function. A functional block does not necessarily indicate a division between hardware circuits. Therefore, for example, one or more functional blocks can be implemented on a single piece of hardware, but they can also be implemented on multiple pieces of hardware. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a medical image processing apparatus or the like that displays a three-dimensional medical image obtained by imaging a patient and performs various display processes with good workability while maintaining the cleanliness of the operator.

Brief Description of the Drawings

[0013] [Figure 1] It is a diagram showing a medical image processing apparatus according to an embodiment. [Figure 2] It is a diagram showing an example of a block diagram of the image processing apparatus in FIG. 1. [Figure 3] It is a diagram showing an example of the components of a hospital system. [Figure 4] It is a flowchart of an operation example of the image processing apparatus in FIG. 1. [Figure 5] It is a diagram showing an example of a three-dimensional medical image displaying the liver and the blood vessels around it. [Figure 6] It is an example of a graphical image when recognizing the position of the operator's hand. [Figure 7A] It is a diagram showing an example of a three-dimensional medical image. [Figure 7B] It is a diagram for explaining a state of designating two points for an arbitrary object on a three-dimensional medical image. [Figure 7C] It is a diagram for explaining a state of finely adjusting the position of the designated point. [Figure 7D] It is a diagram showing an example of dividing the liver. [Figure 7E] It is a diagram for explaining a state of touching a part of the divided liver. [Figure 7F] It is a diagram showing a state where only a part of the divided liver is hidden. [Figure 8] It is a flowchart showing the flow of an operation procedure as an example. [Figure 9A] It is a diagram for explaining the procedure of region designation. [Figure 9B] It is a diagram for explaining the procedure of region designation (an example). [Figure 10] It is a three-dimensional medical image showing an example of displaying blood vessels with different threshold values. [Figure 11] This figure shows an example of a screen displaying a stereo image. [Figure 12] This figure shows an example of a screen related to the anonymization of patient information. [Figure 13] This is an operational flow that displays patient information in specified cases. [Figure 14] This table indicates that individual body parts in a patient's fluoroscopic image are managed separately and independently on the data server. [Figure 15] This diagram schematically shows an example of the placement of foot switches. [Figure 16] This is an example of a screen that appears when voice input is turned on. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below with reference to the drawings. [Section A: Medical image processing apparatus capable of performing various display processes while maintaining good workability and operator hygiene] 1. Structure The medical image processing device 301 of this embodiment is, for example, a portable computer device such as a tablet terminal. Alternatively, it may be a laptop PC (notebook PC) having a touch panel display. Figure 1 shows an example of a tablet terminal, and it may be configured by installing an image processing program according to one embodiment of the present invention on a commercially available tablet terminal. Hereinafter, the medical image processing device will be simply referred to as an image processing device. While not particularly limited to tablet terminals or notebook PCs, in one embodiment, a screen size of 9 inches or more, or 10 inches or more, is preferred. In one embodiment, a thickness of 20 mm or less, or 15 mm or less, is preferred. In one embodiment, a mass of 2 kg or less, or 1.5 kg or less, is preferred.

[0015] As shown in Figure 1, in this example, the image processing device 301 has a thin housing 301a, and a touch panel display 360 is provided on one of its surfaces. The touch panel display 360 consists of a display 361 (see Figure 2) and a touch panel 363 (see Figure 2).

[0016] Such an image processing device 301 can be connected to the hospital system network, for example, as shown in Figure 3. The hospital system in this example includes the following equipment connected to the network 30: an imaging device 1, a drug injection device 10, a hospital information system HIS (Hospital Information System) 21, a radiology information system RIS (Radiology Information System) 22, a picture archiving and communication system PACS (Picture Archiving and Communication Systems) 23, a workstation 24, and a printer 25, etc. Not all of these are essential components, and some can be omitted. Each of the above elements may be present individually or in multiples. The connection to the network may, of course, be wired or wireless.

[0017] Examples of imaging devices 1 include CT scanners, MRI scanners, and angiography scanners. Other types of imaging devices may be used, or multiple imaging devices of the same or different types may be used. Three-dimensional medical images, as described later, may be created using images from multiple modalities, such as combining images acquired with a CT scanner and images acquired with an MRI scanner.

[0018] The drug injection device 10 may be at least a contrast agent injection device that injects a contrast agent, and more specifically, it may be equipped with a drive mechanism that pushes out the drug solution from a container filled with the drug solution (a syringe in one example) and a control circuit that controls its operation. As an example, a contrast agent injection device equipped with an injection head and a console can be used. The drive mechanism may be a piston drive mechanism or a roller pump, etc.

[0019] Refer to the block diagram in Figure 2. The image processing device 301 includes a display 361, a touch panel 363, an input device 365, a communication unit 367, an interface 368, a slot 369, a control unit 350, and a storage unit 359, etc. Not all of these are essential, and some may be omitted.

[0020] Examples of the display 361 include devices such as liquid crystal panels and organic EL panels. A touch panel display with an integrated touch panel 363 can also be used. The touch panel can be of any type, such as resistive, capacitive, electromagnetic induction, surface acoustic wave, or infrared. As a specific example, it may be a capacitive touch panel capable of detecting multi-touch, which is touch at multiple locations. Touch operations can be performed using the user's finger or a stylus. The touch panel may detect the start of a touch operation, the movement of the touch position, the end of a touch operation, etc., and output the type of touch detected and coordinate information.

[0021] Furthermore, as will be described later, the image processing device 301 of this embodiment can perform operations related to image display using voice input and motion input. Therefore, the touch panel 363 may be omitted in some cases.

[0022] Examples of input devices 365 include common devices such as keyboards and mice.

[0023] The storage unit 359 may consist of a hard disk drive (HDD), a solid state drive (SDD), and / or memory, and may store an OS (Operating System) program or a medical image processing program according to one embodiment of the present invention (including algorithm data and graphical user interface data, etc.).

[0024] Furthermore, other programs used for various processes, as well as tables, databases, etc., are stored as needed. Computer programs are executed by being loaded into the memory of the control unit, and they cooperate with hardware such as the CPU, thereby constituting a control unit with functions like those of this embodiment.

[0025] The computer program may be downloaded in whole or in part from an external device via any network when needed. The computer program may be stored on a computer-readable recording medium, and the term "recording medium" includes any "portable physical medium" such as memory cards, USB memory, SD cards (registered trademark), flexible disks, magneto-optical disks, ROMs, EPROMs, EEPROMs, CD-ROMs, MOs, DVDs, and Blu-ray (registered trademark) Discs. The medical image processing device of this embodiment may be provided with a slot 369 for reading such storage mediums.

[0026] The communication unit 367 is a unit that enables communication with an external network or device by wired or wireless means. The communication unit 367 may have a transmitter for sending data to the outside and a receiver for receiving data from the outside. The interface 368 is for connecting various external devices, and although only one is shown in the figure, there may, of course, be multiple interfaces.

[0027] Slot 369 is for reading data from computer-readable media. Interface 368 is for connecting external devices, etc.

[0028] The image processing device 301 of this embodiment is equipped with a microphone 370 as an audio input device. This microphone may be built into the housing, or it may be a separate external microphone connected to the terminal via wired or wireless connection. It is also possible to use a device that integrates the motion sensor 380 and microphone as described below.

[0029] To perform speech recognition, speech recognition software is installed on the image processing device 301, thereby configuring the speech recognition unit 351.

[0030] (Motion sensor) The motion sensor 380 is a sensor that detects the movement of at least a part of the operator's body in a non-contact manner in three dimensions. Motion recognition software is installed on the image processing device 301, which constitutes the motion recognition unit 353.

[0031] As the motion sensor 380, for example, a Leap Motion Controller (manufactured by Leap Motion, Inc., "Leap Motion" is a registered trademark) can be used. This Leap Motion Controller is an input device that can recognize the position, shape, and movement of the operator's fingers, and / or the position and movement of the palm, etc., in real time without contact. The Leap Motion Controller is configured as a sensor unit that incorporates an infrared irradiator and a CCD camera, etc. The upper area of ​​the sensor unit is the recognition area. This sensor unit is used by connecting it to a tablet terminal or laptop PC by wire or wirelessly.

[0032] Other motion sensors such as Kinect (a registered trademark of Microsoft Corporation) can also be used as the motion sensor 380. The motion sensor 380 may comprise one or more cameras and one or more distance sensors, or it may comprise only one of them. The motion sensor 380 unit may also have a built-in microphone. The detection accuracy of the motion sensor 380 for the object being detected (e.g., a hand) is preferably 5 mm or less, and more preferably 1 mm or less.

[0033] The detection principle of the motion sensor 380 is not limited to a single specific method. One method is called Light Coding. In this method, a large number of dot patterns are emitted from an infrared emitter, and the amount of change (distortion) when the dot patterns hit the detection target (person) is read by a camera. Another method is called Time of Flight (TOF). This is suitable for applications where the recognition range is relatively short and for sensing fine movements of fingers and hands. In the TOF method, the distance is measured by analyzing the time it takes for the emitted infrared light to hit the object and return. Generally, it has higher recognition accuracy and less accuracy degradation with distance compared to the Light Coding method described above. Yet another method may be used in which the reflection of light emitted by an infrared LED onto an object is captured by two cameras, and the movement is recognized.

[0034] (Control Unit) Refer to Figure 2 again. The control unit 350 has hardware such as a central processing unit (CPU) and memory, and a computer program is installed on it to perform various calculations. Conceptually, the control unit 350 has an image display unit 355a, an operation determination unit 355b, a display processing unit 355c, and a mode selection unit 355d. It also has a voice recognition unit 351 and a motion recognition unit 353 as described above.

[0035] The image display unit 355a displays medical 3D images on the display 361. For example, the image display unit 355a displays anatomical structures such as the liver and blood vessels as separate objects. It may also display each anatomical structure in a different color. The color used to display each anatomical structure may be manually entered and set by the operator, but is not limited to this. As will be described later, if color assignments etc. have been made in advance on the data server side (by a table, etc.), the display may be configured to follow those assignments.

[0036] The operation determination unit 355b receives input operations such as those for the input device 365 and the touch panel 363.

[0037] The display processing unit 355c performs various image processing. For example, - Rotation of a three-dimensionally displayed image, - Translation of a three-dimensionally displayed image, - Three-dimensional display and image zoom in / out. - Three-dimensional display and change of image transparency, - Switching between showing / hiding a specified object. - A function to cut (split) a specified object. - Function to specify the area of ​​a given object, etc. These are the specific functions, which will be explained in detail later in the series of operations described below.

[0038] The speech recognition unit 351 performs various speech recognition functions. For example, it recognizes the following words: - The command "Rotate" is used to change the display mode. - The command "Move" is used to change the display mode. - The command "Multi" is used to change the display mode. - The command "Stop" is used to change the display mode. - The command for processing is "cut". - The command for processing is "Box". - Names of anatomical structures (for example, organ names like "liver," or blood vessel names like "portal vein" or "hepatic artery").

[0039] The motion recognition unit 353 performs various motion recognition processes. For example, assuming the motion sensor detects a hand, it detects the position and movement of the hand (fingers) within the detection space.

[0040] Furthermore, with respect to inventions whose main features lie in image processing and other data processing, the hardware configuration is not limited to the specific configurations disclosed in the embodiments described above, but various configurations are available. Therefore, it should be noted that, for example, processing performed by other computer means, not just tablet terminals and notebook PCs, can also be subject to one form of the present invention. In addition, those skilled in the art will understand that the inventions disclosed below, primarily in terms of "operation," can also be understood as inventions of products or computer programs, simply by changing the category expression. Therefore, this specification also discloses such inventions.

[0041] 2.Operation Next, an example of image display operation in the image processing device 301 of this embodiment will be described. Below, as an example, an example of displaying a three-dimensional medical image as illustrated in Figure 5 will be described. This three-dimensional medical image includes a liver 371 and blood vessels 375.

[0042] First, as shown in the flowchart in Figure 4, step S11 is the acquisition of 3D medical image data. "3D medical image data" may be created based on data obtained from tomographic imaging of the patient using an imaging device. In particular, volume data obtained through volume rendering may also be used. The data format of the 3D image is not particularly limited and various formats can be used; for example, the STL (Standard Triangulated Language) file format can also be used.

[0043] Image data may be stored in a predetermined data storage area, such as a designated database server, PACS, DICOM server, or workstation. For example, the image processing device 301 reads the data from a predetermined data storage area on the network and stores it in the storage unit 359 within the device.

[0044] Next, the image processing device 301 displays the three-dimensional medical image on the display 361 (step S12). The creation of the three-dimensional medical image can be basically carried out using known methods. The image creation flow according to one embodiment of the present invention will be described later with reference to the drawings. The data of the created three-dimensional medical image may be stored in the image processing device 301 and / or on an external server (for example, a server on the cloud).

[0045] Here, various display modes are available for displaying medical images. For example, - Display specified anatomical structures in a semi-transparent state. - To display a specified anatomical structure in an opaque state. -Display designated anatomical structures using different colors. - Display designated anatomical structures with shading. - At least one of the following: displaying an image of three-dimensional coordinate axes (or something equivalent, such as a cube) on the screen.

[0046] Regarding transparency, for example, the liver may be displayed in a semi-transparent state, while blood vessels may be displayed in an opaque state. If a tumor is present, the tumor may also be displayed in an opaque state. With this display method, by displaying the liver in a semi-transparent state, it becomes possible to confirm the location and course of internal blood vessels that would normally be hidden and invisible from view. Being able to make such confirmations is extremely useful, for example, in surgeries such as laparoscopic surgery in which a portion of the liver is removed, as it allows for good confirmation of the positional relationship between the liver, blood vessels, tumors, etc. The image processing device 301 of this embodiment is portable, and therefore, the device can be operated in the operating room to confirm 3D medical images.

[0047] Regarding color coding, for example, the liver and blood vessels may be displayed in different colors. If a tumor is present, the tumor may be displayed in yet another color. More specifically regarding blood vessels, the liver, portal vein, and hepatic artery may each be displayed in different colors. If blood vessels are grouped together, they may be displayed together in the same color.

[0048] Next, in step S13, the image processing device 301 detects the position of the hand so that the distance between the motion sensor 380 and the operator's hand is appropriate. Specifically, the operator positions their hand above the motion sensor 380. The appropriate distance between the sensor and the operator's hand (height from the sensor to the hand) is preset to a range such as h1 (mm) to h2 (mm). The reason for pre-setting this appropriate range is that if the operator's hand is too close or too far from the sensor, the hand movement may not be recognized properly.

[0049] Regarding the detection of hand position, there are no particular limitations on what image is displayed on the screen, but one example is shown in Figure 6. In this example, a reference circle (first circle) 391 of a predetermined size is displayed approximately in the center of the screen. The first circle 391 is always displayed at a fixed size, regardless of the position of the operator's hand. Meanwhile, a second circle 393 is also displayed on the screen.

[0050] The center of the second circle 393 corresponds to the position of the operator's hand. That is, if the operator's hand is directly above the motion sensor 380 (for example), the center of the second circle 393 will be the same as the center of the second position circle 391. In other words, the two circles 391 and 393 will be displayed as concentric circles.

[0051] If the operator's hand is shifted in a predetermined direction (for example, to the right) from the position directly above the motion sensor 380, the second circle 393 will also shift in the same direction (for example, to the right) in response and be displayed in real time.

[0052] This display configuration allows the operator to check whether their hand is in the correct position (horizontal position) relative to the motion sensor 380 by observing the relative positions of the two circles 391 and 393 on the screen.

[0053] The position in the height direction can be confirmed as follows: The diameter of the second circle 393 corresponds to the height of the operator's hand. For example, if the hand height is the standard height (for example, (h1+h2) / 2), the second circle 393 will be displayed so that its diameter is the same as the diameter of the first circle 391. As the hand position rises, the size of the second circle 393 will decrease accordingly, and conversely, as the hand position falls lower, the size of the second circle 393 will increase accordingly. This display method allows the operator to check whether their hand height is appropriate by looking at the relative sizes of the two circles 391 and 393 on the screen.

[0054] To make it easier to confirm that the correct position is being used, the following indication may be used. That is, when the horizontal position, height position, or combination thereof of the hand is in a predetermined correct position, the second circle 393 may be indicated with a special indication. For example, in one embodiment, it is preferable to indicate the second circle 393 with a different color depending on whether it is outside the correct range as in Figure 6(a) or within the correct range as in Figure 6(b), or to switch between a flashing indication and a steady-on indication.

[0055] As described above, the step of ensuring that the distance between the motion sensor and the operator's hand is appropriate is completed (step S13).

[0056] Next, in step S14, voice input for display mode selection is accepted. For example, the voice input may recognize words such as the following: - "Move" -"Stop" -"rotate" - "Multi"

[0057] Furthermore, the voice recognition function may be triggered to turn ON when the operator's hand position enters a predetermined appropriate range in step S3. In this configuration, the voice recognition function is OFF when the operator's hand position is not within the predetermined appropriate range, and is only ON when it is within the appropriate range. This configuration, in which the voice recognition function is only turned ON under predetermined conditions, makes it possible to prevent voice input due to unintended misrecognition by the operator.

[0058] As schematically shown in Figure 1, it is also preferable that a message such as "Voice Recognition in Progress" is displayed on the screen to inform the operator that the voice recognition function is turned ON.

[0059] In one aspect of the present invention, the step of detecting the hand position in S13 may be omitted.

[0060] <Zoom (enlarge and reduce) / Pan> To change the size or position of the displayed 3D medical image, follow these steps:

[0061] First, the operator says "move" with the voice recognition function turned ON. The image processing device 301 analyzes the voice input from the microphone 370 with the voice recognition unit 351 and recognizes the word "move". Accordingly, it transitions to the "zoom / pan" mode (step S15).

[0062] In "zoom / pan" mode, the image processing device 301 then waits for motion input from the operator's hand. The image processing device 301 uses the motion sensor 380 and the motion recognition unit 353 to recognize the position and movement of the operator's hand in real time. Then, when the operator moves their hand upward (i.e., the hand moves from an initial height h0 to a higher height h0), h If the user moves their hand downwards (from the initial height h0 to a lower h0), the image will gradually shrink in accordance with the movement. LIf the viewer moves to a different location, the medical image will gradually zoom in to match that movement.

[0063] Furthermore, when the hand is moved horizontally, the 3D medical image is panned (translated) in accordance with the direction and amount of that movement.

[0064] As described above, in this mode, the image is zoomed in or out by moving the hand up and down, and the image is moved in parallel by moving the hand horizontally. With the configuration of this embodiment, compared to input methods such as voice recognition or numerical input, motion input, which allows for intuitive analog input, can be used to perform zoom / pan (and even rotation, as described below) of the image. Therefore, it is intuitive and easy to operate for the operator and is also useful in practice.

[0065] Furthermore, since the motion sensor 380 allows input without contact with the device, input can be performed while maintaining the cleanliness of the operator's hands. This configuration is highly advantageous, for example, because it allows a physician to use an image processing device to review images in the operating room during surgery.

[0066] In the case of the liver in particular, multiple blood vessels are present in a branching manner. Therefore, when removing a portion of the liver parenchyma to avoid unnecessarily damaging the blood vessels, it is necessary to carefully confirm the positional relationship between the blood vessels and the tumor. In this regard, the image processing device of this embodiment allows for confirmation of the positional relationship of blood vessels, etc., while viewing a 3D medical image during surgery. Furthermore, in a 3D medical image, for example, there may be cases where a blood vessel is in the foreground and a tumor is hidden behind it (the tumor is not shown, but see Figure 5 for reference). Even in such cases, the image processing device of this embodiment allows for confirmation of the tumor by rotating the image in "rotation" mode. In particular, the configuration of this embodiment does not rotate at predetermined angles, but rather allows for free (stepless) rotation by any angle using motion input, enabling good observation.

[0067] In one configuration, it is preferable that the "rotation" of the 3D medical image (details below) is not performed while in this "zoom / pan" mode. Often, when using this mode, the user simply wants to zoom in or out or pan the image. Therefore, it is more user-friendly for the operator if rotation is prohibited and zooming, panning, and panning are performed while maintaining the desired viewing position.

[0068] The above description explains a mode that allows both zooming and panning, but it is not limited to that. It is also possible to have a mode that allows only one of them.

[0069] <Rotation> To rotate the displayed 3D medical image, follow these steps: First, the operator says "Stop" to deactivate the "Zoom / Pan" mode. The image processing device 301 receives this via its voice recognition function, deactivates the "Zoom / Pan" mode, and transitions to a state where it can accept other modes.

[0070] In this state, the operator says "rotate". The image processing device 301 receives this via its voice recognition function and transitions to "rotate" mode. The image processing device 301 then waits for motion input from the operator's hand.

[0071] In "rotation" mode, the image processing device 301 recognizes the position and movement of the operator's hand in real time. The image processing device 301 then rotates the 3D medical image around predetermined rotation axes (X axis, Y axis, Z axis) in accordance with the operator's hand movements. Specifically, it recognizes the horizontal movement of the operator's hand, or movements such as moving the hand along the surface of a virtual sphere. Then, it rotates the 3D medical image by a predetermined angle in accordance with the direction, speed, and amount of movement.

[0072] In this rotation mode, it is preferable in one embodiment that only rotation is permitted, and panning (translation) and zooming (enlargement / reduction) are prohibited. This makes it possible, for example, to rotate an image to a desired orientation while maintaining a predetermined image size, and then perform predetermined image processing or observation.

[0073] To deactivate "rotation mode," the operator must say "stop," as described above.

[0074] In the explanations so far, we have described a scenario where, if either "movement" or "rotation" is performed, the other is not performed. However, a "multi" mode may be provided to allow both inputs to be performed simultaneously. In this mode, "zoom," "pan," and "rotation" are all performed in response to the operator's hand movements.

[0075] Specifically, when the operator says "multi," the image processing device 301 recognizes this and switches to "multi" mode. The device then rotates, moves, and enlarges / reduces the 3D medical image in response to the operator's hand motion input.

[0076] Furthermore, a function that allows rotation of 3D medical images using only voice input, rather than motion input, may be implemented. For example, by saying "rotate," "left," and "15°," the image processing device 301 recognizes this. Then, it rotates the image by 15° around a predetermined rotation axis (for example, the Z-axis extending in the vertical direction of the screen). To rotate the image 15° upwards around an axis extending in the horizontal direction, for example, one can input "rotate," "up," and "15°" using voice input.

[0077] <Other voice input methods> The image processing device 301 of this embodiment can also select anatomical structures and change the transparency of selected structures via voice input. This will be explained again after the explanation of operation via the touch panel.

[0078] (Regarding various functions via touch panel input) The image processing device 301 displays anatomical structures in the three-dimensional medical image as independent objects. This allows each to be selected individually or its display to be toggled on or off. Blood vessels, for example, the hepatic artery, portal vein, and hepatic vein may be grouped together for selection, or they may be selected individually.

[0079] (Rotation function) The rotation of the 3D medical image can also be performed by operation on the touch panel. When the operator touches the touch panel and moves their finger, the image processing device 301 rotates the 3D medical image accordingly.

[0080] (Zoom in / out function) The image processing device 301 may also enlarge or display an image in response to an operation performed by the operator, such as touching two points on the screen and increasing or decreasing the distance between those two points (pinch-out operation, pinch-in operation).

[0081] (Display transparency switching function) The image processing device 301 may change the display density of an anatomical structure (e.g., the liver) when the operator touches it. Specifically, it may switch between two states: a normal opaque display state and a semi-transparent state. For example, one touch may result in a semi-transparent state, and another touch may return to the normal display state.

[0082] In other configurations, the transparency may be set to multiple levels, such as 0%, 30%, 70%, and 100% (hidden), and the display density may cycle through sequentially with each touch. In this case, 100% transparency (i.e., hidden state) may be excluded from this loop. Naturally, the specific numerical values ​​of the transparency can be changed as appropriate. In short, it is sufficient that the transparency is set to at least multiple levels and that they cycle through.

[0083] This display transparency switching function allows users to switch the transparency of any anatomical structure simply by touching it. Therefore, it is simpler and more intuitive to use compared to methods that require selecting an icon or command to switch transparency.

[0084] Furthermore, when the display switches in a loop as described above, it is preferable because it eliminates the need to select an icon or other element separately to return to the original display state. Moreover, since such loop-like display switching can be achieved simply by changing the display color of the selected object, it is also preferable because it simplifies image processing and enables computation with less memory.

[0085] The gestures used to change transparency are not limited to those described above. For example, the image processing device may accept input when a finger (for example) is swiped up or down or left or right, and the transparency may be changed accordingly. In this case, the transparency may be set in several steps, such as 0%, 30%, 70%, and 100% (hidden), or it may be configured to change continuously without stepwise transitions.

[0086] (Show / hide toggle function) The image processing device 301 sets a predetermined anatomical structure to a "selected state" if the operator has been touching it for a certain period of time or longer (for example). To indicate that it is in a "selected state," the anatomical structure (e.g., the liver) may be displayed in a different color than its initial state or may be made to blink.

[0087] The image processing device 301 hides an anatomical structure (e.g., liver) that has been selected, when the operator moves their fingertip (for example) towards the edge of the screen while touching the selected structure (e.g., by swiping or dragging). In this example, the liver is hidden, and only the three-dimensional image of blood vessels, etc., remains.

[0088] Such a function is useful when the user wants to see only the desired anatomical structures. Furthermore, a method like that of this embodiment, which allows users to hide anatomical structures simply by directly selecting and moving them, is simpler and more intuitive to use than a mode where the display can only be toggled on or off by selecting, for example, an icon.

[0089] (Cutting function) The cutting function is performed as follows. An example of cutting the liver and then hiding a portion of it is described below. Figure 8 is a flowchart of the sequence of operations.

[0090] First, the image processing device 301 displays a three-dimensional medical image as shown in Figure 7A as step S1. Then, when the operator touches two points on an arbitrary anatomical structure (in this case, the liver 71) as shown in Figure 7B, the image processing device 301 determines that the two points have been touched (step S2). The timing of this determination may be that the two points are touched simultaneously or substantially simultaneously.

[0091] Next, in order to enable the function in the case of a so-called long press, the image processing device 301 determines whether the state of two points being in contact has continued for a certain period of time or longer (step S3).

[0092] If the image processing device 301 determines in step S3 that the action has continued for a certain period of time or longer, it displays on the screen in a predetermined display manner that the two points P1 and P2 that were touched have been designated. The "predetermined display manner" can be anything, but for example, it may be (i) displaying both points P1 and P2 and the line L1 connecting them, or (ii) displaying only points P1 and P2 or only line L1. Regarding points P1 and P2, in order to make the designated positions clearly visible, they may be displayed not as simple small dots, but as a slightly larger graphical image as shown in Figure 7B (for example, any shape such as a circle, rectangle, polygon, or star, but a circle is used as an example here).

[0093] The image processing device 301 may continue to display the designated points P1 and P2 even after the operator takes their hand off the screen, as shown in Figure 7C. It may also be configured to accept fine adjustments to the positions of points P1 and P2. To indicate that this fine adjustment mode is in operation, for example, the circular graphical images of P1 and P2 and / or line L1 may be displayed with blinking. In Figure 7C, as an example, point P2 is shown slightly moved and fine-tuned to point P2'.

[0094] This fine-tuning may be performed by the operator, for example, moving the graphical images of points P1 and P2 with their finger (operation on a touch panel). Alternatively, motion input may be used to fine-tune the positions of points P1 and P2 without touching the device. The display of the cutting reference line L1 using voice input will be described again later. Here, we will first explain the cutting function and other features of this embodiment, assuming touch panel operation.

[0095] After specifying points P1 and P2 in this way, in step S4, for example, the operator touches a predetermined icon on the screen (for example, an icon for "OK" input). Then, the cut function cuts the liver along the line L1 connecting points P1 and P2, as shown in Figure 7D (step S5).

[0096] The first part 71-1 and the second part 71-2, which are divided into two parts by line L1, can each be manipulated as independent anatomical structures. In addition to the above operations, the above functions may also be performed by, for example, (i) touching a designated area on the screen instead of touching the icon, or (ii) performing unconventional operations such as touching multiple times (for example, double-tapping). Voice input may also be used.

[0097] Since it can be manipulated as an independent anatomical structure, for example, if you touch the first part 71-1 (step S6), the function described above will select only that part, as shown in Figure 7E. Then, the display density will switch. Specifically, only the first part 71-1 will be displayed semi-transparently. Touching it again will return it to the original display.

[0098] Furthermore, if, for example, the first area 71-1 is long-pressed and then swiped or dragged towards the edge of the screen, that area will become invisible, leaving only the second area 71-2 and the blood vessels 73 and 75. The invisible areas may be displayed as thumbnail images 66, as illustrated in Figure 7F.

[0099] (area specification) The image processing device 301 designates a portion of an anatomical structure as a region through the operator's actions as follows. Figure 9A shows the state where two points P1 and P2 are touched, as described with reference to Figure 7B (the operator's fingers remain touching the two points on the screen, but this is omitted from the illustration).

[0100] From this state, if the operator then moves two fingers (or two fingers simultaneously, though not limited to simultaneous movement), as shown in Figure 9B, the image processing device 301 identifies the positions of the two points P1' and P2' after the movement and designates a roughly rectangular area based on these positions. Specifically, it designates a rectangle enclosed by the four points P1 and P2 before the movement and P1' and P2' after the movement.

[0101] Here as well, it is preferable that the image processing device 301 is configured such that, even after the operator releases their hand, the roughly rectangular area of ​​the designated region remains on the screen, and that the position of each point P1, P2, P1', and P2' can be individually moved to fine-tune the position. As a method for confirming the designated region, for example, the operator may touch a predetermined icon on the screen (for example, an icon for "OK" input).

[0102] As shown in Figure 9B, when the positions of four points P1, P2, P1', and P2' are specified, a graphical image of a circle (for example) of points P1, P2, P1', and P2', or the lines connecting them, may be displayed with blinking lights to allow for fine adjustment of each position.

[0103] The designated region Sa1 (see Figure 9B) is separated from the rest of the area and can be manipulated as an independent object. Therefore, it is possible to change the display density of only the region or to toggle its display on and off. With this function, for example, by hiding only region Sa1, it becomes possible to observe the inner blood vessels 73 and 75, and to check the relationship between blood vessels 73 and 75 and the liver 71.

[0104] Furthermore, the area designation does not necessarily have to be a rectangle; a triangle or a polygon with pentagons or more may also be used for the area designation.

[0105] In the above explanation, medical images of the liver and its surrounding areas were used as examples, but naturally, the anatomical structures in this invention are not limited to specific ones. For example, a medical image of the examiner's head may be displayed, and various image processing operations may be performed on it.

[0106] [Other functions using voice input] (Display / Emergency Switch) It is also desirable that the display / hide switching and cut functions described above can be performed using only voice input or similar methods, without requiring any input on the touch panel.

[0107] First, the selection of a predetermined anatomical structure is performed not by touch, but by speaking the name of the object, and the selection is made when the voice is recognized. For example, if the operator says "liver," the image processing device 301 recognizes it as voice and sets the liver to be selected. To indicate that it is "selected," the anatomical structure (e.g., liver) may be displayed in a different color than its initial state or may be made to blink.

[0108] Then, if the operator wants to change the transparency of the liver, they can say, for example, "transparent." The image processing device 301 recognizes this as voice and switches the display of the liver to a semi-transparent state. situation In this configuration, other anatomical structures (such as blood vessels and tumors) remain opaque and continue to be displayed. This configuration allows for the identification of blood vessels and other structures that would normally be hidden by the liver and therefore invisible.

[0109] In one embodiment, for example, the transparency may be configured to change according to the distance from the motion sensor to the operator's hand. That is, the transparency may gradually increase (or decrease) as the hand is brought closer to the motion sensor, and conversely, the transparency may gradually decrease (or increase) as the hand is moved away from the motion sensor. Specifically, when the operator says "transparent" and the image processing device 301 recognizes it as voice, it enters a mode that accepts motion input as described above. Next, the device detects the distance from the motion sensor to the operator's hand and changes the transparency accordingly.

[0110] (Line cut / Box cut) When performing a line cut, for example, the operator might say "line cut." The image processing device 301 recognizes this as voice and displays a reference line for cutting on the screen. This reference line may be something like line L1 in Figure 7B.

[0111] The image processing device 301 then waits for motion input. The operator can change the position, length, orientation, etc. of the cutting reference line based on the motion input. This allows the reference line to be set to a predetermined position without physical contact.

[0112] Next, for example, "right cut" removes the area to the right of the reference line. Conversely, "left cut" removes the area to the left of the reference line. Instead of removal, the area can be displayed semi-transparently. Figure 5(b) shows an example of this, where the area to the right of the reference line, labeled 371-2, remains opaque, while the area to the left, labeled 371-1, is displayed semi-transparently.

[0113] When performing a box cut, for example, the operator might say "box cut." Although detailed illustrations are omitted, the image processing device 301 recognizes this voice and displays a rectangle (one example) on the screen that will serve as the basis for the cut. The size of the rectangle may be limited to one predetermined size, or multiple sizes such as large, medium, and small may be available.

[0114] This box cut excises the target anatomical structure to a predetermined depth. The image processing device 301 waits for motion input while displaying a rectangle on the screen that serves as the basis for the excision. The size and shape of the rectangle may be fixed or freely changeable. For example, the configuration may allow the position of the corners of the initially displayed default rectangle to be moved, thereby enabling changes to the size and shape of the rectangle. Motion input can be used to move the position of the corners.

[0115] In this state, if the operator moves, for example, their hand closer to the motion sensor 380, a roughly rectangular hole with a predetermined depth corresponding to the distance the hand moves is formed in the liver, with the rectangle as its outline. This makes it possible to obtain a medical observation image in which a portion of the liver is removed, while the internal blood vessels are not removed.

[0116] Furthermore, with the excised portion of a predetermined depth formed in this manner, it is also possible to rotate the entire 3D medical image by a predetermined angle. For example, if the operator speaks "up" or "15°" in rotation mode, the image processing device 301 recognizes this and rotates the medical image with the hole formed by 15°. This configuration is useful because it allows observation of the internal structure of the hole (for example, part of the liver is excised, but blood vessels are still visible) from different angles.

[0117] Although the above explanation described removal using a rectangular outline, it is naturally possible to define the outline using a triangle, polygon, circle, ellipse, or any other geometric shape.

[0118] The above method involved removing the area designated as a box, but conversely, it is also possible to configure the system so that only the area designated as a box remains, and the rest of the area is hidden.

[0119] [Section B: Comprehensive treatment of multiple anatomical structures] 1. Problems of the inventions in this section As illustrated in Figure 5, volume data and the like are used for three-dimensional medical images, as previously described. However, in such three-dimensional medical images (i.e., those containing several different types of anatomical structures), the CT values ​​(signal values) of the liver and blood vessels are different. Furthermore, even within the same blood vessel, for example, arteries, veins, and portal veins each have different CT values ​​(signal values).

[0120] This is because, in fluoroscopic imaging using contrast agents, image data is acquired by performing fluoroscopic imaging after a predetermined time has elapsed since the injection of the contrast agent. However, differences in the time it takes for the contrast agent to reach each part result in differences in CT values ​​(signal values) for various parts such as arteries, veins, portal veins, and liver parenchyma. Conventional methods involved setting thresholds and performing filtering for each blood vessel and organ to create volume data for each part.

[0121] However, even for the same blood vessel, if arteries, veins, and portal veins are registered separately based on differences in CT values ​​(signal values), a relatively time-consuming process may be required in the following cases. Specifically, some 3D medical image viewers allow switching between displaying blood vessels with or without emphasis. This allows, for example, the display of the peripheral parts of blood vessels (where CT values ​​(signal values) are low) to be switched on or off as needed, enabling observation as required (see Figure 10). However, in the case of 3D medical images containing multiple blood vessels of different types, the operator cannot uniformly change the display of all blood vessels without resetting the display threshold or filtering for each blood vessel, which presents a problem as it is difficult to easily make such display changes.

[0122] In contrast, in actual observations, it is sometimes preferable to be able to handle the images by separating them, for example, the vascular system as the vascular system and the parenchyma as the parenchyma. Therefore, in the invention described in this section, the image processing device has the following functions.

[0123] 2. Functions and Operation The image processing device of this embodiment reads volume data based on information obtained by imaging a patient (see also step S1 in Figure 3).

[0124] The system then analyzes the signal values, CT values, and standard deviation (SD) within the volume. For example, if the average CT value is 300 HU or higher, it is automatically determined to be an artery, and if the average CT value is 100 HU or lower, it is automatically determined to be an organ. In addition to the CT value, the histogram shape of the CT value is also recognized. Generally, arteries tend to have high peaks and narrow widths (distributions), while portal veins and veins tend to have low peaks and wide widths (distributions). Therefore, based on these factors, automatic recognition of the type of blood vessel can be achieved.

[0125] As described above, it becomes possible to automatically distinguish and register data for arteries, veins, portal veins, etc., which inherently have different CT values ​​(signal values), using an image processing device. Alternatively, different colors may be automatically assigned to arteries, veins, portal veins, etc., and displayed in separate colors.

[0126] Furthermore, while there is originally one histogram for each blood vessel (artery, vein, portal vein, etc.), these may be integrated and normalized so that a single histogram represents all blood vessels (or any two or more in other embodiments). Specifically, as an example, the mean value and centroid of each histogram may be calculated, and the entire histogram may be shifted to match the higher values ​​to create a single histogram.

[0127] When multiple blood vessels (or other anatomical structures) are represented in a single histogram in this way, the display can be changed collectively by manipulating only the single histogram, rather than manipulating the histogram of each individual blood vessel. In other words, for example, if the peripheral parts of a blood vessel do not need to be displayed, the image processing device accepts a predetermined input from the operator and processes the data so that parts below a certain threshold value are not displayed (or parts below a certain threshold value are not displayed), thereby collectively hiding the peripheral parts of arteries, veins, portal veins, etc. (see, for example, Figure 10(b)). On the other hand, if you want to highlight and display the peripheral parts of a blood vessel, you can set the lower limit of the CT value to be displayed lower, as shown in Figure 10(a) (where the threshold is 130HU).

[0128] With this configuration, there is no need to change the display method for each of the arteries, veins, portal veins, etc. Kuna Therefore, it is very easy to operate and is very practical.

[0129] The aforementioned "predetermined input from the operator" may, for example, be performed by manipulating image buttons such as icons, cursors, or sliders on the screen. Alternatively, it may be performed by recognizing a predetermined gesture of the operator's finger on the touch panel and acting based on that.

[0130] For example, in a mode for changing the display of blood vessels, the system may be configured so that the display of blood vessels changes when the operator touches the touch panel with several fingers and moves the fingers simultaneously in a predetermined direction. More specifically, if several fingers are moved simultaneously upwards on the screen (first direction), the peripheral parts of the blood vessels will be displayed, and conversely, if the fingers are moved downwards (second direction), the peripheral parts of the blood vessels (more precisely, the area around the outer edge of the thicker parts of the blood vessels) will disappear.

[0131] It is also preferable that the above operations be performed not by touch panel operation, but by motion input via the motion sensor 380. In other words, in this configuration, the display of blood vessels (one example; other anatomical structures may also be used) can be switched using only voice input and motion input, so there is no need to touch the touch panel, and 3D medical images can be observed while maintaining cleanliness.

[0132] [Section C: Other Features] (1) 3D-Pointer Examples of how 3D medical images can be used, as illustrated in Figure 5, include multiple medical professionals reviewing the course of blood vessels (one example) before surgery and simulating the actual procedure.

[0133] In the case of two-dimensional medical images, for example, a pointer can be displayed on the screen as an image, and by positioning it on a specific area, that area can be focused on. However, with three-dimensional medical images, the pointer needs to be placed not in a plane, but at any point in three-dimensional space. Performing such operations, where the pointer is moved to any point in three-dimensional space, is relatively difficult with input interfaces such as mice and touch panels.

[0134] Therefore, in this embodiment, motion input may be used to arrange the pointer in three dimensions. Specifically, the medical image processing device 301 first receives input such as "pointer" (for example) using its voice recognition function. Then, it displays a pointer on the screen, for example, in three dimensions.

[0135] For vertical and horizontal movement on the screen, the pointer should be moved in accordance with the horizontal movement of the operator's hand. For depth, the pointer may move towards the back of the 3D medical image when the operator moves their hand closer to the motion sensor 380, and towards the front when the operator moves their hand away.

[0136] Regarding the display of the pointer, a display method may be adopted in which the display size gradually decreases as the pointer moves towards the back and gradually increases as it moves towards the front.

[0137] (2) Exporting schematic images In diagnosis and examination, two-dimensional diagrams called schematic images, which represent parts of a patient's body, are sometimes used. Therefore, an image processing device according to one embodiment of the present invention may also be equipped with a function for writing schematic images.

[0138] For example, when an image processing device receives a predetermined input from an operator, it uses data from a 3D medical image (see, for example, Figure 5) to create a corresponding schematic image. The schematic image can be any 2D image, such as a line drawing, a monochrome image, or a color image. The predetermined input from the operator can be various types of input, such as touching an icon on the screen, voice input, or a predetermined gesture input via a motion sensor.

[0139] An example of creating a corresponding 2D schematic image using 3D medical image data is to take a medical image displayed in an orientation like that shown in Figure 5 (one example) and convert the currently displayed image directly into 2D and export the data. In this case, contour extraction processing may be performed to create a line drawing. The data format can be anything, but for example, PDF (Portable Document Format) or any other image format such as GIF, PNG, or JPEG can be used. Doctors can then write sketches, findings, etc. on the schematic image created in this way, for example, using a stylus or their finger.

[0140] The schematic images created by the image processing device can be sent externally from the device and stored in a predetermined storage area connected to a network (see Figure 3). For example, they may be incorporated as part of an electronic medical record.

[0141] (3-1) Confidentiality The image processing device of this embodiment is portable, such as a tablet device, and can be taken outside the hospital for use in some cases. Such a configuration can be useful, for example, when performing a procedure simulation while viewing a 3D medical image of a specific patient outside the hospital. However, from a security standpoint, it is necessary that the internal information be kept confidential when taking the device outside the hospital.

[0142] Therefore, it is preferable that one embodiment of the present invention has the following functions: (a) a function to recognize the current location of the device; (b) a function to determine whether or not the device is located outside the hospital (or whether or not the device is located inside the hospital) based on the current location; and (c) a function to automatically hide predetermined information held by the device when it is determined that the device is located outside the hospital (or that it is not located inside the hospital).

[0143] Information that is to be hidden includes, for example, information that contains at least information that can identify a patient (identification information). Furthermore, instead of simply displaying the information, the information in question may be encrypted or access to that information may be prohibited.

[0144] One possible method for determining whether an area is inside or outside a hospital is to use whether or not it is within the range of the hospital's wireless network system.

[0145] Thus, a configuration in which specific information is automatically concealed when the device is taken outside the hospital contributes to preventing the leakage of patient information and other sensitive data, making it more desirable from a security standpoint.

[0146] Furthermore, the effects of such concealment are not necessarily limited to inside or outside a "hospital." Such concealment may also be implemented in a specific area (which may be any facility or location).

[0147] (3-2) Regarding anonymization, the following features may also be included. Figure 12 schematically shows the state in which anonymization is implemented. In this screen, all information that could identify the patient is anonymized. Also, icon 441 is displayed on the screen.

[0148] Incidentally, for doctors and other medical professionals viewing 3D medical images, there may be times when they need to temporarily check patient information, such as when they want to confirm which patient the image belongs to.

[0149] Therefore, the medical image processing device in this example first detects that icon 441 has been pressed (Figures 12 and 13), and then displays patient information. The patient information may be data stored inside the device, or it may be data obtained by accessing an external server (for example, a server within the hospital system).

[0150] More specifically, it is preferable that the conditions under which such patient information can be displayed are limited to certain conditions. For example, when the device is authenticated by the operator's fingerprint. Of course, it is also acceptable if the user is authenticated by other authentication methods.

[0151] For example, it is preferable that communication with external servers be configured to be possible only under secure communication conditions such as a VPN (Virtual Private Network).

[0152] The information displayed may include, for example, one, two, or three or more items from the following: patient's initials, date of birth, gender, age, address, date of surgery, attending physician, etc. Patient ID, test ID, etc., may also be displayed.

[0153] In situations like those described above, when reviewing patient information, it is preferable in one form that the system displays only the minimum necessary information about the patient (for example, information that can identify the patient and also identify the date of surgery or the doctor).

[0154] The displayed patient information may be automatically hidden again after a certain period of time. Alternatively, once displayed, it may remain displayed throughout the operation (for example, until the operation is completed (logged out)).

[0155] As described above, by allowing users to check minimal patient and / or surgical information as needed, the possibility of problems such as mistaking the displayed 3D medical image for the actual patient undergoing surgery can be reduced.

[0156] Furthermore, with regard to the above-mentioned features, this specification discloses not only the invention of the apparatus, but also the invention of methods and programs corresponding to the above-mentioned features.

[0157] (4) Stereo images A medical image processing device according to one embodiment of the present invention may have a function for displaying stereo images, as described below.

[0158] As shown in Figure 11, the image may be displayed including a first image 431L and a second image 431R so that the operator can view it in stereoscopically. The first image 431L and the second image 431R display the same subject with a predetermined parallax. They are sometimes referred to as the left-eye image and the right-eye image, etc.

[0159] As shown in Figure 11, an operation pad area 433 may be displayed within the screen containing images 431L and 431R. This operation pad area 433 is an area for changing the display angle of the subject. When the operator touches and moves their finger within this area 433, the medical image processing device simultaneously changes the display angle of the subject (the two images) accordingly. In other words, the orientation of the subject can be changed in conjunction with the movement of the finger.

[0160] While the above operations are intended for use with a touch panel, the input method is not limited to this and various other methods can be used. For example, one or more of the contactless input methods disclosed herein can be used.

[0161] As shown in Figure 11, in an configuration where a single operation pad area 433 is displayed together with the first image 431L and the second image 431R, the operator can intuitively understand that they can change the orientation of the subject image by operating that area.

[0162] The subject image is not particularly limited, but angiographic images are also acceptable.

[0163] As described above, with the configuration that displays the first image 431L and the second image 431R, doctors can more accurately grasp the three-dimensional structure of the subject through stereoscopic vision.

[0164] An apparatus of one embodiment of the present invention described above has the function of displaying a first image and a second image having different parallaxes. More specifically, it further has the function of displaying an operation pad area for manipulating the display angles of these images. For changing the display of these stereoscopic images, one or a combination of, for example, gesture input, voice input, motion input, etc., can be used. This specification also discloses inventions of methods and programs corresponding to the above.

[0165] (5) Image creation flow One possible method for creating medical images is the following: In this method, the patient's fluoroscopic images are first stored in a designated data server (for example, a DICOM server: which stores data received from modalities in a predetermined format).

[0166] Then, on the designated data server (or another computer), the individual parts of the subject are automatically recognized and managed as separate, independent objects. As shown in the table illustrated in Figure 14, they may be managed by dividing them into categories such as arteries, veins, bones, organs, etc. (and even further subdivided categories). For each individual object, information may be set for when it is selected by the voice recognition function (for example, for the aorta, the voice input may be "aorta").

[0167] Additionally, automatic identification keys (symbols, letters, numbers, or combinations thereof) may be set up to facilitate the selective loading of only the necessary individual objects.

[0168] Additionally, offset values ​​may be set for each individual object. In this example, the offset values ​​(see "CombineOfs" in the table) are set as follows: for example, "+50" for the aorta, "+100" for the abdominal artery, and "+400" for the vein.

[0169] Offset values ​​are used in the following way, for example. Suppose the central CT value for an artery is 350 HU and the central CT value for a vein is 200 HU. As such, the degree of contrast enhancement differs between arteries, veins (and even portal veins). Therefore, offset values ​​are used to standardize these differences. 。 In the example above, by setting the venous offset value to +150 HU, the contrast effect is virtually increased, making it possible to treat arteries and veins with the same threshold setting. Of course, offset values ​​can be set not only for arteries and veins, but also for other blood vessels such as portal veins.

[0170] Furthermore, the following uses may be applied to the offset values ​​of the parenchymal system rather than the vascular system. Here, as an example, suppose the offset value of a parenchymal organ (for example, the liver) is set to a large value such as +700HU. Setting such a large value for a parenchymal organ has the advantage that the organ shape is more easily maintained, as shown below.

[0171] In other words, for example, if the display of the artery's periphery is unnecessary, one might expect to raise the threshold so that the periphery is not displayed. However, in this case, the display of parenchymal organs such as the liver would also change accordingly, causing their shapes to become distorted. To prevent this (i.e., to change the display of blood vessels while keeping the parenchymal organs in their original form), parenchymal organs may be displayed with a large offset value such as +700HU.

[0172] (Data conversion) When blood vessels, bones, organs, etc., are managed as separate objects on a designated data server (for example, a DICOM server) (configured in a table, for instance), the following advantages are also available. Specifically, when creating medical images on a tablet device, it becomes possible to easily create medical images without having to manually input information for each individual object (such as text for speech recognition or offset values ​​for image synthesis).

[0173] In this embodiment, it is basically necessary to create medical images specifically for the medical image processing device. In this case, having a setting table like the one described above is preferable as it reduces the effort required for creation. However, it is conceivable that it may be preferable to have several tables prepared depending on the surgical procedure (what kind of surgery is performed, etc.). This is because in one surgical procedure, it is preferable that blood vessel A (or organ A) be clearly visible, but in another surgical procedure, it is preferable that blood vessel B (or organ B) be more clearly visible than blood vessel A (or organ A). In other words, it is preferable in one form to have several tables registered with offset values ​​set according to each surgical procedure.

[0174] In this case, the system or apparatus may also be configured as follows: (i) to display multiple surgical procedures, (ii) to accept that the operator has selected one of them, and (iii) to call up (display, if necessary) the corresponding table.

[0175] The display of the surgical procedure may, for example, be done according to the mode (site selection) for selecting which part of the body the procedure is to be performed on in the user interface for setting the injection conditions. For example, the configuration may display the surgical procedure corresponding to the selected predetermined site (head, chest, abdomen, etc.) (see (i) above).

[0176] ( 6 ) In combination with physical switches, etc. The embodiments described above utilized voice input and motion input (input corresponding to human movement). In one embodiment of the present invention, a predetermined physical switch may be used in combination with these inputs.

[0177] Specifically, switches that detect physical contact (such as foot switches) can be used. A "foot switch" is, for example, one that is placed on the floor and has a switch housing on which sensors and circuit boards are placed, and a pressing part that is pressed by a foot or the like. The pressing part is not limited to, but may be a movable part that is pressed down when stepped on. The detection signal from the foot switch may be supplied externally by a cable (wired) or by wireless connection.

[0178] The foot switch may, but is not limited to, being electrically connected to the medical image processing apparatus of the present invention (for example, the control unit 350, see Figure 2). The foot switch may also be connected to other equipment (for example, provided as part of a drug injection device).

[0179] Figure 15 shows an example of foot switch configuration. In this example, the drug injection device comprises an injection head 475 located near the imaging device 470, a first control unit (power supply unit) 478 connected to it, and a console (second control unit) 476 connected to it. The foot switch 477d is connected to the power supply unit as an example.

[0180] (Examples of using physical switches) In one embodiment of the present invention, multiple acceptance conditions may be set for voice operation. A foot switch may be used as one of these acceptance conditions.

[0181] First, the following settings may be configured as multiple conditions for accepting voice commands (one or more): 1) Accept voice input only when motion input is detected. 2) Voice input is only accepted when the foot switch is ON. 3) Voice input is accepted only when motion input is detected and the foot switch is ON. 4) Always accept voice input.

[0182] Furthermore, the above input conditions may be registered in the device's voice control command table. In other words, input 1) means that the device is configured not to accept voice input when there is no motion input.

[0183] Regarding voice input, it is preferable that the input method be as follows: Specifically, the system accepts command input only when combined information of direction and angle is received, such as "45° left, 30° up." This is because the probability of misrecognition increases if the information is not a combination. A detailed explanation follows.

[0184] If voice input is enabled and it can recognize only single words such as "up" or "front," it is possible that it may recognize words from conversation during treatment, resulting in unintended input. Therefore, it may be better to configure the system to only accept commands when a combination of multiple words is recognized.

[0185] Such voice input methods to prevent misrecognition are not necessarily limited to combinations of direction and angle information as described above. As a concrete example, consider the case of moving an image forward. In this case, the command is accepted when two words are recognized: the word "direction" (for example) to identify that it is a movement mode, and the word "forward" (for example) which indicates the direction in which the image should be moved. From the perspective of preventing misrecognition, the system may be configured to accept inputs such as "direction" + "forward," but not the opposite input such as "forward" + "direction" (i.e., the order of the combinations is fixed). In addition, regarding the command to enlarge an image, the system may be configured to accept the command only when a combination such as "3D" + "enlarge" is recognized, rather than simply recognizing the word "enlarge." Such configurations can prevent unintended voice input and make the system more user-friendly.

[0186] As one form of voice input, in addition to or in conjunction with the above-described form, there may also be a configuration in which voice input is accepted only for a certain period of time (and voice input is not accepted outside of that period).

[0187] Regarding foot switch input, the foot switch may be designed to be ON only while the pressed portion is pressed. The processing for foot switch input may also be performed only if the foot switch is pressed for a certain period of time or longer (i.e., a long press). This prevents unintended actions from occurring due to pressing the foot switch unintentionally. For processes where such actions are a concern, the above-described input may be used. Furthermore, the above operation may be performed when another switch (physical switch) is ON, rather than the foot switch.

[0188] Regarding voice input, the following functions may be provided. Here, a typical image processing device according to one embodiment of the present invention is configured so that voice input is performed only in a predetermined mode suitable for voice input, and input is performed by another input method otherwise. In such a case, when a predetermined input is made (for example, by voice inputting "Voice All ON"), a function may be provided that enables voice input even for those inputs that are not originally set as the default (at least some of them). However, this extension of voice input is just one example, and it may automatically terminate after a certain period of time and return to the original state (timeout function). The timeout period may be a preset of 1 minute, 3 minutes, or 5 minutes, although this is not limited to this.

[0189] While not limited to this, the input for specifying the angle may always be configured to respond only to voice. Alternatively, a table of each voice input word may be prepared, and for each word, settings may be configured to allow input under certain conditions (for example, input of the word "aaa" will only be accepted when the foot switch is ON, while input of the word "bbb" will always be accepted).

[0190] (Home position setting function) Incidentally, the image processing device of the present invention may be configured to always display a medical image viewed from a certain orientation, regardless of the body part, as the default setting. For example, it may be configured to always display an image viewed from the front of the body, regardless of the body part and / or surgical procedure.

[0191] However, depending on the site and / or type of surgical procedure, it is also preferable in one form to have a configuration that defaults to displaying an image viewed from a pre-set angle that is preferable for that procedure. For example, in thoracoscopic surgery, the lateral decubitus position is the basic position. Therefore, in the case of thoracoscopic surgery, the system may be configured to default to display the lateral decubitus position as the home position.

[0192] In other words, in one embodiment of the present invention, depending on the location and / or type of surgical procedure, separate display angles suitable for each, not just a frontal view, may be preset and displayed as the default.

[0193] The selection of the home position may be set manually or automatically, depending on at least one of the surgical procedure, site, and tumor location. For example, whether the lesion is in the left or right lung determines whether the patient should be in the left or right lateral decubitus position. A configuration that automatically recognizes the location of the lesion (tumor) and automatically determines the appropriate position accordingly would also be useful.

[0194] (Example of screen display) The screen displayed when voice input is turned ON may be, for example, as shown in Figure 16. While this screen displays the example shown in Figure 6 with several additional images, it will be readily apparent to those skilled in the art that the images in Figure 6 (see reference numerals 391 and 393) may be omitted.

[0195] As shown in Figure 16, a "Voice Recognition (Voice Input) ON" indicator 397b may be displayed on the screen to indicate that voice recognition (voice input) is enabled. Similarly, a "Motion Input ON" indicator 397a may be displayed for motion input. It is also preferable to have a display unit 398 that displays the recognized voice as text. This makes it possible to visually confirm what kind of voice input has been received.

[0196] Furthermore, the device may have a display unit 399 that indicates whether or not the recognized voice was accepted as a command. If the command was not accepted, a message such as "REJECTED" may be displayed, allowing the operator to visually confirm that the command was not accepted.

[0197] Embodiments of the present invention have been described above with reference to the drawings, but the present invention can be modified in various ways without departing from its spirit, and is not limited to the above examples.

[0198] Although several technical features have been described above, these technical features can be used in combination as appropriate, except in cases where they are mutually contradictory, and this specification also discloses such combinations. Furthermore, even in places where several technical features are described as a single embodiment for the sake of explanation, it is possible to omit one or more of those features. Even if the technical content described does not distinguish between a product invention, a method invention, or a computer program invention, a person skilled in the art will understand that it can be understood as any of these inventions.

[0199] (Note) This specification discloses the following inventions (note that the symbols in parentheses do not limit the invention in any way): 1. Display (361) and, The control unit (350) connected to the display, Voice input device (370), Motion sensor (380) and, A medical image processing device (301) comprising, The control unit (350) is a: An image display unit that displays a three-dimensional medical image on the aforementioned display, b: The aforementioned audio input device( 3 A mode selection unit that recognizes the input voice using 70) and switches the mode for displaying the three-dimensional medical image accordingly, c: A display processing unit that recognizes the operator's motion input via the motion sensor and changes the display of the 3D medical image accordingly, Having, Medical image processing device. Or, a device or system having at least a control unit having the above-described characteristics.

[0200] 2. As the mode for displaying 3D medical images, Zoom mode allows you to enlarge and reduce the image. Pan mode to move the image horizontally, Rotation mode to rotate the image, The medical image processing apparatus described above, having at least one of the above.

[0201] 3. In the zoom mode, moving the operator's hand in the first direction enlarges the 3D medical image, and moving it in the second direction opposite to the first direction shrinks the 3D medical image. The medical image processing device described above.

[0202] 4. The medical image processing apparatus described above, wherein in the zoom mode, the 3D image is enlarged or reduced without rotation.

[0203] 5. In the rotation mode, the 3D medical image rotates in response to the operator's hand movements. The medical image processing device described above.

[0204] 6. The medical image processing apparatus described above, wherein, in the rotation mode, enlargement or reduction is performed without enlarging or reducing the 3D image.

[0205] 7. Equipped with a housing (301a), A portable medical image processing apparatus as described above, wherein at least the display and the control unit are integrally incorporated into the housing.

[0206] 8. The motion sensor is a medical image processing device as described above, having one or more cameras.

[0207] 9. The medical image processing apparatus described above, wherein the three-dimensional medical image includes image data of at least the liver and blood vessels.

[0208] 10. The medical image processing apparatus described above, wherein the audio input device is a microphone.

[0209] 11. On the computer, a: A process to display a 3D medical image on the display, b: Audio input device( 3 The process involves recognizing the input audio using 70) and switching the mode for displaying the 3D medical image accordingly, c: A process that recognizes the operator's motion input via a motion sensor and changes the display of the 3D medical image accordingly. A medical image processing program that performs this task.

[0210] 12. As the mode for displaying 3D medical images, image of Enlarge and reduce Let Zoom mode, Pan mode to move the image horizontally, Rotation mode to rotate the image, The program described above, having at least one of the above.

[0211] 13. The program described above, which instructs the computer to process the image so that when the operator moves their hand in a first direction, the 3D medical image is enlarged, and when they move their hand in a second direction opposite to the first direction, the 3D medical image is reduced.

[0212] 14. The program described above, which causes the computer to process the image so that, in the zoom mode, the 3D image is enlarged or reduced without being rotated.

[0213] 15. The program described above, which causes the computer to process the rotation of the 3D medical image in response to the operator's hand movements in the rotation mode.

[0214] 16. The program described above, which causes the computer to perform processing such that, in the rotation mode, the 3D image is enlarged or reduced without being enlarged or reduced.

[0215] 17. The computer displays a 3D medical image on a display, The computer, voice input device( 3 The steps include: recognizing the input voice using 70) and switching the mode for displaying the three-dimensional medical image accordingly; The computer recognizes the operator's motion input via a motion sensor and changes the display of the three-dimensional medical image accordingly. A method for operating a medical image processing device, including the following.

[0216] 18. The aforementioned modes for displaying 3D medical images are: image of Enlarge and reduce Let Zoom mode, Pan mode to move the image horizontally, Rotation mode to rotate the image, The method described above, comprising at least one of the above.

[0217] 19. The operation method described above, wherein in the zoom mode, the 3D medical image is enlarged when the operator moves their hand in a first direction, and the 3D medical image is reduced when they move their hand in a second direction opposite to the first direction.

[0218] 20. The operating method described above, wherein in the zoom mode, the 3D image is enlarged or reduced without rotation.

[0219] 21. The operating method described above, wherein in the rotation mode, the three-dimensional medical image rotates in response to the movements of the operator's hand. [Explanation of Symbols]

[0220] 1. Imaging device 10. Drug injection device 71, 371 Liver 73, 75, 375 blood vessels 301 Medical Image Processing Equipment 301a enclosure 350 Control Unit 351 Voice Recognition Unit 353 Gesture recognition unit 355a Image display unit 355b Operation judgment section 355c Display Processing Unit 359 Storage section 360° Touchscreen Display 361 displays 363 Touch Panel 365 Input Devices 367 Communications Department 368 Interfaces 369 slots 370 microphones 380 Motion Sensors

Claims

1. The display and A control unit connected to the display, A medical image processing device comprising: The control unit, This involves reading volume data, including multiple anatomical structures such as blood vessels and parenchymal organs, based on information obtained by imaging the patient, The CT values ​​in the volume data are analyzed, and the anatomical structures are determined based on the analysis results. Setting offset values ​​for the CT values ​​of the aforementioned plurality of anatomical structures, wherein the offset values ​​are set such that the CT values ​​of the blood vessels are uniform, and the offset values ​​for the CT values ​​of the parenchymal organs are set to be greater than the offset values ​​of the blood vessels, for each anatomical structure. The system receives a predetermined input operation from the operator for setting a threshold value for the CT value, and based on the received input operation, sets the lower limit of the CT value to be displayed on the display for the image of the anatomical structure. A medical image processing device configured to perform the following.

2. The medical image processing apparatus according to claim 1, wherein the display is a touch panel display.

3. The medical image processing apparatus according to claim 2, wherein the predetermined input operation from the operator is a predetermined movement of the operator's finger on the touch panel display.

4. The medical image processing apparatus according to claim 3, wherein the predetermined action of the operator's fingers is the operator touching the touch panel display with multiple fingers and moving the multiple fingers simultaneously in a predetermined direction.

5. Equipped with a casing, A portable medical image processing apparatus according to any one of claims 1 to 4, wherein at least the display and the control unit are integrally incorporated into the housing.

6. On the computer, This involves reading volume data, including multiple anatomical structures such as blood vessels and parenchymal organs, based on information obtained by imaging the patient, The CT values ​​in the volume data are analyzed, and the anatomical structures are determined based on the analysis results. Setting offset values ​​for the CT values ​​of the aforementioned plurality of anatomical structures, wherein the offset values ​​are set such that the CT values ​​of the blood vessels are uniform, and the offset values ​​for the CT values ​​of the parenchymal organs are set to be greater than the offset values ​​of the blood vessels, for each anatomical structure. The system receives a predetermined input operation from the operator for setting a threshold value for the CT value, and based on the received input operation, sets the lower limit of the CT value to be displayed on the image of the anatomical structure. A medical image processing program that performs this task.

7. The computer reads volume data, including multiple anatomical structures such as blood vessels and parenchymal organs, based on information obtained by imaging the patient. The computer analyzes the CT values ​​in the volume data and determines the anatomical structure based on the analysis results. The computer sets offset values ​​for the CT values ​​of the plurality of anatomical structures, setting the offset values ​​for the blood vessels so that the CT values ​​are uniform, and setting the offset values ​​for the parenchymal organs so that they are greater than the offset values ​​for the blood vessels, for each anatomical structure. The computer receives a predetermined input operation from the operator for setting a threshold value for the CT value, and based on the received input operation, sets a lower limit value for the CT value to be displayed on the display of the image of the anatomical structure. A method for operating a medical image processing device, including the following.

Images