Image processing device, image processing system, image display method, and image processing program

Abstract

The present invention provides an image processing device for displaying, on a display as a three-dimensional image, three-dimensional data including first data that represents living tissue, second data that represents an object, or third data that represents a marker, the image processing device being provided with a control unit for: referencing the three-dimensional data and identifying the positional relationship between the living tissue and the object or the marker; assessing, according to the identified positional relationship, whether there exists interposing tissue that is a portion of the living tissue that is interposed between the object or the marker and a visual point set when the three-dimensional image is displayed; and performing, if it is assessed that the interposing tissue exists, control of displaying an image representing the marker or the object at the position on the three-dimensional image corresponding to the interposing tissue.

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing system, an image display method, and an image processing program. [Background technology]

[0002] Patent Documents 1 to 3 describe techniques for generating three-dimensional images of cardiac chambers or blood vessels using an US imaging system. "US" is an abbreviation for ultrasound. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2010 / 0215238 [Patent Document 2] U.S. Patent No. 6,385,332 [Patent Document 3] U.S. Patent No. 6,251,072 Summary of the Invention [Problem to be solved by the invention]

[0004] Treatments using IVUS are widely performed in cardiac chambers, cardiovascular systems, and lower limb arterial regions. "IVUS" is an abbreviation for intravascular ultrasound. IVUS is a device or method that provides two-dimensional images in a plane perpendicular to the long axis of the catheter.

[0005] Currently, surgeons must perform procedures while mentally reconstructing the three-dimensional structure by stacking 2D IVUS images, which poses a particular barrier for younger or less experienced physicians. To overcome this barrier, a 3D image representing the structure of biological tissues, such as cardiac chambers or blood vessels, could be automatically generated from the 2D IVUS images and displayed to the surgeon. Simply displaying the generated 3D image as is would only allow the surgeon to see the outer wall of the tissue, so a 3D image could be cropped to reveal the inner cavity. If a catheter other than the IVUS catheter, such as an ablation catheter or a catheter for atrial septal puncture, is inserted into the biological tissue, a 3D image representing the other catheter could be displayed.

[0006] However, due to the angle at which the surgeon views these 3D images, if the catheter is behind a ridge in the tissue, or if the fossa ovalis is tenting during atrial septal puncture and the catheter is embedded in the tissue, the surgeon will not be able to see the catheter and will not be able to perform the procedure smoothly.

[0007] Recently, a procedure to electrically isolate cardiac chambers by cauterizing them with an ablation catheter has become widespread. This procedure primarily uses a 3D mapping system, which uses position information from the catheter's contact with myocardial tissue to create a 3D image. However, the catheter must be thoroughly in contact with the surface of the myocardial tissue within the cardiac chamber to create a 3D image, which is extremely time-consuming. Circumferential isolation of the PV or SVC requires marking the ablated areas. If this procedure could be completed using IVUS, it could potentially reduce the time required. "PV" is an abbreviation for pulmonary vein. "SVC" is an abbreviation for superior vena cava. Marking at least one location, such as the ablated area, on the tissue could potentially display a 3D image representing the landmark.

[0008] However, while the coordinates of the landmarks are fixed once they are placed, the coordinates of the tissue are updated continuously by IVUS, so the landmarks may end up inside the tissue due to pulsation or displacement of the IVUS catheter. In such cases, the surgeon cannot see the landmarks, making it difficult to perform procedures such as ablation.

[0009] The objective of the present disclosure is to enable identification of the location of an object located within a lumen of biological tissue, or a landmark associated with biological tissue, even when the object or landmark is located behind or within a portion of the biological tissue. [Means for solving the problem]

[0010] An image processing device according to one aspect of the present disclosure is an image processing device that displays three-dimensional data including first data representing biological tissue, second data representing an object located in the lumen of the biological tissue, or third data representing a landmark associated with the biological tissue, as a three-dimensional image on a display, and includes a control unit that identifies a positional relationship between the biological tissue and the object or the landmark by referring to the three-dimensional data, determines whether or not there is intervening tissue, which is a part of the biological tissue that is interposed between the object or the landmark and a viewpoint that is set when the three-dimensional image is displayed, based on the identified positional relationship, and, if it is determined that there is intervening tissue, controls the display of an image representing the object or the landmark at a position in the three-dimensional image corresponding to the intervening tissue.

[0011] In one embodiment, when it is determined that the intervening tissue is present, the control unit performs control to display an image representing the object or the mark on the surface of the intervening tissue in the three-dimensional image.

[0012] In one embodiment, the control unit applies a texture to the surface of the intervening tissue as an image representing the object or the landmark that is different from the texture applied to the surface of the portion of the biological tissue adjacent to the intervening tissue in the three-dimensional image.

[0013] In one embodiment, when it is determined that the intervening tissue is present, the control unit performs control to display an image representing the object or the mark in a cross section of the intervening tissue in the three-dimensional image.

[0014] In one embodiment, the control unit applies a texture to the cross section of the intervening tissue as an image representing the object or the landmark, which is different from the texture applied to the cross section of the biological tissue adjacent to the intervening tissue in the three-dimensional image.

[0015] In one embodiment, when it is determined that the intervening tissue is present, the control unit performs control to display an image representing the object or the landmark in the three-dimensional image with the intervening tissue being transparent.

[0016] In one embodiment, the control unit positions the three-dimensional image representing the object or the landmark on the opposite side of the intervening tissue from the viewpoint in the three-dimensional image as the image representing the object or the landmark.

[0017] In one embodiment, the three-dimensional data includes, as the first data and the second data, data constructed based on data obtained by a sensor inserted into the inner cavity of the biological tissue and observing the biological tissue and the object, and the object is a catheter inserted into the inner cavity of the biological tissue.

[0018] In one embodiment, the three-dimensional data is constructed based on data obtained by a sensor inserted into the inner cavity of the biological tissue and observing the biological tissue, and includes as the first data data that is updated each time new data is obtained by the sensor, and the control unit acquires designated data that designates the landmark, constructs the third data based on the acquired designated data, and includes the third data in the three-dimensional data.

[0019] An image processing system according to one aspect of the present disclosure includes the image processing device and a sensor for observing the biological tissue and the object.

[0020] In one embodiment, the image processing system further comprises the display.

[0021] An image display method according to one aspect of the present disclosure is an image display method for displaying three-dimensional data including first data representing biological tissue, second data representing an object located in the lumen of the biological tissue, or third data representing a landmark associated with the biological tissue, as a three-dimensional image on a display, wherein a computer refers to the three-dimensional data to identify a positional relationship between the biological tissue and the object or the landmark, determines whether or not there is intervening tissue, which is a part of the biological tissue that is interposed between the object or the landmark and a viewpoint that is set when the three-dimensional image is displayed, based on the identified positional relationship, and if the computer determines that there is intervening tissue, controls to display an image representing the object or the landmark at a position in the three-dimensional image corresponding to the intervening tissue.

[0022] An image processing program according to one aspect of the present disclosure causes a computer to display three-dimensional data including first data representing biological tissue, second data representing an object located in the lumen of the biological tissue, or third data representing a landmark associated with the biological tissue as a three-dimensional image on a display, and to perform the following processes: referring to the three-dimensional data to identify the positional relationship between the biological tissue and the object or the landmark; determining, based on the identified positional relationship, whether or not there is intervening tissue, which is a part of the biological tissue that is interposed between the object or the landmark and a viewpoint that is set when the three-dimensional image is displayed; and, if it is determined that there is intervening tissue, controlling the display of an image representing the object or the landmark at a position in the three-dimensional image corresponding to the intervening tissue. [Effects of the Invention]

[0023] According to the present disclosure, it is possible to confirm the position of an object located in the lumen of biological tissue or a landmark associated with biological tissue even when the object or landmark is located behind or inside a portion of the biological tissue. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a perspective view of an image processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 10 is a cross-sectional view showing an example in which an object is present behind intervening tissue. [Figure 3] FIG. 1 is a block diagram illustrating a configuration of an image processing device according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a perspective view of a probe and a drive unit according to an embodiment of the present disclosure. [Figure 5] 4 is a flowchart illustrating an operation of the image processing system according to an embodiment of the present disclosure. [Figure 6] 4 is a flowchart illustrating an operation of the image processing system according to an embodiment of the present disclosure. [Figure 7] 1 is a cross-sectional view showing an example of the positional relationship between biological tissue, an object, and a viewpoint. [Figure 8] FIG. 2 is a diagram illustrating an example of a screen of a display according to an embodiment of the present disclosure. [Figure 9] FIG. 2 is a diagram illustrating an example of a screen of a display according to an embodiment of the present disclosure. [Figure 10] FIG. 2 is a schematic diagram showing an example of the positional relationship between biological tissue, a landmark, and a viewpoint. [Figure 11] FIG. 10 is a schematic diagram showing another example of the positional relationship between biological tissue, a landmark, and a viewpoint. [Figure 12] 12 is a schematic diagram showing an example of displaying an image representing a mark on the surface of an intervening tissue in the example of FIG. 11. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0026] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0027] The outline of this embodiment will be described with reference to FIGS. 1 to 3. FIG.

[0028] The image processing device 11 according to this embodiment is a computer that displays, on a display 16, three-dimensional data 52 including first data representing a biological tissue 60 and second data representing an object located in a lumen 61 of the biological tissue 60 as a three-dimensional image 53. The image processing device 11 identifies the positional relationship between the biological tissue 60 and the object by referring to the three-dimensional data 52. Based on the identified positional relationship, the image processing device 11 determines whether intervening tissue exists between the object and the viewpoint set when the three-dimensional image 53 is displayed. If the image processing device 11 determines that intervening tissue exists, it controls the display of an image representing the object at a position corresponding to the intervening tissue in the three-dimensional image 53. Therefore, according to this embodiment, the position of the object can be confirmed even when intervening tissue exists.

[0029] The biological tissue 60 includes, for example, an organ such as a blood vessel or the heart. The biological tissue 60 is not limited to a single anatomical organ or a part thereof, but also includes tissue having a lumen spanning multiple organs. A specific example of such tissue is a part of the vascular tissue extending from the upper part of the inferior vena cava, through the right atrium, and to the lower part of the superior vena cava.

[0030] 2, the biological tissue 60 is the right atrium. In this example, a portion of the right atrium adjacent to the fossa ovalis 65 is raised inward to form a ridge 64. A catheter 63, such as an ablation catheter or a catheter for atrial septal puncture, is inserted into the right atrium.

[0031] Suppose an image representing the structure of biological tissue 60 is automatically generated as the three-dimensional image 53 and displayed to the surgeon. Suppose a portion of the structure of biological tissue 60 is cut out from the generated image to allow a view of the lumen 61. Suppose an image representing a catheter 63 is also displayed. In this case, depending on the direction from which the lumen 61 is viewed, the catheter 63 may be hidden behind the ridges 64, making it invisible to the surgeon. However, in this case, in this embodiment, the image representing the catheter 63 is displayed on the surface of the ridges 64. That is, at least the portion of the ridges 64 that hides the catheter 63 appears transparent, allowing the surgeon to see the catheter 63 through this portion. This allows the surgeon to smoothly perform procedures such as ablation or atrial septal puncture.

[0032] In the above example, the portion of the ridge 64 that hides the catheter 63 corresponds to the "intervening tissue." This embodiment can be applied not only when the catheter 63 is behind the ridge 64, but also when the catheter 63 is behind or inside any tissue, such as when the fossa ovalis 65 is tented during atrial septal puncture and the catheter 63 is embedded in the tissue.

[0033] In the above example, the catheter 63 corresponds to the “object.” The object is not limited to the catheter 63, but may be another object located in the lumen 61 of the biological tissue 60, such as a stent.

[0034] 2, the X direction and the Y direction perpendicular to the X direction correspond to the short-side direction of the lumen 61 of the biological tissue 60. The Z direction perpendicular to the X direction and the Y direction corresponds to the long-side direction of the lumen 61 of the biological tissue 60.

[0035] The configuration of an image processing system 10 according to this embodiment will be described with reference to FIG.

[0036] The image processing system 10 includes an image processing device 11 , a cable 12 , a drive unit 13 , a keyboard 14 , a mouse 15 , and a display 16 .

[0037] In this embodiment, the image processing device 11 is a dedicated computer specialized for image diagnosis, but it may also be a general-purpose computer such as a PC. "PC" is an abbreviation for personal computer.

[0038] The cable 12 is used to connect the image processing device 11 and the drive unit 13 .

[0039] The drive unit 13 is connected to the probe 20 shown in FIG. 4 and drives the probe 20. The drive unit 13 is also called an MDU. "MDU" is an abbreviation for motor drive unit. The probe 20 is applied to IVUS. The probe 20 is also called an IVUS catheter or an imaging diagnostic catheter.

[0040] The keyboard 14, mouse 15, and display 16 are connected to the image processing device 11 via any cable or wirelessly. The display 16 is, for example, an LCD, an organic EL display, or an HMD. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electroluminescence. "HMD" is an abbreviation for head-mounted display.

[0041] The image processing system 10 further includes a connection terminal 17 and a cart unit 18 as options.

[0042] The connection terminal 17 is used to connect the image processing device 11 to an external device. The connection terminal 17 is, for example, a USB terminal. "USB" is an abbreviation for Universal Serial Bus. The external device is, for example, a recording medium such as a magnetic disk drive, a magneto-optical disk drive, or an optical disk drive.

[0043] The cart unit 18 is a cart with casters for mobility. The image processing device 11, the cable 12, and the drive unit 13 are installed in the cart body of the cart unit 18. The keyboard 14, the mouse 15, and the display 16 are installed on the top table of the cart unit 18.

[0044] The configurations of the probe 20 and the drive unit 13 according to this embodiment will be described with reference to FIG.

[0045] The probe 20 includes a drive shaft 21, a hub 22, a sheath 23, an outer tube 24, an ultrasonic transducer 25, and a relay connector 26.

[0046] The drive shaft 21 passes through a sheath 23 inserted into a body cavity of a living body and an outer tube 24 connected to the proximal end of the sheath 23, and extends to the inside of a hub 22 provided at the proximal end of the probe 20. The drive shaft 21 has an ultrasonic vibrator 25 at its tip that transmits and receives signals, and is rotatably provided within the sheath 23 and outer tube 24. A relay connector 26 connects the sheath 23 and outer tube 24.

[0047] The hub 22, drive shaft 21, and ultrasonic transducer 25 are connected to one another so that they can integrally move forward and backward in the axial direction. Therefore, for example, when the hub 22 is pushed toward the distal end, the drive shaft 21 and ultrasonic transducer 25 move toward the distal end inside the sheath 23. For example, when the hub 22 is pulled toward the proximal end, the drive shaft 21 and ultrasonic transducer 25 move toward the proximal end inside the sheath 23, as shown by the arrows.

[0048] The drive unit 13 includes a scanner unit 31 , a slide unit 32 , and a bottom cover 33 .

[0049] The scanner unit 31 is connected to the image processing device 11 via the cable 12. The scanner unit 31 includes a probe connection section 34 that connects to the probe 20, and a scanner motor 35 that is a drive source that rotates the drive shaft 21.

[0050] The probe connection part 34 is detachably connected to the probe 20 via a socket 36 of a hub 22 provided at the base end of the probe 20. The base end of the drive shaft 21 is rotatably supported inside the hub 22, and the rotational force of the scanner motor 35 is transmitted to the drive shaft 21. Signals are transmitted and received between the drive shaft 21 and the image processing device 11 via a cable 12. The image processing device 11 generates a tomographic image of the biological lumen and performs image processing based on the signal transmitted from the drive shaft 21.

[0051] The slide unit 32 carries the scanner unit 31 so that the scanner unit 31 can move back and forth, and is mechanically and electrically connected to the scanner unit 31. The slide unit 32 includes a probe clamp section 37, a slide motor 38, and a group of switches 39.

[0052] The probe clamp portion 37 is disposed coaxially with the probe connection portion 34 on the distal side thereof, and supports the probe 20 connected to the probe connection portion 34 .

[0053] The slide motor 38 is a drive source that generates a driving force in the axial direction. The scanner unit 31 moves forward and backward when driven by the slide motor 38, and accordingly, the drive shaft 21 moves forward and backward in the axial direction. The slide motor 38 is, for example, a servo motor.

[0054] The switch group 39 includes, for example, a forward switch and a pull-back switch that are pressed when moving the scanner unit 31 forward or backward, and a scan switch that is pressed when starting and ending image drawing. The switches are not limited to the examples given here, and various other switches may be included in the switch group 39 as needed.

[0055] When the forward switch is pressed, the slide motor 38 rotates forward, moving the scanner unit 31 forward. On the other hand, when the pullback switch is pressed, the slide motor 38 rotates backward, moving the scanner unit 31 backward.

[0056] When the scan switch is pressed, image drawing begins, and the scanner motor 35 and slide motor 38 are driven to move the scanner unit 31 backward. A user such as an operator connects the probe 20 to the scanner unit 31 in advance so that when image drawing begins, the drive shaft 21 rotates and moves axially toward the proximal end. When the scan switch is pressed again, the scanner motor 35 and slide motor 38 stop, and image drawing ends.

[0057] The bottom cover 33 covers the bottom surface of the slide unit 32 and the entire periphery of the side surface on the bottom side, and can be moved toward and away from the bottom surface of the slide unit 32.

[0058] The configuration of the image processing device 11 will be described with reference to FIG.

[0059] The image processing device 11 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, and an output unit 45.

[0060] The control unit 41 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. An example of the programmable circuit is an FPGA. "FPGA" is an abbreviation for field-programmable gate array. An example of the dedicated circuit is an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 41 controls each part of the image processing system 10 including the image processing device 11, and executes processing related to the operation of the image processing device 11.

[0061] The storage unit 42 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, RAM or ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. RAM is, for example, SRAM or DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. ROM is, for example, EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The storage unit 42 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 42 stores data used in the operation of the image processing device 11, such as tomographic data 51, and data obtained by the operation of the image processing device 11, such as three-dimensional data 52 and three-dimensional images 53.

[0062] The communication unit 43 includes at least one communication interface. The communication interface is, for example, a wired LAN interface, a wireless LAN interface, or an image diagnosis interface that receives and A / D converts IVUS signals. "LAN" is an abbreviation for local area network. "A / D" is an abbreviation for analog to digital. The communication unit 43 receives data used in the operation of the image processing device 11 and transmits data obtained by the operation of the image processing device 11. In this embodiment, the drive unit 13 is connected to the image diagnosis interface included in the communication unit 43.

[0063] The input unit 44 includes at least one input interface. The input interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface. The input unit 44 accepts user operations such as operations to input data used in the operation of the image processing device 11. In this embodiment, the keyboard 14 and the mouse 15 are connected to a USB interface included in the input unit 44 or an interface compatible with short-range wireless communication. If a touch screen is provided integrally with the display 16, the display 16 may be connected to a USB interface or an HDMI (registered trademark) interface included in the input unit 44.

[0064] The output unit 45 includes at least one output interface. The output interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). The output unit 45 outputs data obtained by the operation of the image processing device 11. In this embodiment, the display 16 is connected to the USB interface or the HDMI (registered trademark) interface included in the output unit 45.

[0065] The functions of the image processing device 11 are realized by executing an image processing program according to this embodiment on a processor serving as the control unit 41. That is, the functions of the image processing device 11 are realized by software. The image processing program causes a computer to execute the operations of the image processing device 11, thereby causing the computer to function as the image processing device 11. That is, the computer functions as the image processing device 11 by executing the operations of the image processing device 11 in accordance with the image processing program.

[0066] The program can be stored on a non-transitory computer-readable medium. Examples of non-transitory computer-readable media include flash memory, magnetic recording devices, optical disks, magneto-optical recording media, and ROMs. The program can be distributed by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs that store the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program can also be distributed by storing it in the storage of a server and transferring it from the server to another computer. The program can also be provided as a program product.

[0067] A computer temporarily stores a program stored on a portable medium or transferred from a server in its main storage device. The computer then reads the program stored in the main storage device using a processor and executes processing in accordance with the read program. The computer may also read the program directly from a portable medium and execute processing in accordance with the program. The computer may also execute processing in accordance with the received program each time a program is transferred from a server to the computer. Processing may also be executed through a so-called ASP-type service that achieves its functions simply by issuing execution instructions and obtaining results, without transferring the program from the server to the computer. "ASP" is an abbreviation for application service provider. Programs include information used for processing by a computer that is equivalent to a program. For example, data that does not directly instruct a computer but has properties that define computer processing falls under the category of "equivalent to a program."

[0068] Some or all of the functions of the image processing device 11 may be realized by a programmable circuit or a dedicated circuit as the control unit 41. In other words, some or all of the functions of the image processing device 11 may be realized by hardware.

[0069] The operation of the image processing system 10 according to this embodiment will be described with reference to Fig. 5. The operation of the image processing system 10 corresponds to the image display method according to this embodiment.

[0070] 5 begins, the user primes the probe 20. Thereafter, the probe 20 is fitted into the probe connection portion 34 and the probe clamp portion 37 of the drive unit 13, and is connected and fixed to the drive unit 13. Then, the probe 20 is inserted to a target site in biological tissue 60, such as a blood vessel or the heart.

[0071] In step S101, a scan switch included in the switch group 39 is pressed, and then a pullback switch also included in the switch group 39 is pressed, thereby performing a so-called pullback operation. The probe 20 transmits ultrasonic waves from the ultrasonic transducer 25, which is moved back in the axial direction by the pullback operation, inside the biological tissue 60. The ultrasonic transducer 25 transmits ultrasonic waves radially while moving inside the biological tissue 60. The ultrasonic transducer 25 receives reflected waves of the transmitted ultrasonic waves. The probe 20 inputs signals of the reflected waves received by the ultrasonic transducer 25 to the image processing device 11. The control unit 41 of the image processing device 11 processes the input signals to sequentially generate cross-sectional images of the biological tissue 60, thereby acquiring tomographic data 51 including a plurality of cross-sectional images.

[0072] Specifically, the probe 20 transmits ultrasonic waves from the ultrasonic transducer 25 in multiple directions outward from the center of rotation while rotating the ultrasonic transducer 25 circumferentially and moving it axially inside the biological tissue 60. The probe 20 receives reflected waves from reflecting objects present in each of multiple directions inside the biological tissue 60 using the ultrasonic transducer 25. The probe 20 transmits signals of the received reflected waves to the image processing device 11 via the drive unit 13 and the cable 12. The communication unit 43 of the image processing device 11 receives the signals transmitted from the probe 20. The communication unit 43 A / D converts the received signals. The communication unit 43 inputs the A / D converted signals to the control unit 41. The control unit 41 processes the input signals to calculate an intensity value distribution of the reflected waves from reflecting objects present in the transmission direction of the ultrasonic waves from the ultrasonic transducer 25. The control unit 41 sequentially generates two-dimensional images having a brightness value distribution corresponding to the calculated intensity value distribution as cross-sectional images of the biological tissue 60, thereby acquiring tomographic data 51, which is a data set of the cross-sectional images. The control unit 41 stores the acquired tomographic data 51 in the storage unit 42 .

[0073] In this embodiment, the reflected wave signal received by the ultrasonic transducer 25 corresponds to the raw data of the tomographic data 51, and the cross-sectional image generated by the image processing device 11 by processing the reflected wave signal corresponds to the processed data of the tomographic data 51.

[0074] As a modified example of this embodiment, the control unit 41 of the image processing device 11 may store the signal input from the probe 20 as it is in the storage unit 42 as the tomographic data 51. Alternatively, the control unit 41 may process the signal input from the probe 20 and store data indicating the intensity value distribution of the reflected wave calculated in the storage unit 42 as the tomographic data 51. In other words, the tomographic data 51 is not limited to a data set of cross-sectional images of the biological tissue 60, but may be data that represents in some form the cross section of the biological tissue 60 at each movement position of the ultrasound transducer 25.

[0075] As a modification of this embodiment, instead of the ultrasonic transducer 25 that transmits ultrasonic waves in multiple directions while rotating in the circumferential direction, an ultrasonic transducer that transmits ultrasonic waves in multiple directions without rotating may be used.

[0076] As a modification of this embodiment, the tomographic data 51 may be acquired using OFDI or OCT instead of using IVUS. "OFDI" is an abbreviation for optical frequency domain imaging. "OCT" is an abbreviation for optical coherence tomography. When OFDI or OCT is used, a sensor that acquires the tomographic data 51 by emitting light in the lumen 61 of the biological tissue 60 is used as a sensor that acquires the tomographic data 51 while moving through the lumen 61 of the biological tissue 60, instead of the ultrasonic transducer 25 that acquires the tomographic data 51 by transmitting ultrasonic waves in the lumen 61 of the biological tissue 60.

[0077] As a modified example of this embodiment, instead of the image processing device 11 generating a data set of cross-sectional images of the biological tissue 60, another device may generate a similar data set, and the image processing device 11 may acquire the data set from the other device. That is, instead of the control unit 41 of the image processing device 11 processing an IVUS signal to generate a cross-sectional image of the biological tissue 60, another device may process the IVUS signal to generate a cross-sectional image of the biological tissue 60, and the generated cross-sectional image may be input to the image processing device 11.

[0078] In step S102, the control unit 41 of the image processing device 11 generates three-dimensional data 52 of the biological tissue 60 based on the tomographic data 51 acquired in step S101. That is, the control unit 41 generates the three-dimensional data 52 based on the tomographic data 51 acquired by the sensor. Here, if already generated three-dimensional data 52 exists, it is preferable to update only the data corresponding to the updated tomographic data 51 rather than regenerating all of the three-dimensional data 52 from scratch. In this case, the amount of data processing when generating the three-dimensional data 52 can be reduced, and the real-time performance of the three-dimensional image 53 in the subsequent step S103 can be improved.

[0079] Specifically, the control unit 41 of the image processing device 11 generates three-dimensional data 52 of the biological tissue 60 by stacking cross-sectional images of the biological tissue 60 included in the tomographic data 51 stored in the storage unit 42 to create a three-dimensional image. As a three-dimensionalization method, any of a rendering method such as surface rendering or volume rendering, and various processes associated therewith, such as texture mapping including environment mapping and bump mapping, is used. The control unit 41 stores the generated three-dimensional data 52 in the storage unit 42.

[0080] When a catheter 63 other than the IVUS catheter, such as an ablation catheter or a catheter for atrial septal puncture, is inserted into the biological tissue 60, the tomographic data 51 includes data on the catheter 63 as well as data on the biological tissue 60. Therefore, in step S102, the three-dimensional data 52 generated by the control unit 41 also includes data on the catheter 63 as second data as well as data on the biological tissue 60 as first data.

[0081] With reference to FIG. 6, the process performed in step S102 when the catheter 63 is inserted into the living tissue 60 will be described in detail.

[0082] Before starting the flow of FIG. 6 , the control unit 41 of the image processing device 11 classifies a group of pixels of the cross-sectional image included in the tomographic data 51 acquired in step S101 into two or more classes. These two or more classes include at least a class of “biological tissue” and a class of “catheter,” and may further include a class of “blood cells,” a class of “medical instruments” other than “catheters” such as guidewires, a class of “indwelling objects” such as stents, or a class of “lesions” such as calcification or plaque. Any classification method may be used, but in this embodiment, a method of classifying a group of pixels of the cross-sectional image using a trained model is used. The trained model is trained in advance by machine learning so that it can detect areas corresponding to each class from a sample IVUS cross-sectional image.

[0083] In step S200a, the control unit 41 of the image processing device 11 constructs a three-dimensional object of the biological tissue 60 by stacking regions classified into the class "biological tissue." The control unit 41 reflects the constructed three-dimensional object of the biological tissue 60 in three-dimensional space. In step S200b, the control unit 41 constructs a three-dimensional object of the catheter 63 by stacking regions classified into the class "catheter." In this embodiment, the catheter 63 is extracted by segmentation, but it may be extracted by other methods, such as object detection. For example, it is possible to use a method of extracting only the catheter position, and construct an object taking that position into consideration as the object of the catheter 63. The control unit 41 reflects the constructed three-dimensional object of the catheter 63 in three-dimensional space. In step S200c, the control unit 41 executes the processes of steps S201 to S205 for each tissue voxel, which is a voxel of the biological tissue 60. At this point, the control unit 41 may place a virtual camera 71 and a virtual light source 72 as shown in Fig. 7 at any positions in the three-dimensional space. The position of the camera 71 corresponds to the "viewpoint" when the three-dimensional image 53 is displayed on the display 16. The number and relative positions of the light sources 72 are not limited to those shown in the figure and can be changed as appropriate. The three-dimensional object of the biological tissue 60 may be cut along any cross-section.

[0084] In step S201, the control unit 41 of the image processing device 11 determines whether or not there is a catheter voxel, which is a voxel of the catheter 63, on an extension of the line, including the line connecting the viewpoint and the tissue voxel. If there is no catheter voxel on the extension, in step S202, the control unit 41 applies a first color, which is a color pre-assigned to the class of "biological tissue," as the color of the tissue voxel. If there is a catheter voxel on the extension, in step S203, the control unit 41 calculates a first distance, which is the distance between the viewpoint and the tissue voxel, and a second distance, which is the distance between the viewpoint and the catheter voxel on the extension. If the first distance is longer than the second distance, i.e., if the catheter 63 is present in front of the biological tissue 60, in step S204, the control unit 41 applies a second color, which is a color pre-assigned to the class of "catheter," as the color of the tissue voxel. If the first distance is shorter than the second distance, i.e., if the catheter 63 is located behind or inside the biological tissue 60, in step S205, the control unit 41 applies a third color different from the first and second colors, such as an intermediate color between the first and second colors, as the color of the tissue voxels. In this embodiment, the control unit 41 applies a color that is 70% first color and 30% second color as the intermediate color equivalent to the third color.

[0085] Data relating to the coloring of the voxels performed in the flow of FIG. 6 is stored in the storage unit 42 as part of the three-dimensional data 52.

[0086] In step S103, the control unit 41 of the image processing device 11 causes the display 16 to display the three-dimensional data 52 generated in step S102 as a three-dimensional image 53.

[0087] Specifically, the control unit 41 of the image processing device 11 generates a three-dimensional image 53 from three-dimensional data 52 stored in the storage unit 42. The control unit 41 causes the generated three-dimensional image 53 to be displayed on the display 16 via the output unit 45.

[0088] In step S104, if a user operation is performed, the processes of steps S105 to S108 are carried out, whereas if a user operation is not performed, the processes of steps S105 to S108 are skipped.

[0089] In step S105, the control unit 41 of the image processing device 11 receives an operation to set the position of the opening 62 as shown in Fig. 7 via the input unit 44. The position of the opening 62 is set to a position such that the inner cavity 61 of the biological tissue 60 is exposed through the opening 62 in the three-dimensional image 53 displayed in step S103.

[0090] Specifically, the control unit 41 of the image processing device 11 accepts, via the input unit 44, an operation by the user to cut off a portion of the biological tissue 60 in the three-dimensional image 53 displayed on the display 16 using the keyboard 14, the mouse 15, or a touch screen integrated with the display 16. In the example of FIG. 7, the control unit 41 accepts an operation to cut off a portion of the biological tissue 60 so that the cross section of the biological tissue 60 has an open shape. The "cross section of the biological tissue 60" may be a transverse section of the biological tissue 60, a longitudinal section of the biological tissue 60, or another cross section of the biological tissue 60. The "transverse section of the biological tissue 60" refers to a cross section of the biological tissue 60 cut perpendicular to the direction in which the ultrasonic transducer 25 moves within the biological tissue 60. The "longitudinal section of the biological tissue 60" refers to a cross section of the biological tissue 60 cut along the direction in which the ultrasonic transducer 25 moves within the biological tissue 60. "Another cross section of the biological tissue 60" refers to a cross section of the biological tissue 60 cut obliquely with respect to the direction in which the ultrasound transducer 25 moves within the biological tissue 60. The "open shape" refers to, for example, an approximately C-shape, an approximately U-shape, an approximately 3-shape, or a shape in which any of these is partially missing due to the presence of a hole originally present in the biological tissue 60, such as a branching point of a blood vessel or the opening of a pulmonary vein. In the example of Figure 7, the cross section of the biological tissue 60 is approximately C-shaped.

[0091] In step S106, the control unit 41 of the image processing device 11 determines the position of the opening 62 to be the position set by the operation accepted in step S105.

[0092] Specifically, the control unit 41 of the image processing device 11 specifies, in the three-dimensional data 52 stored in the storage unit 42, the three-dimensional coordinates of the boundary of the portion of the biological tissue 60 that has been cut off by the user's operation as the three-dimensional coordinates of the edge of the opening 62. The control unit 41 stores the specified three-dimensional coordinates in the storage unit 42.

[0093] In step S107, the control unit 41 of the image processing device 11 forms an opening 62 in the three-dimensional data 52, which exposes the lumen 61 of the biological tissue 60 in the three-dimensional image 53.

[0094] Specifically, the control unit 41 of the image processing device 11 sets the part of the three-dimensional data 52 stored in the memory unit 42 that is identified by the three-dimensional coordinates stored in the memory unit 42 to be hidden or transparent when the three-dimensional image 53 is displayed on the display 16.

[0095] In step S108, the control unit 41 of the image processing device 11 adjusts the viewpoint when displaying the three-dimensional image 53 on the display 16, depending on the position of the opening 62 formed in step S107. In this embodiment, the control unit 41 positions the viewpoint on a straight line extending from the inner surface of the biological tissue 60 through the opening 62 to the outside of the biological tissue 60. Therefore, the user can virtually observe the lumen 61 of the biological tissue 60 by looking into the inside of the biological tissue 60 through the opening 62.

[0096] Specifically, the control unit 41 of the image processing device 11 places a virtual camera 71 at a position in the three-dimensional image 53 displayed on the display 16 where the lumen 61 of the biological tissue 60 can be seen through a portion set to be hidden or transparent. In the example of FIG. 7 , the control unit 41 places the virtual camera 71 in an area AF in the cross section of the biological tissue 60, which is sandwiched between a first line L1 extending from the inner surface of the biological tissue 60 through a first edge E1 of the opening 62 to the outside of the biological tissue 60 and a second line L2 extending from the inner surface of the biological tissue 60 through a second edge E2 of the opening 62 to the outside of the biological tissue 60. The point where the first line L1 intersects with the inner surface of the biological tissue 60 is the same point Pt as the point where the second line L2 intersects with the inner surface of the biological tissue 60. Therefore, regardless of where the virtual camera 71 is placed in the area AF, the user can observe point Pt on the inner surface of the biological tissue 60.

[0097] 7, point Pt is the same as the point where a fourth line L4, which is drawn perpendicularly to a third line L3 connecting the first edge E1 of the opening 62 and the second edge E2 of the opening 62 from a midpoint Pc of the third line L3, intersects with the inner surface of the biological tissue 60. Therefore, the user can easily observe point Pt on the inner surface of the biological tissue 60 through the opening 62. In particular, if a virtual camera 71 is placed on an extension of the fourth line L4 as shown in FIG. 7, the user can easily observe point Pt on the inner surface of the biological tissue 60.

[0098] The position of the virtual camera 71 may be any position that allows observation of the lumen 61 of the biological tissue 60 through the opening 62, but in this embodiment, it is within a range facing the opening 62. The position of the virtual camera 71 is preferably set to an intermediate position facing the center of the opening 62.

[0099] In step S108, if the catheter 63 is inserted into the biological tissue 60, the processes of steps S201 to S205 are also executed for each tissue voxel. In the example of FIG. 7, an intervening portion 66, which is part of the biological tissue 60, is present between the camera 71 and the catheter 63. That is, the catheter 63 is present behind the intervening portion 66. Therefore, the control unit 41 applies a third color, such as a color intermediate between the first and second colors, to the voxels corresponding to the intervening portion 66. The control unit 41 applies the first color to the voxels corresponding to the portion of the biological tissue 60 visible from the camera 71, excluding the intervening portion 66. In the intervening portion 66, of the surface 68 and the cross section 69 of the biological tissue 60, only the cross section 69 faces the camera 71. Therefore, as a result, the region of the cross section 69 corresponding to the intervening portion 66 is colored with the third color, and the remaining region is colored with the first color.

[0100] The control unit 41 of the image processing device 11 may switch the display mode between a first mode in which the processes of steps S201 to S205 are not performed even when the catheter 63 is inserted into the biological tissue 60, and a second mode in which the processes of steps S201 to S205 are performed when the catheter 63 is inserted into the biological tissue 60. For example, in the first mode, as shown in FIG. 8, even when the catheter 63 is located behind the intervening portion 66, the region of the cross section 69 corresponding to the intervening portion 66 is colored in the first color, as with the remaining regions. That is, the same texture as that applied to the cross section of the portion of the biological tissue 60 adjacent to the intervening portion 66 in the three-dimensional image 53 is applied to the cross section of the intervening portion 66. In contrast, in the second mode, as shown in FIG. 9, when the catheter 63 is located behind the intervening portion 66, the region of the cross section 69 corresponding to the intervening portion 66 is colored in the third color. That is, for the image 67 representing the catheter 63, a texture different from the texture applied to the cross section of the portion of the biological tissue 60 adjacent to the interposition portion 66 in the three-dimensional image 53 is applied to the cross section of the interposition portion 66.

[0101] The display mode may be switched manually by a user operation, or automatically when triggered by an arbitrary event.

[0102] In step S109, if the tomographic data 51 has been updated, the processes of steps S110 and S111 are performed. If the tomographic data 51 has not been updated, in step S104, it is again confirmed whether or not a user operation has been performed.

[0103] In step S110, the control unit 41 of the image processing device 11 processes the signal input from the probe 20 to generate a new cross-sectional image of the biological tissue 60, similar to the processing in step S101, thereby obtaining cross-sectional data 51 including at least one new cross-sectional image.

[0104] In step S111, the control unit 41 of the image processing device 11 updates the three-dimensional data 52 of the biological tissue 60 based on the tomographic data 51 acquired in step S110. That is, the control unit 41 updates the three-dimensional data 52 based on the tomographic data 51 acquired by the sensor. Then, in step S103, the control unit 41 displays the three-dimensional data 52 updated in step S111 on the display 16 as a three-dimensional image 53. In step S111, it is preferable to update only the data at a location corresponding to the updated tomographic data 51. In this case, the amount of data processing when updating the three-dimensional data 52 can be reduced, and the real-time nature of the three-dimensional image 53 can be improved in step S111.

[0105] Also in step S111, if the catheter 63 is inserted into the biological tissue 60, the processes from step S201 to step S205 are executed for each tissue voxel.

[0106] In steps S105 to S108 from the second time onwards, when changing the position of the opening 62 from the first position to the second position, the control unit 41 of the image processing device 11 moves the viewpoint from a third position corresponding to the first position to a fourth position corresponding to the second position. The control unit 41 moves the virtual light source 72 used when displaying the three-dimensional image 53 on the display 16 in accordance with the movement of the viewpoint from the third position to the fourth position.

[0107] When changing the circumferential position of the opening 62 in the cross section of the biological tissue 60, the control unit 41 moves the virtual light source 72 using the rotation matrix used to move the virtual camera 71.

[0108] When changing the position of opening 62 from the first position to the second position, control unit 41 may instantaneously switch the viewpoint from the third position to the fourth position, but in this embodiment, a moving image in which the viewpoint gradually moves from the third position to the fourth position is displayed on display 16 as three-dimensional image 53. This makes it easy for the user to understand that the viewpoint has moved.

[0109] As a variant of this embodiment, in step S105, the control unit 41 of the image processing device 11 may accept, via the input unit 44, an operation to set the position of the target point that the user wants to see, along with an operation to set the position of the opening 62.

[0110] Specifically, the control unit 41 of the image processing device 11 may accept, via the input unit 44, an operation by the user to specify the position of a target point in the three-dimensional image 53 displayed on the display 16 using the keyboard 14, the mouse 15, or a touch screen integrally provided with the display 16. In the example of Fig. 7, the control unit 41 may accept, via the input unit 44, an operation to set the position of a point Pt as the position of a point where the first line L1 and the second line L2 intersect with the inner surface of the biological tissue 60.

[0111] As a modification of this embodiment, in step S105, the control unit 41 of the image processing device 11 may accept an operation to set the position of a target point that the user wants to see via the input unit 44, instead of an operation to set the position of the opening 62. Then, in step S106, the control unit 41 may determine the position of the opening 62 according to the position set by the operation accepted in step S105.

[0112] Specifically, the control unit 41 of the image processing device 11 may accept, via the input unit 44, an operation by the user to specify the position of a target point on the 3D image 53 displayed on the display 16 using the keyboard 14, the mouse 15, or a touch screen integrated with the display 16. The control unit 41 may then determine the position of the opening 62 according to the position of the target point. In the example of FIG. 7 , the control unit 41 may accept, via the input unit 44, an operation to set the position of point Pt as the position of the point where the first line L1 and the second line L2 intersect with the inner surface of the biological tissue 60. The control unit 41 may determine, on the cross section of the biological tissue 60, a sector-shaped region having point Pt as its center and a central angle that is set in advance or specified by the user, as region AF. The control unit 41 may determine, as the position of the opening 62, a position of the biological tissue 60 that overlaps with region AF. The control unit 41 may determine, as the fourth line L4, a normal line perpendicular to a tangent line passing through point Pt on the inner surface of the biological tissue 60.

[0113] The area AF may be set to be narrower than the width of the opening 62. In other words, the area AF may be set so as not to include at least either the first edge E1 of the opening 62 or the second edge E2 of the opening 62.

[0114] As a modified example of this embodiment, the point where the first straight line L1 intersects with the inner surface of the biological tissue 60 does not have to be the same as the point where the second straight line L2 intersects with the inner surface of the biological tissue 60. For example, point P1, which is the point where the first straight line L1 intersects with the inner surface of the biological tissue 60, and point P2, which is the point where the second straight line L2 intersects with the inner surface of the biological tissue 60, may be on a circumference of a circle centered at point Pt. In other words, point P1 and point P2 may be approximately equidistant from point Pt.

[0115] As described above, in this embodiment, the control unit 41 of the image processing device 11 causes the display 16 to display three-dimensional data 52, including first data representing the biological tissue 60 and second data representing an object located in the lumen 61 of the biological tissue 60, as a three-dimensional image 53. The control unit 41 refers to the three-dimensional data 52 to identify the positional relationship between the biological tissue 60 and the object. Based on the identified positional relationship, the control unit 41 determines whether intervening tissue, which is a portion of the biological tissue 60 that is interposed between the object and the viewpoint set when the three-dimensional image 53 is displayed, is present. If the control unit 41 determines that intervening tissue is present, it controls the display of an image representing the object at a position corresponding to the intervening tissue in the three-dimensional image 53. Therefore, according to this embodiment, the position of the object can be confirmed even when intervening tissue is present.

[0116] In this embodiment, the control unit 41 of the image processing device 11 identifies the positional relationship between the biological tissue 60 and the object by referring to data obtained by a sensor observing the biological tissue 60 and the object. The sensor is not limited to the ultrasound transducer 25 used in IVUS, but may be any sensor such as a sensor used in OFDI, OCT, CT examination, extracorporeal echography, or X-ray examination. "CT" is an abbreviation for computed tomography. In accordance with the identified positional relationship, the control unit 41 detects, as intervening tissue, a portion of the biological tissue 60 that is interposed between the object and the viewpoint set when the three-dimensional image 53 is displayed. In the example of FIG. 7 , the catheter 63 corresponds to the object, and the intervening portion 66 corresponds to the intervening tissue.

[0117] In this embodiment, when the control unit 41 of the image processing device 11 determines that intervening tissue is present, it controls the display of an image representing the object on a cross section of the intervening tissue in the three-dimensional image 53. However, as a variant of this embodiment, it may also control the display of an image representing the object on the surface of the intervening tissue in the three-dimensional image 53.

[0118] 9, because the cross section of the intervening tissue faces the camera 71 in three-dimensional space, the control unit 41 of the image processing device 11 applies, as an image 67 representing the object, a texture to the cross section of the intervening tissue that is different from the texture applied to the cross section of the portion of the biological tissue 60 adjacent to the intervening tissue in the three-dimensional image 53. If the surface of the intervening tissue faces the camera 71 in three-dimensional space, the control unit 41 may apply, as an image representing the object, a texture to the surface of the intervening tissue that is different from the texture applied to the surface of the portion of the biological tissue 60 adjacent to the intervening tissue in the three-dimensional image 53.

[0119] As a modification of this embodiment, when it is determined that an intervening tissue is present, the control unit 41 of the image processing device 11 may perform control to display an image representing the object through the intervening tissue in the 3D image 53. In the example of FIG. 7 , the catheter 63 is present behind the intervening portion 66, so the control unit 41 changes the color of the voxels corresponding to the intervening portion 66, but instead, the control unit 41 may increase the transparency of the intervening portion 66. In this modification, the control unit 41 may place a 3D image representing the object on the opposite side of the viewpoint of the intervening tissue in the 3D image 53 as the image representing the object.

[0120] This embodiment may be applied not only to an object such as a catheter 63 as shown in FIG. 7 , but also to a landmark 73 associated with biological tissue 60 as shown in FIGS. 10 to 12 . In this case, the control unit 41 of the image processing device 11 causes the display 16 to display three-dimensional data 52 including first data representing the biological tissue 60 and third data representing the landmark 73 as a three-dimensional image 53. The control unit 41 identifies the positional relationship between the biological tissue 60 and the landmark 73 by referring to the three-dimensional data 52. Based on the identified positional relationship, the control unit 41 determines whether or not there is intervening tissue, which is a portion of the biological tissue 60 that is interposed between the landmark 73 and the viewpoint set when the three-dimensional image 53 is displayed. If it is determined that there is intervening tissue, the control unit 41 controls the display of an image 74 representing the landmark 73 at a position corresponding to the intervening tissue in the three-dimensional image 53. Therefore, even if there is intervening tissue, the position of the landmark 73 can be confirmed.

[0121] The first data included in the three-dimensional data 52 is constructed based on data obtained by a sensor inserted into the lumen 61 of the biological tissue 60 and observing the biological tissue 60, and is updated each time new data is obtained by the sensor. That is, the first data is configured to be successively updated by the sensor. When the mark 73 is placed, the control unit 41 of the image processing device 11 acquires designation data that designates the mark 73. Specifically, the control unit 41 acquires data designating the mark 73 as designation data by accepting a user operation that designates at least one location in three-dimensional space as the mark 73. The control unit 41 stores the acquired designation data in the storage unit 42. The control unit 41 constructs third data based on the designation data stored in the storage unit 42. The control unit 41 includes the constructed third data in the three-dimensional data 52 and displays the three-dimensional data 52 as a three-dimensional image 53 on the display 16. That is, the control unit 41 displays the third data as part of the three-dimensional image 53 on the display 16.

[0122] The mark 73 is associated with the biological tissue 60 by being attached to the biological tissue 60 or at least one location in the cavity 61 or the periphery of the biological tissue 60 in three-dimensional space, such as a cauterized location in the biological tissue 60, a start point and an end point for measuring distance in three-dimensional space, or the location of a nerve that must be avoided by cauterization. When the mark 73 is attached, the probe 20 is advanced or retreated, the three-dimensional data 52 is updated, and the coordinates of the location where the mark 73 is attached are recorded as a fixed point in three-dimensional space.

[0123] In the example of FIG. 10 , the biological tissue 60 is myocardium. In this example, a mark 73 is attached to the surface of the myocardium. When a pullback operation is performed, the mark 73 may be buried in the myocardial tissue, as shown in FIG. 11 . That is, an intervening portion 66, which is part of the myocardial tissue, may be present between the camera 71 and the mark 73. In this case, the mark 73 cannot be confirmed on the three-dimensional image 53. Therefore, when the control unit 41 of the image processing device 11 determines that the intervening portion 66 is present, the control unit 41 performs control to display an image 74 representing the mark 73 on the surface of the intervening portion 66 in the three-dimensional image 53, as shown in FIG. 12 . Specifically, the control unit 41 changes the color of the voxel corresponding to the intervening portion 66 to a third color different from the first and second colors, such as an intermediate color between the first color, which is the color of the myocardial tissue, and the second color, which is the color of the mark 73.

[0124] 12 , because the surface of the intervening portion 66 faces the camera 71 in three-dimensional space, the control unit 41 of the image processing device 11 applies, as an image 74 representing the mark 73, a texture to the surface of the intervening portion 66 that is different from the texture applied to the surface of the portion of the biological tissue 60 adjacent to the intervening portion 66 in the three-dimensional image 53. If it is assumed that the cross-section of the intervening portion 66 faces the camera 71 in three-dimensional space, the control unit 41 may apply, as an image representing the mark 73, a texture to the cross-section of the intervening portion 66 that is different from the texture applied to the cross-section of the portion of the biological tissue 60 adjacent to the intervening portion 66 in the three-dimensional image 53.

[0125] In the example of FIG. 12 , the mark 73 is present inside the intervening portion 66, and therefore the control unit 41 of the image processing device 11 changes the color of the voxel corresponding to the intervening portion 66. If the mark 73 is present behind the intervening portion 66, the control unit 41 may increase the transparency of the intervening portion 66. That is, when the control unit 41 determines that the intervening portion 66 is present, the control unit 41 may perform control to display an image representing the mark 73 by making the intervening portion 66 transparent in the three-dimensional image 53. In this case, the control unit 41 may place the three-dimensional image representing the mark 73 on the opposite side of the viewpoint of the intervening tissue in the three-dimensional image 53 as the image representing the mark 73.

[0126] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks shown in the block diagrams may be integrated, or one block may be divided. Two or more steps shown in the flowcharts may be executed in parallel or in a different order, instead of being executed in chronological order as described, depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible within the scope of the present disclosure. [Explanation of symbols]

[0127] 10 Image Processing System 11 Image processing device 12 Cable 13 Drive unit 14 keyboards 15 Mouse 16 Display 17 Connection terminal 18 Cart Unit 20 probes 21 Drive shaft 22 Hub 23 Sheath 24 Outer tube 25 Ultrasonic transducer 26 Relay connector 31 Scanner unit 32 Slide unit 33 Bottom cover 34 Probe connection 35 Scanner motor 36 Outlet 37 Probe clamp part 38 Slide motor 39 Switches 41 Control Unit 42 Storage section 43 Communications Department 44 Input section 45 Output section 51 Fault data 52 3D data 53 3D images 60 Biological Tissue 61 Lumen 62 Aperture 63 Catheter 64 Ridge 65 Fossa ovalis 66 Interposition part 67 images 68 Surface 69 cross section 71 Camera 72 Light source 73 Landmark 74 images 80 screens

Claims

1. 1. An image processing device that displays three-dimensional data, including first data representing biological tissue, and second data representing an object located in a lumen of the biological tissue, or third data representing a landmark associated with the biological tissue, on a display as a three-dimensional image, An image processing device comprising: a control unit that, by referring to the three-dimensional data, identifies the positional relationship between the biological tissue and the object or the landmark; determines, based on the identified positional relationship, whether or not there is intervening tissue, which is a part of the biological tissue that is interposed between the object or the landmark and the viewpoint that is set when the three-dimensional image is displayed; and, if it is determined that there is intervening tissue, controls the display of an image representing the object or the landmark at a position in the three-dimensional image that corresponds to the intervening tissue.

2. The image processing device according to claim 1 , wherein the control unit performs control to display an image representing the object or the mark on a surface of the intervening tissue in the three-dimensional image when it is determined that the intervening tissue is present.

3. The image processing device described in claim 2, wherein the control unit applies a texture to the surface of the intervening tissue that is different from the texture applied to the surface of the portion of the biological tissue adjacent to the intervening tissue in the three-dimensional image as an image representing the object or the landmark.

4. The image processing device according to claim 1 , wherein the control unit performs control to display an image representing the object or the mark on a cross section of the intervening tissue in the three-dimensional image when it is determined that the intervening tissue is present.

5. The image processing device described in claim 4, wherein the control unit applies a texture to the cross section of the intervening tissue that is different from a texture applied to a cross section of a portion of the biological tissue adjacent to the intervening tissue in the three-dimensional image as an image representing the object or the landmark.

6. The image processing device according to claim 1 , wherein the control unit, when determining that the intervening tissue is present, performs control to display an image representing the object or the landmark in the three-dimensional image by making the intervening tissue transparent.

7. The image processing device according to claim 6 , wherein the control unit positions a three-dimensional image representing the object or the landmark on the opposite side of the viewpoint of the intervening tissue in the three-dimensional image as the image representing the object or the landmark.

8. the three-dimensional data includes, as the first data and the second data, data constructed based on data obtained by a sensor inserted into the cavity of the biological tissue and observing the biological tissue and the object; The image processing device according to claim 1 , wherein the object is a catheter inserted into the lumen of the living tissue.

9. the three-dimensional data is constructed based on data obtained by a sensor inserted into the lumen of the biological tissue and observing the biological tissue, and includes, as the first data, data that is updated every time new data is obtained by the sensor; 8. The image processing device according to claim 1, wherein the control unit acquires designation data that designates the landmark, constructs the third data based on the acquired designation data, and includes the third data in the three-dimensional data.

10. The image processing device according to any one of claims 1 to 9; a sensor for observing the biological tissue and the object; An image processing system comprising:

11. The image processing system of claim 10 further comprising the display.

12. 1. An image display method for displaying three-dimensional data, including first data representing biological tissue, and second data representing an object located in a lumen of the biological tissue, or third data representing a landmark associated with the biological tissue, on a display as a three-dimensional image, comprising: a computer that refers to the three-dimensional data to identify a positional relationship between the biological tissue and the object or the landmark; the computer determines, according to the identified positional relationship, whether or not there is an intervening tissue, which is a part of the biological tissue that is interposed between the object or the landmark and a viewpoint that is set when the three-dimensional image is displayed; An image display method in which, when the computer determines that the intervening tissue exists, it controls to display an image representing the object or the landmark at a position corresponding to the intervening tissue in the three-dimensional image.

13. a computer that displays, on a display, three-dimensional data including first data representing a biological tissue and second data representing an object located in a lumen of the biological tissue or third data representing a landmark associated with the biological tissue as a three-dimensional image; A process of identifying a positional relationship between the biological tissue and the object or the landmark by referring to the three-dimensional data; a process of determining whether or not there is an intervening tissue, which is a part of the biological tissue that is interposed between the object or the landmark and a viewpoint that is set when the three-dimensional image is displayed, according to the identified positional relationship; a process of controlling, when it is determined that the intervening tissue exists, to display an image representing the object or the landmark at a position corresponding to the intervening tissue in the three-dimensional image; An image processing program that executes the following.

Images