Image processing apparatus, image processing system, image display method and image processing program

Abstract

An image processing apparatus that displays three-dimensional data representing biological tissue as three-dimensional graphics on a display screen, the image processing apparatus comprising a control unit that detects a series of positions of a catheter inserted into the biological tissue based on time-series data obtained by sensors that observe the surrounding area while moving within the lumen of the biological tissue, and switches between a first mode and a second mode, the first mode being a mode that displays a first graphic representing the series of positions in the three-dimensional image, and the second mode being a mode that displays a second graphic representing one of the series of positions in the three-dimensional image.

Description

Technical Field

[0001] This invention relates to an image processing apparatus, an image processing system, an image display method, and an image processing program. Background Technology

[0002] Patent documents 1 to 3 describe techniques for generating three-dimensional images of heart chambers or blood vessels using a US imaging system. "US" is an abbreviation for ultrasound.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: U.S. Patent Application Publication No. 2010 / 0215238

[0006] Patent Document 2: US Patent No. 6,385,332

[0007] Patent Document 3: US Patent No. 6251072 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] IVUS is widely used for treatments targeting the intracardiac chambers, coronary vessels, and lower limb arteries. "IVUS" is an abbreviation for intravascular ultrasound. IVUS refers to an instrument or method that provides two-dimensional images in a plane perpendicular to the long axis of the catheter.

[0010] Currently, surgeons need to reconstruct three-dimensional structures by overlaying two-dimensional IVUS images in their minds while performing surgery, which presents a significant challenge, particularly for younger or less experienced doctors. To overcome this obstacle, it is proposed to automatically generate three-dimensional images representing the structures of biological tissues such as heart chambers or blood vessels from the two-dimensional IVUS images and display these generated three-dimensional images to the surgeon. In cases where catheters used for procedures such as ablation are inserted into biological tissue, it is also proposed to display the catheter in the three-dimensional image.

[0011] Suppose we use past pullback information to display a series of catheter positions. If the pullback unit remains in the same position for a certain period, the correlation between the series of catheter positions extracted from the past pullback information and the current position of the catheter almost disappears. In particular, if the catheter is operated with the pullback unit in a fixed position, the correlation between the past series of positions and the current position is significantly lost. As a result, it becomes difficult to determine the position of the catheter.

[0012] The purpose of this invention is to make it easier to identify the location of ducts when displaying them in a three-dimensional image of biological tissue.

[0013] Methods for solving problems

[0014] As one aspect of the present invention, the image processing apparatus is an image processing apparatus that displays three-dimensional data representing biological tissue as three-dimensional graphics on a display. The image processing apparatus includes a control unit that detects a series of positions of a catheter inserted into the biological tissue based on data obtained in a time series by a sensor that observes the surrounding area while moving within the cavity of the biological tissue, and switches between a first mode and a second mode. The first mode is a mode in which a first graphic representing the series of positions is displayed in the three-dimensional image, and the second mode is a mode in which a second graphic representing one of the series of positions is displayed in the three-dimensional image.

[0015] As one implementation, when the control unit switches the display mode from the first mode to the second mode, the first graphic is not displayed.

[0016] As one embodiment, the control unit causes the display of a linear or tubular graphic connecting the series of positions as the first graphic.

[0017] As one embodiment, the control unit causes the display of a spherical graphic representing the one position as the second graphic, and displays at least one concentric circle or concentric sphere surrounding the spherical graphic.

[0018] As one embodiment, the control unit makes a second graphic displaying a shape showing that the diameter of the catheter near the end is greater than the diameter of the opposite side.

[0019] As one embodiment, the control unit displays a bullet-shaped graphic representing the one position as the second graphic.

[0020] As one embodiment, when displaying a graphic representing a position other than the tip of the catheter as the second graphic, the control unit causes the second graphic to be displayed in a different color than when displaying a graphic representing the position of the tip of the catheter.

[0021] As one implementation, the control unit switches the display mode according to changes in the movement of the sensor.

[0022] As one implementation, the change in the movement of the sensor includes the sensor moving at a speed exceeding a threshold.

[0023] As one implementation, if the moving speed of the sensor exceeds the threshold, the control unit switches the display mode based on whether the moving speed of the sensor is below the upper limit value.

[0024] As one implementation, the control unit switches the display mode based on whether the conduit exists in the image plane captured by the sensor.

[0025] As one implementation, when the duct is present in the image plane, the control unit switches the display mode according to the magnitude of the movement of the duct in the image plane.

[0026] As one implementation, when the conduit is not present in the image plane, the control unit switches the display mode based on the elapsed time since the last position among the series of positions was detected.

[0027] As one embodiment, the control unit adjusts at least one of the size and color of the second graphic in the second mode according to the distance from the inner wall surface of the biological tissue.

[0028] As one implementation, the control unit adjusts at least one of the size and color of the second graphic in the second mode based on whether or not it is in contact with the inner wall surface of the biological tissue.

[0029] As one implementation, the control unit also displays the shadow of the second graphic in the second mode.

[0030] As one implementation, the control unit adjusts the frequency at which the second graphic is updated along with the data updates in the second mode.

[0031] An image processing system according to one aspect of the present invention comprises: a sensor that acquires tomographic data of the biological tissue while moving within the cavity of the biological tissue; and the image processing apparatus described above that generates the three-dimensional data based on the tomographic data acquired by the sensor.

[0032] As one embodiment, the image processing system further includes the display.

[0033] As one aspect of the present invention, the image display method is a method for displaying three-dimensional data representing biological tissue as a three-dimensional image on a display screen. The method involves a computer detecting a series of locations of a catheter inserted into the biological tissue based on time-series data obtained by sensors that observe the surrounding area while moving within the lumen of the biological tissue. The computer switches between a first mode and a second mode, wherein the first mode displays a first graphic representing the series of locations in the three-dimensional image, and the second mode displays a second graphic representing one of the series of locations in the three-dimensional image.

[0034] As one aspect of the present invention, an image processing program causes a computer displaying three-dimensional data representing biological tissue as a three-dimensional image on a monitor to perform the following processing: processing to detect a series of locations of a catheter inserted into the biological tissue based on data obtained in a time series from sensors that observe the surrounding area while moving within the lumen of the biological tissue; and processing to switch display modes between a first mode and a second mode, wherein the first mode is a mode in which a first graphic representing the series of locations is displayed in the three-dimensional image, and the second mode is a mode in which a second graphic representing one of the series of locations is displayed in the three-dimensional image.

[0035] Invention Effects

[0036] According to the present invention, the location of ducts is easily known when displaying ducts in a three-dimensional image of biological tissue. Attached Figure Description

[0037] [ Figure 1 [This is a perspective view of an image processing system as one embodiment of the present invention.]

[0038] [ Figure 2 [This is a perspective view of the probe and driving unit of an image processing system as one aspect of the present invention.]

[0039] [ Figure 3 [A block diagram illustrating the configuration of an image processing system as an embodiment of the present invention.]

[0040] [ Figure 4 [This is a diagram illustrating a three-dimensional image and a first graphic shown in one embodiment.]

[0041] [ Figure 5 [This is a diagram illustrating a three-dimensional image and a second graphic shown in one embodiment.]

[0042] [ Figure 6 This is a flowchart illustrating the operation of an image processing system as one aspect of the present invention.

[0043] [ Figure 7 This is a flowchart illustrating the operation of an image processing system as one aspect of the present invention.

[0044] [ Figure 8 To show Figure 7 The flowchart shows the steps of the first mode determination process.

[0045] [ Figure 9 [This is a diagram illustrating an example where no conduit exists in the image plane.]

[0046] [ Figure 10 To show Figure 7 The flowchart shows the steps of the second mode determination process.

[0047] [ Figure 11 [A diagram illustrating an example of a sensor moving too fast.]

[0048] [ Figure 12 [A diagram showing a three-dimensional image and a second graphic in a modified example.]

[0049] [ Figure 13 [A diagram showing a three-dimensional image and a second graphic in a modified example.]

[0050] [ Figure 14 [A diagram showing a three-dimensional image and a second graphic in a modified example.]

[0051] [ Figure 15 [A diagram showing a three-dimensional image and a second graphic in a modified example.] Detailed Implementation

[0052] Hereinafter, an embodiment that is a specific example of the present invention will be described with reference to the accompanying drawings.

[0053] In each figure, the same or equivalent parts are labeled with the same reference numerals. In the description of this embodiment, descriptions of the same or equivalent parts are appropriately omitted or simplified.

[0054] See Figure 1 , Figure 3 , Figure 4 and Figure 5 This section provides an overview of the implementation method.

[0055] The image processing apparatus 11 involved in this embodiment is a computer that displays three-dimensional data 52 representing biological tissue 60 as a three-dimensional image 53 on a display 16. The image processing apparatus 11 detects a series of positions of a catheter 61 inserted into the biological tissue 60 by observing data obtained in a time series from surrounding sensors while moving within the lumen of the biological tissue 60. The image processing apparatus 11 switches between a first mode and a second mode, wherein the first mode displays a first graphic 71 representing a series of positions in the three-dimensional image 53, and the second mode displays a second graphic 72 representing one of the positions in the three-dimensional image 53.

[0056] According to this embodiment, the position of the catheter 61 is easily known when it is displayed in the three-dimensional image 53 of the biological tissue 60. For example, if the user is the surgeon and operates the catheter 61 in a fixed pull-back position, the current position of the catheter 61 is displayed instead of displaying a series of positions of the catheter 61 using past pull-back information. Therefore, the user can perform surgery without being misled by past information that is not related to the current position of the catheter 61.

[0057] In this embodiment, an ultrasonic transducer 25 is used as a sensor that moves within the cavity of the biological tissue 60 while observing the surroundings.

[0058] Biological tissues 60 include organs such as blood vessels or the heart. Figure 4 and Figure 5 In the example, 60% of the biological tissue is blood vessels.

[0059] Catheter 61 is, for example, an ablation catheter.

[0060] exist Figure 4 and Figure 5 In this diagram, the X direction and the Y direction, which are orthogonal to the X direction, correspond to the transverse width of the biological tissue 60. The Z direction, which is orthogonal to both the X and Y directions, corresponds to the longitudinal length of the biological tissue 60.

[0061] exist Figure 4In this example, the current display mode is the first mode. Before a certain time point T1, based on data obtained in a time sequence from the ultrasonic transducer 25, a series of positions P1, P2, P3, and P4 of the conduit 61 are detected. Position P1 is the position of the tip 62 of the conduit 61 at time point T1. Position P2 is the position of the tip 62 of the conduit 61 at time point T2, which is earlier than time point T1. Position P3 is the position of the tip 62 of the conduit 61 at time point T3, which is earlier than time point T2. Position P4 is the position of the tip 62 of the conduit 61 at time point T4, which is earlier than time point T3. The display 16 displays a first graphic 71 representing the series of positions P1, P2, P3, and P4 as the position of the conduit 61 at time point T1.

[0062] exist Figure 5 In this example, the current display mode is the second mode. During a certain period starting from time point T1, the same position P1 is continuously detected based on data obtained from the ultrasonic transducer 25 in a time sequence. At a subsequent time point T1', a different position P1' is detected based on data obtained from the ultrasonic transducer 25. Position P1' is the position of the front end 62 of the conduit 61 at time point T1', a time period after time point T1. The Z-direction position is the same at positions P1' and P1. That is, the corresponding positions of the ultrasonic transducer 25 are the same at positions P1' and P1. After displaying a second graphic 72 representing one position P1 from a series of positions P1, P2, P3, and P4 as the position of the conduit 61 during a certain period starting from time point T1, the display 16 displays a second graphic 72 representing one position P1' from a series of positions P1', P1, P2, P3, and P4 as the position of the conduit 61 at time point T1'.

[0063] See Figure 1 This section explains the configuration of the image processing system 10 involved in this embodiment.

[0064] The image processing system 10 includes an image processing device 11, a cable 12, a drive unit 13, a keyboard 14, a mouse 15, and a monitor 16.

[0065] The image processing device 11 is a computer specifically designed for image diagnosis in this embodiment, but it can also be a general-purpose computer such as a PC. "PC" is an abbreviation for personal computer.

[0066] Cable 12 is used to connect image processing device 11 and drive unit 13.

[0067] Drive unit 13 is used for, for example Figure 2The device shown connects to and drives probe 20. The drive unit 13 is also called an MDU. "MDU" is an abbreviation for motor drive unit. Probe 20 is used in IVUS. Probe 20 is also called an IVUS catheter or an imaging diagnostic catheter.

[0068] The keyboard 14, mouse 15, and monitor 16 are connected to the image processing device 11 via any cable or wirelessly. The monitor 16 is, for example, an LCD, an OLED display, or an HMD. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electroluminescence. "HMD" is an abbreviation for head-mounted display.

[0069] As an option, the image processing system 10 further includes a connection terminal 17 and a cart unit 18.

[0070] Connection terminal 17 is used to connect the image processing device 11 to an external machine. Connection terminal 17 is, for example, a USB terminal. "USB" is an abbreviation for Universal Serial Bus. The external machine is, for example, a recording medium such as a disk drive, magneto-optical disk drive, or optical disk drive.

[0071] The vehicle unit 18 is a cart equipped with casters for movement. An image processing device 11, a cable 12, and a drive unit 13 are installed in the main body of the vehicle unit 18. A keyboard 14, a mouse 15, and a monitor 16 are installed on the table at the top of the vehicle unit 18.

[0072] See Figure 2 This section explains the configuration of the probe 20 and the driving unit 13 involved in this embodiment.

[0073] The probe 20 includes a drive shaft 21, a hub 22, a sheath 23, an outer tube 24, an ultrasonic transducer 25, and a relay connector 26.

[0074] The drive shaft 21 passes through the sheath 23 inserted into the body cavity of the organism and the outer tube 24 connected to the base of the sheath 23, extending into the interior of the hub 22 disposed within the base of the probe 20. The front end of the drive shaft 21 has an ultrasonic transducer 25 for receiving and transmitting signals, and the drive shaft 21 is configured to rotate within the sheath 23 and the outer tube 24. A relay connector 26 connects the sheath 23 and the outer tube 24.

[0075] The hub 22, drive shaft 21, and ultrasonic transducer 25 are interconnected in such a way that each unit can move forward and backward axially. Therefore, when an operation is performed, for example, pressing the hub 22 towards the front end, the drive shaft 21 and ultrasonic transducer 25 move towards the front end within the sheath 23. For example, when an operation is performed, pulling the hub 22 towards the base end, as indicated by the arrow, the drive shaft 21 and ultrasonic transducer 25 move towards the base end within the sheath 23.

[0076] The drive unit 13 includes a scanning unit 31, a sliding unit 32, and a bottom cover 33.

[0077] The scanning unit 31 is connected to the image processing device 11 via cable 12. The scanning unit 31 includes a probe connection part 34 connected to the probe 20 and a scanning motor 35 that serves as a drive source for rotating the drive shaft 21.

[0078] The probe connection 34 is detachably connected to the probe 20 via an insertion port 36 in the hub 22 located at the base of the probe 20. The base of the drive shaft 21 is rotatably supported inside the hub 22, and the rotational force of the scanning motor 35 is transmitted to the drive shaft 21. Furthermore, the drive shaft 21 receives and transmits signals to the image processing device 11 via a cable 12. In the image processing device 11, based on the signals transmitted from the drive shaft 21, a tomographic image of the biological lumen is generated and processed.

[0079] The sliding unit 32 supports the scanning unit 31 in a manner that allows the scanning unit 31 to move forward and backward freely, and is mechanically and electrically connected to the scanning unit 31. The sliding unit 32 includes a probe clamp 37, a sliding motor 38, and a switch assembly 39.

[0080] The probe clamp 37 is arranged coaxially with the probe connector 34 at a position further forward than the probe connector 34, and supports the probe 20 connected to the probe connector 34.

[0081] The sliding motor 38 is the drive source that generates axial driving force. The scanning unit 31 moves forward and backward by driving the sliding motor 38, and the drive shaft 21 moves forward and backward axially along with this movement. The sliding motor 38 is, for example, a servo motor.

[0082] The switch group 39 includes, for example, a forward switch and a pullback switch pressed during the forward and backward operation of the scanning unit 31, and a scan switch pressed at the start and end of image depiction. This example is not particularly limited; the switch group 39 may include various switches as needed.

[0083] When the forward switch is pressed, the sliding motor 38 rotates forward, and the scanning unit 31 moves forward. On the other hand, when the return switch is pressed, the sliding motor 38 rotates in reverse, and the scanning unit 31 moves backward.

[0084] When the scan switch is pressed, image writing begins, the scan motor 35 is driven, and the slide motor 38 is driven to retract the scan unit 31. The operator or user connects the probe 20 to the scan unit 31 beforehand, begins image writing, and moves the drive shaft 21 axially towards the base end while rotating it. Pressing the scan switch again stops and terminates the image writing for both the scan motor 35 and the slide motor 38.

[0085] The bottom cover 33 covers the entire circumference of the bottom surface and the side surface of the sliding unit 32, and can freely approach and separate relative to the bottom surface of the sliding unit 32.

[0086] See Figure 3 This explains the configuration of the image processing device 11.

[0087] The image processing apparatus 11 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, and an output unit 45.

[0088] The control unit 41 includes at least one dedicated circuit, or a combination thereof. The processor may be a general-purpose processor such as a CPU or GPU, or a processor specifically designed for a particular process. "CPU" is an abbreviation for Central Processing Unit. "GPU" is an abbreviation for Graphics Processing Unit. The dedicated circuit may be, for example, an FPGA or an ASIC. "FPGA" is an abbreviation for Field-Programmable Gate Array. "ASIC" is an abbreviation for Application-Specific Integrated Circuit. The control unit 41 controls the various parts of the image processing system 10, including the image processing device 11, while performing processing related to the operation of the image processing device 11.

[0089] Storage unit 42 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of them. The semiconductor memory is, for example, RAM or ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read-only memory. RAM is, for example, SRAM or DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. ROM is, for example, EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read-only memory. Storage unit 42 is, for example, used as a main storage device, an auxiliary storage device, or a cache memory. Storage unit 42 stores data such as tomographic data 51 used for the operation of image processing device 11, and data such as three-dimensional data 52 and three-dimensional images 53 obtained according to the operation of image processing device 11.

[0090] The communication unit 43 includes at least one communication interface. This communication interface may be, for example, a wired LAN interface, a wireless LAN interface, or an image diagnostic interface for receiving IVUS signals and performing A / D conversion. "LAN" is an abbreviation for local area network. "A / D" is an abbreviation for analog-to-digital conversion. The communication unit 43 receives data for the operation of the image processing apparatus 11 and also transmits data obtained based on the operation of the image processing apparatus 11. In this embodiment, the drive unit 13 is connected to the image diagnostic interface included in the communication unit 43.

[0091] The input unit 44 includes at least one input interface. The input interface may be, for example, a USB interface, an HDMI (registered trademark) interface, or a Bluetooth (registered trademark) interface, which corresponds to short-range wireless communication. "HDMI" is an abbreviation for High-Definition Multimedia Interface. The input unit 44 accepts user operations, such as data input for the operation of the image processing device 11. In this embodiment, the USB interface or the short-range wireless communication interface included in the input unit 44 is connected to the keyboard 14 and the mouse 15. If the touchscreen and display 16 are integrated, the USB interface or HDMI (registered trademark) interface included in the input unit 44 may also be connected to the display 16.

[0092] The output unit 45 includes at least one output interface. The output interface may be, for example, a USB interface, an HDMI interface, or a Bluetooth interface, which are interfaces corresponding to short-range wireless communication. The output unit 45 outputs data obtained according to the operation of the image processing device 11. In this embodiment, the display 16 is connected to the USB interface or HDMI interface included in the output unit 45.

[0093] By executing the image processing program according to this embodiment using a processor equivalent to the control unit 41, the functions of the image processing apparatus 11 can be realized. That is, the functions of the image processing apparatus 11 can be implemented by software. The image processing program is a program that causes a computer to execute each process of the image processing apparatus 11, thereby enabling the computer to function as the image processing apparatus 11. In other words, the computer functions as the image processing apparatus 11 by executing each process of the image processing apparatus 11 according to the image processing program.

[0094] Programs can be stored on non-transitory computer-readable media. Examples of non-transitory computer-readable media include flash memory, magnetic recording devices, optical discs, magneto-optical recording media, or ROM. Program distribution can be achieved, for example, by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs containing the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for Digital Versatile Disc. "CD-ROM" is an abbreviation for Compact Disc Read Only Memory. Programs can be distributed by storing them in server storage and transferring them from the server to other computers. Programs can also be provided as program products.

[0095] Computers may temporarily store programs stored on portable media or transferred from servers in main storage. The computer then uses its processor to read the program stored in main storage and executes it. The computer can directly read programs from portable media and execute them. Alternatively, the computer can execute programs sequentially each time they are transferred from a server. Processing can also be performed without transferring programs from the server to the computer, through so-called ASP-type services that only execute instructions and obtain results. "ASP" is an abbreviation for Application Service Provider. The program contains information provided for computer processing and is based on the program's content. For example, data that has the nature of specifying computer processing but is not a direct instruction to the computer is equivalent to "based on the program's content."

[0096] Some or all of the functions of the image processing apparatus 11 can be implemented by a dedicated circuit equivalent to the control unit 41. That is, some or all of the functions of the image processing apparatus 11 can be implemented by hardware.

[0097] See Figures 6 to 11 This section explains the operation of the image processing system 10 according to this embodiment. The operation of the image processing system 10 is equivalent to the image display method according to this embodiment.

[0098] Figure 6 Before the procedure begins, the user primes the probe 20. Then, the probe 20 is embedded in the probe connector 34 and probe clamp 37 of the drive unit 13, and connected and fixed to the drive unit 13. The probe 20 is then inserted into the target site within biological tissue 60, such as a blood vessel or cardiac chamber.

[0099] In step S101, a so-called return operation is performed by pressing the scan switch included in switch group 39 and further pressing the return switch included in switch group 39. The probe 20 transmits ultrasound waves inside the biological tissue 60 using an ultrasonic transducer 25 that retracts axially through the return operation. The ultrasonic transducer 25 transmits ultrasound waves radially while moving inside the biological tissue 60. The ultrasonic transducer 25 receives reflected waves from the transmitted ultrasound waves. The probe 20 inputs the signal of the reflected wave received by the ultrasonic transducer 25 to the image processing device 11. The control unit 41 of the image processing device 11 processes the input signal and sequentially generates cross-sectional images of the biological tissue 60, thereby obtaining tomographic data 51 containing multiple cross-sectional images.

[0100] Specifically, probe 20 rotates the ultrasonic transducer 25 circumferentially and moves it axially inside the biological tissue 60, while simultaneously transmitting ultrasonic waves from the center of rotation outwards in multiple directions. Probe 20 receives reflected waves from reflectors present in multiple directions within the biological tissue 60 via the ultrasonic transducer 25. Probe 20 transmits the received reflected wave signals to image processing device 11 via drive unit 13 and cable 12. Communication unit 43 of image processing device 11 receives the signal transmitted by probe 20. Communication unit 43 performs A / D conversion on the received signal. Communication unit 43 inputs the A / D converted signal to control unit 41. Control unit 41 processes the input signal and calculates the intensity distribution of reflected waves from reflectors present in the direction of ultrasonic wave transmission of ultrasonic transducer 25. Control unit 41 sequentially generates a two-dimensional image with a luminance distribution corresponding to the calculated intensity distribution as a cross-sectional image of biological tissue 60, thereby obtaining a dataset of cross-sectional images, i.e., tomographic data 51. The control unit 41 stores the acquired tomographic data 51 in the storage unit 42.

[0101] In this embodiment, the signal of the reflected wave received by the ultrasonic transducer 25 is equivalent to the original data of the tomographic data 51, and the cross-sectional image generated by the image processing device 11 after processing the signal of the reflected wave is equivalent to the processed data of the tomographic data 51.

[0102] As a variation of this embodiment, the control unit 41 of the image processing apparatus 11 may also directly store the signal input from the probe 20 as tomographic data 51 in the storage unit 42. Alternatively, the control unit 41 may also store data representing the intensity distribution of reflected waves calculated by processing the signal input from the probe 20 as tomographic data 51 in the storage unit 42. That is, the tomographic data 51 is not limited to a dataset of cross-sectional images of biological tissue 60, but may also be data representing the cross-section of biological tissue 60 at each moving position of the ultrasonic transducer 25 in any form.

[0103] As a variation of this embodiment, an ultrasonic transducer that transmits ultrasonic waves in multiple directions without rotation can be used instead of an ultrasonic transducer 25 that transmits ultrasonic waves in multiple directions while rotating circumferentially.

[0104] As a variation of this embodiment, the tomographic data 51 can also be acquired using OFDI or OCT instead of IVUS. "OFDI" is an abbreviation for optical frequency domain imaging. "OCT" is an abbreviation for optical coherence tomography. When using OFDI or OCT, instead of an ultrasonic transducer 25 that transmits ultrasound waves into the cavity of the biological tissue 60 to acquire the tomographic data 51 while moving within the cavity of the biological tissue 60, a sensor that emits light into the cavity of the biological tissue 60 to acquire the tomographic data 51 is used as the sensor.

[0105] As a variation of this embodiment, other devices may be used to generate the same dataset, and the image processing device 11 may obtain this dataset from the other device to generate the dataset of cross-sectional images of the biological tissue 60 instead of the image processing device 11. That is, other devices may be used to process the IVUS signal and generate cross-sectional images of the biological tissue 60, and the generated cross-sectional images may be input into the image processing device 11 to replace the control unit 41 of the image processing device 11 in processing the IVUS signal and generating cross-sectional images of the biological tissue 60.

[0106] In step S102, the control unit 41 of the image processing device 11 generates three-dimensional data 52 of the biological tissue 60 based on the tomographic data 51 obtained in step S1.

[0107] Specifically, the control unit 41 of the image processing apparatus 11 overlays and three-dimensionalizes the cross-sectional images of the biological tissue 60 contained in the tomographic data 51 stored in the storage unit 42 to generate three-dimensional data 52 of the biological tissue 60. As a method for three-dimensionalization, any of the following methods can be used: surface rendering or volume rendering, and accompanying methods such as texture mapping including environment mapping and bump mapping. The control unit 41 stores the generated three-dimensional data 52 in the storage unit 42.

[0108] In step S103, the control unit 41 of the image processing device 11 displays the three-dimensional data 52 generated in step S102 as a three-dimensional image 53 on the display 16.

[0109] Specifically, the control unit 41 of the image processing apparatus 11 generates a three-dimensional image 53 based on the three-dimensional data 52 stored in the storage unit 42. The control unit 41 displays the generated three-dimensional image 53 on the display 16 via the output unit 45.

[0110] If there is an update to the fault data 51 in step S104, then proceed to steps S105 and S106.

[0111] In step S105, the control unit 41 of the image processing device 11 processes the signal input from the probe 20 in the same way as in step S101 to generate a new cross-sectional image of the biological tissue 60, thereby obtaining tomographic data 51 containing at least one new cross-sectional image.

[0112] In step S106, the control unit 41 of the image processing device 11 updates the three-dimensional data 52 of the biological tissue 60 based on the tomographic data 51 obtained in step S105. Furthermore, in step S103, the control unit 41 displays the updated three-dimensional data 52 from step S108 as a three-dimensional image 53 on the display 16.

[0113] As a variation of this embodiment, the control unit 41 of the image processing apparatus 11 may also accept user operation via the input unit 44 to set the angle of the displayed three-dimensional image 53. In this case, the control unit 41 adjusts the angle of the displayed three-dimensional image 53 to the set angle. Then, the control unit 41 displays the three-dimensional image 53 on the display 16 at the set angle.

[0114] As a variation of this embodiment, the control unit 41 of the image processing apparatus 11 may also accept, via the input unit 44, an operation to set an opening in the three-dimensional image 53 that exposes the inner cavity of the biological tissue 60, as a user operation. In this case, the control unit 41 forms the set opening in the three-dimensional data 52. The control unit 41 displays the three-dimensional data 52 as a three-dimensional image 53 on the display 16, and adjusts the viewpoint when displaying the three-dimensional image 53 on the display 16 according to the position of the opening formed in the three-dimensional data 52. The "viewpoint" refers to the position of an imaginary camera arranged in three-dimensional space. For example, the control unit 41 arranges the viewpoint on a straight line extending from the inner surface of the biological tissue 60 through the opening to the outside of the biological tissue 60. Thus, the user can virtually observe the inner surface of the biological tissue 60 while observing the inside of the biological tissue 60 through the opening.

[0115] In step S103, with the location of the catheter 61 inserted into the biological tissue 60 displayed in the three-dimensional image 53, the process begins... Figure 7 The process. Whether the position of catheter 61 is displayed can be automatically determined based on the presence or absence of catheter 61, or it can be manually determined based on user operation.

[0116] In step S103, following step S102, the processing from steps S201 to S205 is performed on the multiple cross-sectional images included in the fault data 51 obtained in step S101. In step S103, following step S106, the processing from steps S201 to S205 is performed on the new cross-sectional images included in the fault data 51 obtained in step S105.

[0117] The following describes the steps of processing the cross-sectional image obtained at a certain time point Ti using steps S201 to S205.

[0118] In step S201, the control unit 41 of the image processing device 11 detects the position Pi of the conduit 61 using any method such as machine learning based on the cross-sectional image obtained at time point Ti.

[0119] In step S202, if the current display mode is the first mode, then the first mode determination process in step S203 is performed. If the current display mode is the second mode, then the second mode determination process in step S204 is performed.

[0120] Figure 8 The steps of the first mode determination process in step S203 are shown.

[0121] In step S301, the control unit 41 of the image processing device 11 determines whether the conduit 61 exists in the image plane captured by the ultrasonic transducer 25 at time point Ti. If the position Pi of the conduit 61 is detected in step S201, then the conduit 61 exists in the image plane captured by the ultrasonic transducer 25, and therefore the processing in step S302 is performed. If the position Pi of the conduit 61 is not detected in step S201, then... Figure 9 As shown, the conduit 61 is not present in the image plane captured by the ultrasonic transducer 25, therefore the processing in step S304 is performed.

[0122] In step S302, the control unit 41 of the image processing device 11 determines whether the moving speed V of the ultrasonic transducer 25 at time point Ti exceeds the threshold Tz. If it does not exceed the threshold, the process proceeds to step S303. If it does exceed the threshold, the display mode is not switched, and the first mode is maintained, proceeding to step S205.

[0123] The moving speed V of the ultrasonic transducer 25 refers to the distance that the ultrasonic transducer 25 moves per unit time along the axial direction of the catheter 61, that is, in the direction of travel within the lumen of the biological tissue 60, rather than the distance that the ultrasonic transducer 25 moves in the direction of rotation.

[0124] The threshold Tz only needs to be low enough to detect the cessation of the ultrasonic transducer 25's movement; in this embodiment, it is 0. The threshold Tz can also be set to a value greater than 0.01 mm / FPS and less than 1 cm / FPS. "FPS" is an abbreviation for frames per second. FPS is the number of cross-sectional images updated per second.

[0125] In step S303, the control unit 41 of the image processing device 11 determines whether the magnitude of the movement ABLxyV of the catheter 61 per unit time in the image plane captured by the ultrasonic transducer 25 at time point Ti exceeds the threshold Txy. If it does not exceed the threshold, the display mode is not switched, and the processing in step S205 is performed while maintaining the first mode. If it exceeds the threshold, the control unit 41 switches the display mode from the first mode to the second mode. Then, the processing in step S205 is performed. For example, when the operator performs ablation, the ablation catheter is moved in the same plane to circumferentially burn the inner surface of the biological tissue 60, so it is beneficial to switch the display mode to the second mode. On the other hand, if there is no movement of the ablation catheter in the same plane, it is not necessary to switch the display mode to the second mode.

[0126] In step S304, the control unit 41 of the image processing device 11 determines whether the elapsed time since the last time the position of the conduit 61 was detected is greater than or equal to a threshold D. If it is not greater than or equal to the threshold D, the display mode is not switched, and the first mode is maintained while proceeding to step S205. If it is greater than or equal to the threshold D, the control unit 41 switches the display mode from the first mode to the second mode. Then, the processing of step S205 is performed.

[0127] The threshold D can be any value, but in this embodiment it is set to a value between 0 seconds and 10 seconds.

[0128] Thus, in this embodiment, the control unit 41 of the image processing apparatus 11 switches the display mode based on changes in the movement of the ultrasonic transducer 25. Changes in the movement of the ultrasonic transducer 25 include the ultrasonic transducer 25's movement speed V exceeding a threshold Tz. Furthermore, the control unit 41 switches the display mode based on whether the conduit 61 is present within the image plane captured by the ultrasonic transducer 25.

[0129] Specifically, when the conduit 61 is present in the image plane captured by the ultrasonic transducer 25, the control unit 41 switches the display mode according to the magnitude of the movement of the conduit 61 in the image plane. More specifically, when the conduit 61 is present in the image plane captured by the ultrasonic transducer 25, the moving speed V of the ultrasonic transducer 25 does not exceed the threshold Tz, and the magnitude of the movement ABLxyV of the conduit 61 per unit time in the image plane exceeds the threshold Txy, the control unit 41 switches the display mode from the first mode to the second mode.

[0130] Furthermore, when the conduit 61 is not present in the image plane captured by the ultrasonic transducer 25, the control unit 41 switches the display mode based on the elapsed time since the last position of a series of detected positions of the conduit 61. More specifically, when the conduit 61 is not present in the image plane captured by the ultrasonic transducer 25, and the elapsed time since the last confirmation of the conduit 61 is greater than or equal to a threshold D, the control unit 41 switches the display mode from the first mode to the second mode.

[0131] As a variation of this embodiment, the control unit 41 may switch the display mode from the first mode to the second mode only when the moving speed V of the ultrasonic transducer 25 does not exceed the threshold Tz. That is, the processing steps S301, S303 and S304 may be omitted.

[0132] As a variation of this embodiment, the control unit 41 may switch the display mode from the first mode to the second mode only when the conduit 61 is within the image plane captured by the ultrasonic transducer 25 and the moving speed V of the ultrasonic transducer 25 does not exceed the threshold Tz. That is, the processing in step S303 may be omitted.

[0133] Figure 10 This indicates the second mode determination process in step S204.

[0134] In step S401, the control unit 41 of the image processing device 11 determines whether the moving speed V of the ultrasonic transducer 25 at time point Ti exceeds the threshold Tz. If it does, the processing in step S402 is performed. If it does not exceed the threshold, the display mode is not switched, and the processing in step S205 is performed while maintaining the second mode.

[0135] In step S402, the control unit 41 of the image processing device 11 determines whether the conduit 61 exists in the image plane captured by the ultrasonic transducer 25 at time point Ti. If the position Pi of the conduit 61 is detected in step S201, then the conduit 61 exists in the image plane captured by the ultrasonic transducer 25, and therefore the processing in step S403 is performed. If the position Pi of the conduit 61 is not detected in step S201, then... Figure 9 As shown, there is no conduit 61 in the image plane captured by the ultrasonic transducer 25, so the display mode is not switched, and the second mode is maintained to perform the processing of step S205.

[0136] In step S403, the control unit 41 of the image processing device 11 determines whether the moving speed V of the ultrasonic transducer 25 at time point Ti exceeds the upper limit value Tzmax. If it does, the display mode is not switched, and the second mode is maintained, proceeding to step S205. If it does not exceed the upper limit, the control unit 41 switches the display mode from the second mode to the first mode. Then, the processing of step S205 is performed. For example, as... Figure 11 As shown, if the ultrasonic transducer 25 moves too fast, even if the previous position P2 of the catheter 61 is directly connected to the current position P1, the correct trajectory of the catheter 61 may not be obtained. In such cases, keeping the display mode in the second mode will prompt the operator to move the ultrasonic transducer 25 to rescan.

[0137] Thus, in this embodiment, the control unit 41 of the image processing apparatus 11 switches the display mode based on changes in the movement of the ultrasonic transducer 25. Changes in the movement of the ultrasonic transducer 25 include the ultrasonic transducer 25's movement speed V exceeding a threshold Tz. Furthermore, the control unit 41 switches the display mode based on whether the conduit 61 is present within the image plane captured by the ultrasonic transducer 25.

[0138] Specifically, when the moving speed V of the ultrasonic transducer 25 exceeds the threshold Tz, the control unit 41 switches the display mode based on whether the moving speed V of the ultrasonic transducer 25 is below the upper limit value Tzmax. More specifically, when the moving speed V of the ultrasonic transducer 25 exceeds the threshold Tz, the conduit 61 is present in the image plane captured by the ultrasonic transducer 25, and the moving speed V of the ultrasonic transducer 25 is below the maximum allowable value, i.e., the upper limit value Tzmax, the control unit 41 switches the display mode from the second mode to the first mode.

[0139] As a variation of this embodiment, the control unit 41 may switch the display mode from the second mode to the first mode only when the moving speed V of the ultrasonic transducer 25 exceeds the threshold Tz. That is, the processing steps S402 and S403 may be omitted.

[0140] As a variation of this embodiment, the control unit 41 may switch the display mode from the second mode to the first mode only when the moving speed V of the ultrasonic transducer 25 exceeds the threshold Tz and the conduit 61 is present in the image plane captured by the ultrasonic transducer 25. That is, the processing of step S403 may be omitted.

[0141] In step S205, if the current display mode is the first mode, the control unit 41 of the image processing device 11... Figure 4As in the example, a first graphic 71 representing a series of positions of the conduit 61, including the position Pi detected in step S201 as the nearest position, is displayed in the three-dimensional image 53. Figure 4 In the example, when time point T1 is equivalent to time point Ti, position P1 is equivalent to position Pi.

[0142] exist Figure 4 In the example, the first mode is to connect the past positions of the conduit and display the conduit 61 as a tube. However, as another example, the first mode can also be to display the conduit 61 as a line. That is, as the first graphic 71, the control unit 41 of the image processing device 11 can also display a line-shaped graphic connecting a series of positions of the conduit 61, instead of displaying a tubular graphic connecting a series of positions of the conduit 61.

[0143] In step S205, if the current display mode is the second mode, the control unit 41 of the image processing device 11... Figure 5 As shown in the example, a second graphic 72 is displayed in the 3D image 53, representing the position Pi detected in step S201 as the nearest position. Figure 5 In the example, when time point T1 corresponds to time point Ti, position P1 corresponds to position Pi. When time point T1' corresponds to time point Ti, position P1' corresponds to position Pi.

[0144] exist Figure 5 In the example, the second mode is one that displays the current duct position like a ball, but it can also be like... Figure 12 As in the example, the second mode is a mode that displays the current catheter position like Saturn. That is, the control unit 41 of the image processing device 11 can also display a spherical graphic representing one of a series of positions of the catheter 61 as a second graphic 72, and display at least one concentric circle 73 surrounding the spherical graphic. Alternatively, the control unit 41 can also display a spherical graphic representing one of a series of positions of the catheter 61 as a second graphic 72, and display at least one concentric sphere surrounding the spherical graphic. By having concentric circles 73 or concentric spheres, the distance to the blood vessel wall is easily known in the second mode. Figure 12 In the example, three concentric circles 73 are shown, but the number of concentric circles 73 can be less than three or more than three.

[0145] As another example where the distance to the blood vessel wall is easily known in the second mode, the size of the second graphic 72 can be changed according to the distance to the blood vessel wall, or the color of the second graphic 72 can be changed according to the distance to the blood vessel wall. Alternatively, a shadow of the second graphic 72 can be added in the second mode. That is, the control unit 41 of the image processing device 11 can also adjust at least one of the size and color of the second graphic 72 according to the distance from the inner wall surface 63 of the biological tissue 60 in the second mode. Alternatively, the control unit 41 can further display the shadow of the second graphic 72 in the second mode.

[0146] like Figure 13 As shown in the example, the second mode can also be a mode that displays the current position of the conduit, like a bullet. That is, the control unit 41 of the image processing device 11 can also display a graphic as the second graphic 72, where the thickness of the side near the end of the conduit 61 is larger than the thickness of the opposite side. Specifically, the control unit 41 can also display a bullet-shaped graphic representing one of a series of positions of the conduit 61 as the second graphic 72. In particular, the control unit 41 can also display a bullet-shaped graphic representing the position of the tip 62 of the conduit 61 as the second graphic 72. By representing the tip 62 of the conduit 61 with a bullet-like shape, it is easy to know the direction of movement of the conduit 61, such as whether the conduit 61 came from the top or bottom.

[0147] exist Figure 5 In the example, the current catheter position is the position of the tip 62 of catheter 61, but sometimes the current catheter position is not the position of the tip 62 of catheter 61. Therefore, as... Figure 14 As in the example, the color can be changed even when the current catheter position is not the position of the tip 62 of the catheter 61. That is, the control unit 41 of the image processing device 11 can also display the second graphic 72 in a different color than when displaying the graphic indicating the position of the tip 62 of the catheter 61 as the second graphic 72, when displaying a graphic indicating the position of the tip 62 of the catheter 61 as the second graphic 72.

[0148] In the case of adding the shadow of the second graphic 72 in the second mode, such as Figure 15 As shown, the closer the second graphic 72 is to the blood vessel wall on the side furthest from the imaginary light source 75, the smaller the shadow. Therefore, it is preferable to represent a large catheter position when the distance from the tissue is far, and a small catheter position when the distance from the tissue is close.

[0149] When the display mode is changed from the first mode to the second mode, it is easier to observe if the second graphic 72 is made larger than the corresponding part of the first graphic 71.

[0150] As an example where it is easy to know whether the catheter 61 is in contact with the tissue, if the catheter 61 is in contact with the blood vessel wall, the size of the second graphic 72 can be changed, or the color of the second graphic 72 can also be changed. That is, the control unit 41 of the image processing device 11 can also adjust at least one of the size and color of the second graphic 72 in the second mode according to whether or not it is in contact with the inner wall surface 63 of the biological tissue 60.

[0151] In this embodiment, the control unit 41 of the image processing device 11 switches the display mode according to the change in the movement of the ultrasonic transducer 25. However, as a variation of this embodiment, the control unit 41 may also switch the display mode according to the change in the positional relationship between the ultrasonic transducer 25 and the guide tube 61.

[0152] In this embodiment, the control unit 41 of the image processing apparatus 11 determines whether the moving speed V of the ultrasonic transducer 25 exceeds a threshold Tz based on a time sequence. If the moving speed V of the ultrasonic transducer 25 does not exceed the threshold Tz, and other conditions are also met, the control unit 41 switches the display mode from the first mode to the second mode. That is, when the ultrasonic transducer 25 remains in the same location, the control unit 41 represents the conduit 61 using the most recent position instead of a series of positions. It should be noted that "other conditions" may not be required.

[0153] In a modified example where the display mode is switched based on the change in the positional relationship between the ultrasonic transducer 25 and the conduit 61, the control unit 41 determines, in a time sequence, whether the magnitude ABLxyV of the conduit 61's motion per unit time within the image plane captured by the ultrasonic transducer 25 exceeds a threshold Txy. If the magnitude ABLxyV of the conduit 61's motion exceeds the threshold Txy, and if other conditions are also met, the control unit 41 switches the display mode from the first mode to the second mode. That is, when the conduit 61 moves significantly in the Z direction, the control unit 41 represents the conduit 61 using the most recent position, rather than using a series of positions. It should be noted that "other conditions" may not be required.

[0154] As described above, in this embodiment, the control unit 41 of the image processing apparatus 11 displays three-dimensional data 52 representing the biological tissue 60 as a three-dimensional image 53 on the display 16. The control unit 41 detects a series of positions of the catheter 61 inserted into the biological tissue 60 based on tomographic data 51 obtained in a time series by observing the surrounding ultrasound transducers 25 while moving within the cavity of the biological tissue 60. The control unit 41 switches between a first mode that displays a first graphic 71 representing a series of positions in the three-dimensional image 53 and a second mode that displays a second graphic 72 representing one position in the series of positions in the three-dimensional image 53.

[0155] According to this embodiment, when the duct 61 is displayed in the three-dimensional image 53 of the biological tissue 60, the position of the duct 61 can be easily determined.

[0156] In this embodiment, the control unit 41 of the image processing apparatus 11 checks the movement status of the pull-back units one by one. When the pull-back unit stops moving, the control unit 41 displays the position of the conduit 61 using a display method that only shows the current position as a "dot". When the pull-back unit starts moving again, the control unit 41 displays the travel history of the conduit 61 using a display method that shows a series of positions as a "line". When the pull-back unit returns to its original path, the control unit 41 temporarily resets the display of the travel history and displays the travel history from that point in time.

[0157] According to this embodiment, if the user is the surgeon, the user can perform the surgery without being misled by past information that is not related to the current position of the catheter 61 when the movement of the pull-back unit stops.

[0158] In this embodiment, if tomographic data 51 containing at least one new cross-sectional image is obtained, then for that new cross-sectional image, the processing steps S201 to S205 are performed. That is, whenever the tomographic data 51 is updated in step S105, the first mode determination processing in step S203 or the second mode determination processing in step S204 is performed. Then, in step S205, if the current display mode is the first mode, then... Figure 4 As shown in the example, a first graphic 71 representing a series of positions of the conduit 61, including the position Pi detected in step S201 as the nearest position, is displayed in the 3D image 53. This means that, as a result of the first mode determination process accompanying the update of the tomographic data 51, the first graphic 71 is updated while maintaining the first mode and performing the processing in step S205 without switching the display mode. On the other hand, in step S205, if the current display mode is the second mode, then as... Figure 5 As shown in the example, a second graphic 72 is displayed in the 3D image 53, representing the position Pi detected in step S201 as the nearest position. This means that, as a result of the second mode determination process accompanying the update of the tomographic data 51, the second graphic 72 is updated while the second mode remains unchanged during the processing of step S205 without switching the display mode.

[0159] Even though the first graphic 71, such as "line," is frequently updated in the first mode, it doesn't cause much concern for the user. However, if the second graphic 72, such as "dot," is frequently updated in the second mode, it may appear flickering, potentially causing discomfort or fatigue for the user. If the user is the surgeon, their concentration is reduced, which could adversely affect the surgery. Therefore, as a variation of this embodiment, the update speed of the second graphic 72 can be intentionally slowed down in the second mode. That is, the control unit 41 of the image processing apparatus 11 can also adjust the frequency at which the second graphic 72 is updated along with the tomographic data 51 in the second mode. For example, the control unit 41 can adjust the update speed of the second graphic 72 to be reduced by a fixed or variable ratio relative to the update speed of the tomographic data 51. Alternatively, the control unit 41 can update the second graphic 72 fewer times than the tomographic data 51 is updated multiple times. In the first mode, the control unit 41 can update the first graphic 71 each time the tomographic data 51 is updated, or it can adjust the frequency at which the first graphic 71 is updated along with the tomographic data 51.

[0160] This variation will be further explained.

[0161] In the second mode, the control unit 41 of the image processing device 11 causes a frame rate delay specifically for the conduit 61. For example, when updating the 3D image 53 at 30 FPS, the intense vibrations during display in the second mode can make the display difficult to observe. Therefore, the control unit 41 controls the display by reducing the frame rate only for the conduit position. Specifically, the control unit 41 controls the display of the second graphic 72 at a frame rate of approximately 70% to 20% of the frame rate for displaying the 3D image 53.

[0162] Alternatively, the control unit 41 of the image processing device 11 may cause a delay in the second mode. For example, when updating the three-dimensional image 53 at 30 FPS, severe vibrations in the display of the second mode may cause the display to be difficult to observe. Therefore, the control unit 41 limits the movement distance in the second mode. Specifically, the control unit 41 sets an arbitrary distance within the range of 1 / 10 to 1 / 1000 of the inner diameter of the biological tissue 60 as the maximum movement distance. For example, the movement distance from the duct position P[t] of the second mode at a certain time point t to the duct position P[t+1] of the second mode at a time point t+1 a unit time later is longer than the maximum movement distance. In this case, the control unit 41 moves the second graphic 72 gradually to P[t+1] over a period from 1 s to 10 ms, instead of moving the second graphic 72 instantaneously from P[t] to P[t+1]. This operation enables smooth movement. However, in cases where the actual duct position moves rapidly and significantly, this method may produce inconsistencies with reality. Therefore, when the moving distance from P[t] to P[t+1] is significantly long, the control unit 41 instantly moves the second graphic 72 from P[t] to P[t+1]. That is, the control unit 41 directly displays the conduit position. Specifically, if the moving distance per unit time is less than the maximum moving distance, or is a reference distance longer than the maximum moving distance or more, the control unit 41 does not cause a delay and directly displays the conduit position. If the moving distance per unit time is greater than the maximum moving distance but less than the reference distance, the control unit 41 causes a delay.

[0163] In the first mode, the control unit 41 of the image processing apparatus 11 may cause a frame rate delay only for the conduit 61. In the first mode, the first graphic 71 is generated considering the past position of the conduit, but the calculation used to generate the first graphic 71 may not converge within an update speed of 30 FPS or the like. Therefore, the control unit 41 may also control the display to reduce the frame rate only for the conduit position. For example, when updating the 3D image 53 at 30 FPS, the control unit 41 may also control the display of the first graphic 71 at a frame rate slower than 30 FPS, such as 10 FPS.

[0164] As described above, in this modified example, the control unit 41 of the image processing apparatus 11 displays three-dimensional data 52 representing the biological tissue 60 as a three-dimensional image 53 on the display 16. The control unit 41 constructs three-dimensional biological tissue data representing the biological tissue 60 and first graphic data representing a series of positions of a catheter 61 inserted into the biological tissue 60, respectively, based on tomographic data 51 obtained by observing the surrounding area while moving within the cavity of the biological tissue 60 using ultrasound transducers 25 in a time series. The control unit 41 sequentially displays the three-dimensional biological tissue data and the first graphic data as a three-dimensional image 53 and a first graphic 71 on the display 16. The control unit 41 controls the update rate of the first graphic 71 to be lower than the update rate of the three-dimensional image 53.

[0165] According to this variation, even if the update speed of the three-dimensional image 53 is not fast enough to update the first graphic 71, the image display will not lag.

[0166] This invention is not limited to the embodiments described. For example, multiple blocks described in the block diagram can be integrated, or a single block can be divided. Instead of executing multiple steps based on the flowchart in a time sequence as described, the steps can be executed in parallel or in a different order, depending on the processing capability of the device executing each step or as needed. Furthermore, modifications can be made without departing from the spirit of this invention.

[0167] Explanation of reference numerals in the attached figures

[0168] 10 Image Processing System

[0169] 11 Image processing device

[0170] 12 Cables

[0171] 13 Drive Units

[0172] 14 Keyboards

[0173] 15. Mouse

[0174] 16 monitors

[0175] 17 Connection terminals

[0176] 18 car units

[0177] 20 probes

[0178] 21 drive shafts

[0179] 22 Hub

[0180] 23 Sheath

[0181] 24 outer tubes

[0182] 25 Ultrasonic transducers

[0183] 26 Relay Connector

[0184] 31 Scanning Units

[0185] 32 sliding units

[0186] 33 Bottom Cover

[0187] 34 Probe Connection Part

[0188] 35 Scanning motor

[0189] 36 Insertion Port

[0190] 37 Probe clamp

[0191] 38 Sliding electric motor

[0192] 39 Switchgear

[0193] 41 Control Department

[0194] 42 Storage Section

[0195] 43 Ministry of Communications

[0196] 44 Input Section

[0197] 45 Output Section

[0198] 51 Fault Data

[0199] 52 Three-dimensional data

[0200] 53 Three-dimensional images

[0201] 60 biological tissues

[0202] 61 catheters

[0203] 62 Frontend

[0204] 63 inner wall

[0205] 71 First Figure

[0206] 72 Second Figure

[0207] 73 concentric circles

[0208] 74 Third Figure

[0209] 75 Light Source

Claims

1. An image processing apparatus that displays three-dimensional data representing biological tissue as a three-dimensional image on a display screen. The image processing apparatus includes a control unit, which is configured to: The three-dimensional data is generated based on tomographic data of the biological tissue acquired by a sensor inserted into the lumen of the biological tissue, the sensor observing the surroundings while moving within the lumen of the biological tissue. As the sensor moves within the lumen of the biological tissue, the tomographic data obtained from the sensor in a time series is used to detect a series of locations of ducts inserted separately from the probe into the biological tissue. Furthermore, the control unit switches the display mode between the first mode and the second mode. The first mode is a mode in which a first graphic representing the series of positions is displayed in the three-dimensional image. The second mode is a mode in which a second graphic representing one of the series of locations is displayed in the three-dimensional image.

2. The image processing apparatus as claimed in claim 1, wherein, When the control unit switches the display mode from the first mode to the second mode, the first graphic is not displayed.

3. The image processing apparatus as claimed in claim 1 or 2, wherein, The control unit causes the display of a linear or tubular graphic connecting the series of positions as the first graphic.

4. The image processing apparatus as claimed in claim 1 or 2, wherein, The control unit causes the display of a spherical graphic representing the one position as the second graphic, and displays at least one concentric circle or concentric sphere surrounding the spherical graphic.

5. The image processing apparatus as claimed in claim 1 or 2, wherein, The control unit causes a second graphic to be displayed showing that the diameter of the catheter near the end is greater than the diameter of the opposite side.

6. The image processing apparatus as claimed in claim 5, wherein, The control unit displays a bullet-shaped graphic representing the position as the second graphic.

7. The image processing apparatus as claimed in claim 1 or 2, wherein, When a graphic representing a location other than the tip of the catheter is displayed as the second graphic, the control unit causes the second graphic to be displayed in a different color than when the graphic representing the location of the tip of the catheter is displayed.

8. The image processing apparatus as claimed in claim 1 or 2, wherein, The control unit switches the display mode according to changes in the movement of the sensor.

9. The image processing apparatus as claimed in claim 8, wherein, Changes in the movement of the sensor include the sensor's movement speed exceeding a threshold.

10. The image processing apparatus of claim 9, wherein, If the moving speed of the sensor exceeds the threshold, the control unit switches the display mode based on whether the moving speed of the sensor is below the upper limit value.

11. The image processing apparatus as claimed in claim 1 or 2, wherein, The control unit switches the display mode based on whether the conduit exists within the image plane captured by the sensor.

12. The image processing apparatus of claim 11, wherein, When the duct is present in the image plane, the control unit switches the display mode according to the magnitude of the duct's movement within the image plane.

13. The image processing apparatus of claim 11, wherein, If the conduit is not present in the image plane, the control unit switches the display mode based on the elapsed time since the last position in the series of positions was detected.

14. The image processing apparatus as claimed in claim 1 or 2, wherein, In the second mode, the control unit adjusts at least one of the size and color of the second graphic based on the distance from the inner wall of the biological tissue.

15. The image processing apparatus as claimed in claim 1 or 2, wherein, In the second mode, the control unit adjusts at least one of the size and color of the second graphic based on whether or not it is in contact with the inner wall surface of the biological tissue.

16. The image processing apparatus as claimed in claim 1 or 2, wherein, The control unit also displays a shadow of the second graphic in the second mode.

17. The image processing apparatus as claimed in claim 1 or 2, wherein, In the second mode, the control unit adjusts the frequency at which the second graphic is updated along with the data.

18. An image processing system, which includes: A sensor that acquires tomographic data of a biological tissue while moving within the tissue's lumen; and The image processing apparatus according to any one of claims 1 to 17 generates the three-dimensional data based on tomographic data acquired by the sensor.

19. The image processing system of claim 18, further comprising the display.

20. An image display method, which is an image display method for displaying three-dimensional data representing biological tissue as a three-dimensional image on a display screen, wherein, The computer generates the three-dimensional data based on tomographic data of the biological tissue acquired by a sensor inserted into the lumen of the biological tissue, the sensor observing the surroundings while moving within the lumen of the biological tissue. As the sensor moves within the lumen of the biological tissue, the tomographic data obtained from the sensor in a time series is used to detect a series of locations of ducts inserted separately from the probe into the biological tissue. The computer switches between a first mode and a second mode, wherein the first mode displays a first graphic representing the series of locations in the three-dimensional image, and the second mode displays a second graphic representing one of the series of locations in the three-dimensional image.

21. A storage medium containing an image processing program, said image processing program causing a computer that displays three-dimensional data representing biological tissue as a three-dimensional image on a display to perform the following processing: The three-dimensional data is generated based on tomographic data of the biological tissue acquired by a sensor inserted into the lumen of the biological tissue, the sensor observing the surroundings while moving within the lumen of the biological tissue. As the sensor moves within the lumen of the biological tissue, the tomographic data obtained from the sensor in a time series is used to detect a series of locations of ducts inserted separately from the probe into the biological tissue; and The process of switching display modes between a first mode and a second mode, wherein the first mode is a mode in which a first graphic representing a series of locations is displayed in the three-dimensional image, and the second mode is a mode in which a second graphic representing one of the series of locations is displayed in the three-dimensional image.

Images