Image processing device, image processing system, image processing method, and image processing program

Abstract

This image processing device is provided with a control unit that: sets a virtual axis that exists within a three-dimensional space on the basis of the locations of an axis of a catheter, which has been inserted into a body lumen, in respective two-dimensional images included in a plurality of two-dimensional images obtained as a sensor provided to the catheter moves along the axis of the catheter; visualizing the three-dimensional structure of the body lumen on the basis of the plurality of two-dimensional images and generating a three-dimensional image having a plurality of cross-sections that are spaced apart from one another along the virtual axis within the three-dimensional space and also are orthogonal to the virtual axis; setting a rotational angle by which to rotate the three-dimensional image; and rotating each cross-section included in the plurality of cross-sections by the rotation angle, centered on the location of the virtual axis in the relevant cross-section, to adjust the shape of the three-dimensional image.

Description

Image processing device, image processing system, image processing method, and image processing program

[0001] The present disclosure relates to an image processing device, an image processing system, an image processing method, and an image processing program.

[0002] Patent Document 1 discloses a method for identifying the direction of a patient's ventral side in a three-dimensional image of biological tissue that is generated and displayed based on a two-dimensional ultrasound image acquired by a transducer of an ultrasound catheter.

[0003] International Publication No. 2022 / 045182

[0004] During catheterization procedures such as stent graft placement, 3D CT images taken before surgery or DSA images taken during surgery can be merged with real-time X-ray images to reduce X-ray exposure or assist in catheter manipulation. "CT" stands for computed tomography. "DSA" stands for digital subtraction angiography. DSA images are created by subtracting a mask X-ray image taken before contrast administration from a contrast image taken after contrast administration. Merging 3D CT or DSA images with X-ray images allows for simultaneous confirmation of the detailed vascular shape and real-time status. However, when a stiff guidewire or catheter is inserted into a blood vessel, the vessel stretches, potentially creating a large discrepancy between the 3D CT image and the actual organ shape. X-ray exposure during 3D CT imaging can also be an issue. Because DSA images use a large amount of contrast agent, contrast-induced nephropathy can occur.

[0005] Therefore, instead of displaying a 3D CT image taken before surgery, it is conceivable to display a 3D image generated by a method that does not involve X-ray exposure, such as that disclosed in Patent Document 1, during surgery. However, in cases where the alignment between the X-ray image and the 3D image is incomplete, it may be necessary to adjust the shape of the 3D image. For example, it is conceivable to rotate the 2D ultrasound image so that the abdominal direction is at the top, and then stack the rotated 2D ultrasound images to construct a 3D image. In this case, the orientation of the displayed 3D image may be slightly misaligned with the orientation of the abdominal direction, and in order to eliminate such misalignment, it is necessary to adjust the shape of the 3D image.

[0006] An object of the present disclosure is to enable adjustment of the shape of a three-dimensional image generated by imaging the three-dimensional structure of a biological lumen.

[0007] Some aspects of the present disclosure are set forth below.

[0008] [1] An image processing device comprising: a control unit that sets a virtual axis existing in three-dimensional space based on the position of the axis of a catheter in each of a plurality of two-dimensional images included in the two-dimensional images obtained by moving a sensor provided on a catheter inserted into a biological lumen along the axis of the catheter; images the three-dimensional structure of the biological lumen based on the plurality of two-dimensional images, and generates a three-dimensional image in the three-dimensional space having a plurality of cross sections spaced apart along the virtual axis and perpendicular to the virtual axis; sets a rotation angle for rotating the three-dimensional image; and adjusts the shape of the three-dimensional image by rotating each cross section included in the plurality of cross sections by the rotation angle around the position of the virtual axis in the respective cross section.

[0009] [2] The image processing device according to [1], wherein, when the control unit receives an operation by a user to manually specify an angle, the control unit sets the rotation angle to the angle specified by the operation.

[0010] [3] The image processing device according to [1], wherein the control unit automatically sets the rotation angle based on the three-dimensional image and an X-ray image taken by an X-ray imaging device that sees through the biological lumen.

[0011] [4] The image processing device according to any one of [1] to [3], wherein, when the control unit receives an operation by a user to manually change the shape of the virtual axis before generating the three-dimensional image, the control unit adjusts the shape of the virtual axis to the shape changed by the operation.

[0012] [5] The image processing device described in any of [1] to [3], wherein the plurality of two-dimensional images are obtained by moving the sensor along the axis of the catheter through a section from a first point to a second point within the biological lumen, and the control unit identifies the position of the axis of the catheter in each of the plurality of two-dimensional images included in the plurality of two-dimensional images as a sensor position, sets a movement line in the three-dimensional space as the virtual axis, extending in a shape corresponding to the shape of the axis of the catheter in the section from a position in the three-dimensional space corresponding to the sensor position identified for the two-dimensional image obtained by the sensor when the sensor is located at the first point, and when generating the three-dimensional image, sets the shape of the three-dimensional image to a shape in which the sensor position identified for each of the plurality of two-dimensional images included in the plurality of two-dimensional images is located on the virtual axis.

[0013] [6] The image processing device according to [5], wherein each cross section included in the plurality of cross sections is a cross section corresponding to a sensor position identified for each two-dimensional image included in the plurality of two-dimensional images.

[0014] [7] The image processing device according to any one of [1] to [6], wherein the control unit, when adjusting the shape of the three-dimensional image, displays the three-dimensional image on a display in a state where the three-dimensional image is superimposed on an X-ray image taken by an X-ray imaging device that sees through the biological lumen.

[0015] [8] The image processing device described in [7], wherein the control unit adjusts the shape of the three-dimensional image, and then displays the three-dimensional image on the display superimposed on the X-ray image or another X-ray image taken by the X-ray imaging device.

[0016] [9] An image processing system comprising: the image processing device according to [7] or [8]; and the display.

[0017]

[10] An image processing method comprising: setting a virtual axis existing in three-dimensional space based on the position of the axis of a catheter in each of a plurality of two-dimensional images included in a plurality of two-dimensional images obtained by moving a sensor provided on a catheter inserted into a biological lumen along the axis of the catheter; imaging the three-dimensional structure of the biological lumen based on the plurality of two-dimensional images to generate a three-dimensional image having a plurality of cross sections in the three-dimensional space that are spaced apart along the virtual axis and perpendicular to the virtual axis; setting a rotation angle for rotating the three-dimensional image; and adjusting the shape of the three-dimensional image by rotating each cross section included in the plurality of cross sections by the rotation angle around the position of the virtual axis in the respective cross section.

[0018]

[11] An image processing program that causes a computer to perform operations including: setting a virtual axis that exists in three-dimensional space based on the position of the axis of a catheter in each of a plurality of two-dimensional images included in the two-dimensional images obtained by moving a sensor provided on a catheter inserted into a biological lumen along the axis of the catheter; imaging the three-dimensional structure of the biological lumen based on the plurality of two-dimensional images to generate a three-dimensional image in the three-dimensional space having a plurality of cross sections that are spaced apart along the virtual axis and perpendicular to the virtual axis; setting a rotation angle for rotating the three-dimensional image; and adjusting the shape of the three-dimensional image by rotating each cross section included in the plurality of cross sections by the rotation angle around the position of the virtual axis in the respective cross section.

[0019] According to the present disclosure, when a three-dimensional image is generated by imaging the three-dimensional structure of a biological lumen, it is possible to adjust the shape of the three-dimensional image.

[0020] FIG. 1 is a block diagram showing a configuration of an image processing system according to an embodiment of the present disclosure. FIG. 2 is a flowchart showing the operation of an image processing device according to an embodiment of the present disclosure. FIG. 3 is a diagram showing an example of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 4 is a diagram showing an example of updating, in step S4, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 5 is a diagram showing an example of a three-dimensional image generated by an image processing device according to an embodiment of the present disclosure. FIG. 6 is a diagram showing an example of shape adjustment, in step S7, of a three-dimensional image generated by an image processing device according to an embodiment of the present disclosure. FIG. 7 is a diagram showing an example of updating, in step S7, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 8 is a flowchart showing a first example of detailed procedures of step S6. FIG. 9 is a diagram showing an example of updating, in step S603, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 10 is a diagram showing an example of updating, in step S604, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 11 is a flowchart showing a second example of detailed procedures of step S6. FIG. 12 is a flowchart showing a third example of detailed procedures of step S6. FIG. 13 is a diagram showing an example of updating, in step S623, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 14 is a flowchart showing a fourth example of detailed procedures of step S6. FIG. 15 is a flowchart showing a fifth example of detailed procedures of step S6. FIG. 16 is a diagram showing an example of contrast agent injection for step S9. 10 is a diagram showing an example of a three-dimensional image display before alignment in step S9. FIG. 11 is a diagram showing an example of a three-dimensional image display after alignment in step S9. FIG. 12 is a flowchart showing a modified example of the operation of the image processing device according to an embodiment of the present disclosure.

[0021] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0022] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0023] The configuration of an image processing system 10 according to this embodiment will be described with reference to FIG.

[0024] The image processing system 10 includes an image processing device 20, a sensor 28, an X-ray imaging device 30, an input device 40, and a display 50. The image processing device 20 is connected to the sensor 28, the X-ray imaging device 30, the input device 40, and the display 50 via a cable, a network, or wirelessly.

[0025] The image processing device 20 is, for example, a general-purpose computer such as a PC, a server computer such as a cloud server, or a dedicated computer. "PC" is an abbreviation for personal computer. The image processing device 20 may be installed in a medical facility such as a hospital, or may be installed in a facility separate from the medical facility, such as a data center.

[0026] The sensor 28 is, for example, an ultrasound transducer used in IVUS (intravascular ultrasound). The sensor 28 transmits ultrasound waves while moving along the axis 27 of a catheter 26 inserted into a body lumen such as a blood vessel, as shown in FIG. 3 , and receives reflected waves of the transmitted ultrasound waves.

[0027] The X-ray imaging device 30 is, for example, a C-arm. The X-ray imaging device 30 is installed in a medical facility and performs transillumination of a living body lumen.

[0028] The input device 40 is, for example, a pointing device such as a mouse, a keyboard, or a touch screen that is integrated with the display 50. The input device 40 is installed in a medical facility and is used by an operator such as a doctor or a clinical engineer to control the display of various information, including images, on the display 50.

[0029] The display 50 is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electroluminescent. The display 50 is installed in a medical facility and displays various information, including images, to an operator to support catheter surgery such as a stent graft placement procedure.

[0030] The configuration of an image processing apparatus 20 according to this embodiment will be described with reference to FIG.

[0031] The image processing device 20 includes a control unit 21 , a storage unit 22 , a communication unit 23 , an input unit 24 , and an output unit 25 .

[0032] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. An example of the programmable circuit is an FPGA. "FPGA" is an abbreviation for field-programmable gate array. An example of the dedicated circuit is an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 controls each part of the image processing device 20 and executes processing related to the operation of the image processing device 20.

[0033] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. Flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. Magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 22 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the image processing device 20 and data obtained by the operation of the image processing device 20 .

[0034] The communication unit 23 includes at least one communication module. The communication module is, for example, a module compatible with a wired LAN communication standard such as Ethernet (registered trademark) or a wireless LAN communication standard such as IEEE 802.11. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. The communication unit 23 receives data used in the operation of the image processing device 20 and transmits data obtained by the operation of the image processing device 20. The communication unit 23 is connected to the sensor 28 and the X-ray imaging device 30. Note that the communication unit 23 does not have to be a communication module as described above, as long as it can receive image signals from the sensor 28 and the X-ray imaging device 30.

[0035] The input unit 24 includes at least one input interface. The input interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface. The input unit 24 accepts an operation to input data used in the operation of the image processing device 20. The input unit 24 is connected to an input device 40.

[0036] The output unit 25 includes at least one output interface. The output interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). The output unit 25 outputs data obtained by the operation of the image processing device 20. The output unit 25 is connected to a display 50.

[0037] The functions of the image processing device 20 are realized by executing an image processing program according to this embodiment on a processor serving as the control unit 21. That is, the functions of the image processing device 20 are realized by software. The image processing program causes a computer to execute the operations of the image processing device 20, thereby causing the computer to function as the image processing device 20. That is, the computer functions as the image processing device 20 by executing the operations of the image processing device 20 in accordance with the image processing program.

[0038] The program can be stored on a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include flash memory, magnetic recording devices, optical disks, magneto-optical recording media, and ROMs. The program can be distributed by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs that store the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program can also be distributed by storing it in the storage of a server and transferring it from the server to another computer. The program can also be provided as a program product.

[0039] A computer temporarily stores a program stored on a portable medium or transferred from a server in its main storage device. The computer then reads the program stored in the main storage device with a processor and executes processing in accordance with the read program. The computer may also read the program directly from the portable medium and execute processing in accordance with the program. The computer may also execute processing in accordance with the received program each time a program is transferred from the server to the computer. Processing may also be executed using a so-called ASP-type service that realizes functions simply by issuing execution instructions and obtaining results, without transferring the program from the server to the computer. "ASP" is an abbreviation for application service provider. Programs include information used for processing by a computer that is equivalent to a program. For example, data that is not a direct instruction to a computer but has properties that define computer processing falls under the category of "equivalent to a program."

[0040] Some or all of the functions of the image processing device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, some or all of the functions of the image processing device 20 may be realized by hardware.

[0041] The operation of the image processing device 20 according to this embodiment will be described with reference to Fig. 2. The operation described below corresponds to the image processing method according to this embodiment. That is, the image processing method according to this embodiment includes steps S1 to S9 shown in Fig. 2.

[0042] In S1, the control unit 21 acquires one or more X-ray images taken by the X-ray imaging device 30. Specifically, the control unit 21 receives one or more X-ray images from the X-ray imaging device 30 via the communication unit 23.

[0043] In S2, the controller 21 acquires multiple two-dimensional images obtained by the sensor 28 while the sensor 28 moves through a section from the first point P1 to the second point P2 within the biological lumen. Specifically, the controller 21 receives, via the communication unit 23, reflected wave information from the sensor 28 regarding reflected waves received by the sensor 28 while the sensor 28 moves through a section from the first point P1 to the second point P2 within the biological lumen. The controller 21 generates multiple two-dimensional cross-sectional images by imaging the cross-sectional structure of the biological lumen based on the received reflected wave information. Alternatively, the controller 21 may receive, via the communication unit 23, multiple two-dimensional cross-sectional images generated by a separately provided IVUS device based on the reflected wave information from the IVUS device.

[0044] Step S1 may be executed before, during, or after S2. By the time both steps S1 and S2 are completed at the latest, the control unit 21 causes the display 50 to display a screen 51 such as that shown in FIG. 3 via the output unit 25.

[0045] The screen 51 includes a first area 52 and a second area 53 adjacent to the first area 52. In the example shown in FIG. 3 , the second area 53 is disposed on the right side of the first area 52, but the second area 53 may also be disposed on the left side of the first area 52.

[0046] The first area 52 displays the most recent two-dimensional image 60 from the multiple two-dimensional images acquired in S2 and a human body model image 61. The human body model image 61 is displayed so that its orientation is aligned with that of the two-dimensional image 60. In the example shown in FIG. 3 , the catheter 26 is inserted from the groin of the surgical patient, with the upper side of the two-dimensional image 60 corresponding to the patient's abdomen and the lower side of the two-dimensional image 60 corresponding to the patient's back. Therefore, the human body model image 61 is displayed so that the bottom (the soles of the feet) are visible, with the abdomen facing upward and the back facing downward. In the example shown in FIG. 3 , the human body model image 61 is positioned in the upper left corner of the two-dimensional image 60, but the human body model image 61 may be positioned anywhere near the two-dimensional image 60. The position of the human body model image 61 may be fixed or may be freely changeable.

[0047] A slider 54 is also displayed in the first area 52. The slider 54 indicates the position of the sensor 28 in the longitudinal direction of the catheter 26. The slider 54 is located at the end of the first area 52 that is closer to the second area 53. Therefore, in the example shown in FIG. 3 , the slider 54 is located at the right end of the first area 52.

[0048] The second area 53 displays the X-ray image 80 acquired in S1. The X-ray image 80 was acquired by the X-ray imaging device 30 when the axis 27 of the catheter 26 was present at least over the section from the first point P1 to the second point P2 within the biological lumen. The X-ray image 80 includes an image of at least a portion of the catheter 26, specifically, an image of the axis 27 of the catheter 26 and an image of the sensor 28. In the example shown in FIG. 3 , the X-ray image 80 further includes an image of the guidewire 29.

[0049] In S3, the control unit 21 visualizes the three-dimensional structure of the biological lumen based on the multiple two-dimensional images acquired in S2, to generate a three-dimensional image 70 as shown in Fig. 5. Specifically, the control unit 21 stacks the multiple cross-sectional images generated or received in S2 to generate the three-dimensional image 70. The three-dimensional image 70 is still provisional in S3, and its generation is completed in S7, as described below. In other words, the control unit 21 merely generates a provisional three-dimensional image in S3.

[0050] In S4, the control unit 21 causes the three-dimensional image 70 generated in S3 to be superimposed on the X-ray image 80 acquired in S1 and displayed on the display 50. For example, the control unit 21 updates the screen 51 displayed on the display 50 via the output unit 25 as shown in FIG.

[0051] In the second area 53, the 3D image 70 generated in S3 and the human body model image 71 are displayed superimposed on the X-ray image 80 acquired in S1. In the example shown in FIG. 4, the angle of the 3D image 70 corresponds to the angle when the surgical subject is viewed from the front, the upper side of the 3D image 70 corresponds to the side where the surgical subject's head is located, and the lower side of the 3D image 70 corresponds to the side where the surgical subject's feet are located. Therefore, the human body model image 71 is displayed so that the front is visible, with the head facing up and the feet facing down. In the example shown in FIG. 4, the human body model image 71 is positioned in the upper left corner of the 3D image 70, but the human body model image 71 may be positioned anywhere near the 3D image 70. The position of the human body model image 71 may be fixed or may be arbitrarily changeable. The 3D image 70 may be partially cropped along the longitudinal axis of the catheter 26 so that the internal structure of the biological lumen can be observed, and the cropped portion may be hidden.

[0052] A cross-sectional image 72 is also displayed in the second area 53. The cross-sectional image 72 is a reduced version of the two-dimensional image 60 displayed in the first area 52. A camera icon 73 indicating the position of the viewpoint is placed on the edge of the cross-sectional image 72. This makes it easy to see from which direction the three-dimensional image 70 displayed in the second area 53 is viewed in the cross-sectional image 72.

[0053] The image display orientation in which the three-dimensional image 70 is displayed on the display 50 is adjusted so that it can be viewed from a viewpoint corresponding to the imaging angle at which the X-ray image 80 was captured by the X-ray imaging device 30. The relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 have already been adjusted in the example shown in Fig. 4, but in reality they do not need to be adjusted yet. Fig. 4 shows a first position L1 corresponding to the first point P1 and a second position L2 corresponding to the second point P2 on the X-ray image 80, but the first position L1 and the second position L2 may be manually specified or automatically detected by step S6.

[0054] In S5, the control unit 21 identifies the position of the sensor 28 as the sensor position for each of the multiple two-dimensional images acquired in S2. That is, the control unit 21 identifies the position of the axis 27 of the catheter 26 in each of the multiple two-dimensional images acquired in S2 as the sensor position. Specifically, the control unit 21 identifies the position of the center of the two-dimensional image 60 acquired by the sensor 28 when the sensor 28 is located at a first point P1 as the first sensor position V1. The control unit 21 identifies the position of the center of the two-dimensional image 60 acquired by the sensor 28 when the sensor 28 is located at a second point P2 as the second sensor position V2. The control unit 21 identifies the positions of the center of the two-dimensional image 60 acquired by the sensor 28 when the sensor 28 is located at a third point P3 and a fourth point P4, which are different from the first point P1 and the second point P2, as the third sensor position V3 and the fourth sensor position V4, respectively. The control unit 21 may further specify, as the sensor position, the position of the center of the two-dimensional image 60 obtained by the sensor 28 when the sensor 28 is located at a point different from the first point P1, the second point P2, the third point P3, and the fourth point P4. Alternatively, the control unit 21 may not specify the fourth sensor position V4.

[0055] In S6, the control unit 21 sets a movement line 74 as shown in FIG. 6 in the three-dimensional image 70 generated in S3. The movement line 74 is a line extending from the position of the first sensor position V1 identified in S5 in the three-dimensional image 70 in a shape corresponding to the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen. The shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen is identified using at least one X-ray image acquired in S1. Specifically, the shape of the axis 27 of the catheter 26 from the first position L1 to the second position L2 on the X-ray image 80 acquired in S1 is identified as the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen. The movement line 74 corresponds to a virtual axis existing in three-dimensional space, and as will be described later, serves as the axis of rotation when rotating multiple cross sections in the three-dimensional image 70 in S9. In other words, the control unit 21 has set the virtual axis in S6.

[0056] In S7, the control unit 21 adjusts the shape of the 3D image 70 generated in S3 to match the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen, as determined using at least one X-ray image. Specifically, as shown in FIG. 6 , the control unit 21 adjusts the shape of the 3D image 70 generated in S3 by arranging each sensor position determined in S5 on the movement line 74 set in S6. For example, as shown in FIG. 7 , the control unit 21 updates the screen 51 displayed on the display 50 via the output unit 25. The 3D image 70 has multiple cross sections spaced apart from each other along a virtual axis and perpendicular to the virtual axis. In the example shown in FIG. 6 , each cross section included in the multiple cross sections in the 3D image 70 corresponds to the sensor position determined in S5. For example, the cross section corresponding to the first sensor position V1 perpendicularly intersects the tangent to the movement line 74, which corresponds to the virtual axis, at a position corresponding to the first sensor position V1 in three-dimensional space. Similarly, the cross section corresponding to the second sensor position V2 perpendicularly intersects with the tangent to the movement line 74, which corresponds to the virtual axis, at a position in the three-dimensional space corresponding to the second sensor position V2. The three-dimensional image 70 is no longer temporary in S7, and its generation has already been completed. In other words, the control unit 21 has generated the three-dimensional image in S7.

[0057] Step S7 may be executed after S6, but is preferably executed simultaneously with S6. That is, it is preferable that the control unit 21 updates the three-dimensional image 70 in accordance with changes in the movement line 74 while the movement line 74 is being set.

[0058] To reduce the amount of calculation required to adjust the shape of the three-dimensional image 70 in S7, the control unit 21 preferably converts the voxel data of the three-dimensional image 70 into triangular or quadrangular mesh data when generating the three-dimensional image 70 in S3. Any method, such as the Marching Cubes method, Surface Nets, or Delaunay triangulation, can be used to convert the voxel data into mesh data. Any method, such as RMF or Frenet-Serret, can be used to deform the mesh data when adjusting the shape of the three-dimensional image 70. "RMF" is an abbreviation for rotation minimizing frames.

[0059] As a modification of this embodiment, the control unit 21 may cause the three-dimensional image 70 and the X-ray image 80 to be separately displayed on the display 50 in S4. As another modification, the control unit 21 may further acquire another X-ray image taken by the X-ray imaging device 30 at an angle different from that of the X-ray image 80 in S1, and further adjust the shape of the three-dimensional image 70 in the depth direction using the another X-ray image in S4 to S7.

[0060] In S8, the control unit 21 sets a rotation angle for rotating the three-dimensional image 70 whose shape has been adjusted in S7. Specifically, when the control unit 21 receives an operation by the user to manually specify an angle, the control unit 21 sets the rotation angle to the angle specified by that operation. Alternatively, the control unit 21 may automatically set the rotation angle based on the three-dimensional image 70 and the X-ray image 80 or other X-ray image.

[0061] In S9, the control unit 21 rotates each of the cross sections included in the multiple cross sections in the three-dimensional image 70, the shape of which has been adjusted in S7, by the rotation angle set in S8 around the position of the virtual axis in each cross section, thereby adjusting the shape of the three-dimensional image 70. Specifically, as shown in Fig. 6, the control unit 21 adjusts the shape of the three-dimensional image 70 by rotating each of the cross sections included in the multiple cross sections in the three-dimensional image 70 by the rotation angle set in S8 around a position in the three-dimensional space corresponding to the sensor position identified in S5.

[0062] Step S9 may be executed after S8, but is preferably executed simultaneously with S8. That is, it is preferable that the control unit 21 updates the three-dimensional image 70 in accordance with changes in the rotation angle while the rotation angle is being set.

[0063] For example, suppose that a contrast agent is injected to obtain an X-ray image of a blood vessel 85 as shown in FIG. 16 , and the position and two-dimensional shape of a bifurcation 86 can be confirmed from the X-ray image. Suppose that a three-dimensional image of a blood vessel 75 as shown in FIG. 17 is displayed as a three-dimensional image 70 superimposed on the X-ray image. In the example shown in FIG. 17 , there is little overlap between the bifurcation 86 included in the X-ray image and the bifurcation 76 included in the three-dimensional image. Therefore, the control unit 21 accepts an operation to specify an angle numerically or using a GUI such as a bar via the input device 40 via the input unit 24. "GUI" is an abbreviation for graphical user interface. Each time the specified angle changes, the control unit 21 adjusts the shape of the three-dimensional image 70 by rotating each of the multiple cross sections included in the three-dimensional image 70 by the specified angle. When the voxel data of the three-dimensional image 70 has been converted into mesh data, the control unit 21 adjusts the shape of the three-dimensional image 70 by reconstructing the mesh data by rotating the mesh points included in each cross section around the position of the movement line 74 in each cross section, i.e., the position corresponding to the sensor position in three-dimensional space. Each time the control unit 21 adjusts the shape of the three-dimensional image 70, it updates the screen 51 displayed on the display 50 via the output unit 25. The user can manually and visually align the branch portion 86 included in the X-ray image with the branch portion 76 included in the three-dimensional image. As a result, as shown in FIG. 18 , the overlap between the branch portion 86 included in the X-ray image and the branch portion 76 included in the three-dimensional image can be improved. The contrast agent may be injected in a small amount sufficient to allow the position and two-dimensional shape of the branch portion 86 to be confirmed from the X-ray image.

[0064] Instead of accepting an operation to specify an angle, the control unit 21 may automatically change the angle, compare the overlap between the branch portion 86 included in the X-ray image and the branch portion 76 included in the three-dimensional image for each angle, and adjust the shape of the three-dimensional image 70 to maximize the overlap. That is, the alignment between the branch portion 86 included in the X-ray image and the branch portion 76 included in the three-dimensional image may be performed automatically. As a parameter, IoU or Precision may be used. "IoU" is an abbreviation for Intersection over Union.

[0065] As described above, in this embodiment, the control unit 21 sets a virtual axis existing in three-dimensional space based on the position of the axis 27 of the catheter 26 in each of a plurality of two-dimensional images obtained by moving the sensor 28 provided on the catheter 26 inserted into the biological lumen along the axis 27 of the catheter 26. The control unit 21 visualizes the three-dimensional structure of the biological lumen based on the plurality of two-dimensional images, and generates a three-dimensional image 70 having a plurality of cross sections in three-dimensional space that are spaced apart along the virtual axis and perpendicular to the virtual axis. The control unit 21 sets a rotation angle for rotating the three-dimensional image 70. The control unit 21 adjusts the shape of the three-dimensional image 70 by rotating each of the plurality of cross sections by the rotation angle around the position of the virtual axis in that cross section.

[0066] Therefore, according to this embodiment, when the three-dimensional structure of a biological lumen is visualized to generate a three-dimensional image 70, even if the alignment between the X-ray image 80 and the three-dimensional image 70 is incomplete, correction can be made by adjusting the shape of the three-dimensional image 70.

[0067] In this embodiment, when adjusting the shape of the three-dimensional image 70 in S9, the control unit 21 causes the three-dimensional image 70 to be superimposed on the X-ray image 80 and displayed on the display 50, but the display of the X-ray image 80 may be omitted and only the three-dimensional image 70 may be displayed on the display 50. Alternatively, after adjusting the shape of the three-dimensional image 70, the control unit 21 may cause the three-dimensional image 70 to be superimposed on the X-ray image 80 or another X-ray image captured by the X-ray imaging device 30 and displayed on the display 50.

[0068] In this embodiment, the control unit 21 visualizes the three-dimensional structure of the biological lumen to generate a provisional three-dimensional image in S3, and displays the provisional three-dimensional image on the display 50 in S4. However, the three-dimensional structure of the biological lumen may be three-dimensionally modeled to generate only volume data, and the display of the three-dimensional image may be omitted up to S7. If the alignment is automatically performed in S8 and S9, the display of the three-dimensional image may be omitted up to S9.

[0069] A first example of the detailed procedure of step S6 will be described with reference to FIG.

[0070] In S601, the control unit 21 associates the first position L1 with one end of the movement line 74 in response to an operation of manually specifying the first position L1 when the sensor 28 is located at the first point P1. Specifically, the control unit 21 accepts, via the input unit 24, an operation of clicking or touching, using the input device 40, a position at which the sensor 28 appears on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at the first point P1. The control unit 21 receives, via the communication unit 23, notification from the MDU driving the catheter 26 regarding the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is located at the first point P1. "MDU" is an abbreviation for motor drive unit. The control unit 21 identifies the first sensor position V1 identified in S5 within the cross section located at the position notified by the MDU in the 3D image 70 as the position of the first sensor position V1 in the 3D image 70. The control unit 21 stores the clicked or touched position as the first position L1 and the position of the first sensor position V1 in the three-dimensional image 70 as the position of one end of the movement line 74 in the memory unit 22 in association with each other.

[0071] In S602, the control unit 21 associates the second position L2 with the other end of the movement line 74 in response to an operation of manually specifying the second position L2 when the sensor 28 is located at the second point P2. Specifically, the control unit 21 accepts, via the input unit 24, an operation of clicking or touching, using the input device 40, the position at which the sensor 28 appears on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at the second point P2. The control unit 21 receives, via the communication unit 23, a notification from the MDU driving the catheter 26 regarding the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is located at the second point P2. The control unit 21 identifies the second sensor position V2 identified in S5 within the cross section located at the position notified by the MDU in the 3D image 70 as the position of the second sensor position V2 in the 3D image 70. The control unit 21 stores the clicked or touched position as the second position L2 and the position of the second sensor position V2 in the three-dimensional image 70 as the other end position of the movement line 74 in the memory unit 22 in association with each other.

[0072] In S603, the control unit 21 sets a connecting line 82 on the X-ray image 80 as shown in FIG. 9 . The connecting line 82 is a line connecting the first position L1 and the second position L2. For example, as shown in FIG. 9 , the control unit 21 places marks such as circles at the first position L1 and the second position L2 on the X-ray image 80 displayed on the screen 51, and draws a straight line connecting the placed marks as the connecting line 82. In the example shown in FIG. 9 , the X-ray image 80 was taken when the sensor 28 was located at the first point P1. However, the X-ray image 80 may be taken at any time as long as the axis 27 of the catheter 26 was located at least over the section from the first point P1 to the second point P2 within the biological lumen.

[0073] The control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 so that the position and length of the movement line 74 match the position and length of the connecting line 82. Specifically, the control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 so that the positions of both ends of the movement line 74 stored in the storage unit 22 match the first position L1 and second position L2 stored in the storage unit 22 in association with them.

[0074] In S604, the control unit 21 sets the shape of the movement line 74 in response to an operation to manually change the shape of the connecting line 82 set in S603. Specifically, the control unit 21 accepts an operation to bend the connecting line 82 by dragging any position on the connecting line 82 other than the first position L1 and the second position L2 with the input device 40 on the X-ray image 80 displayed on the screen 51 via the input unit 24. Every time the connecting line 82 is bent, the control unit 21 adjusts the shape of the movement line 74 so that it matches the bent shape of the connecting line 82.

[0075] After S604, when step S7 is executed, for example, the control unit 21 updates the screen 51 displayed on the display 50 via the output unit 25 as shown in FIG.

[0076] A second example of the detailed procedure of step S6 will be described with reference to FIG.

[0077] Step S611 is the same as step S601 shown in FIG. 8, and therefore a description thereof will be omitted.

[0078] In S612, the control unit 21 acquires ratio information indicating the ratio of the dimensions of the X-ray image 80 to the dimensions of the three-dimensional image 70. Specifically, when the actual scale of the X-ray image 80 is known, the control unit 21 calculates the ratio of the distance on the X-ray image 80 to the movement distance of the sensor 28 as the ratio information.

[0079] In S613, the control unit 21 associates a position on the X-ray image 80 that is a distance away from the first position L1 corresponding to the distance of the movement line 74 as a second position L2, with the other end of the movement line 74, based on the ratio information acquired in S612. Specifically, the control unit 21 receives notification from the MDU driving the catheter 26 via the communication unit 23 regarding the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is located at the second point P2. The control unit 21 identifies the second sensor position V2 identified in S5 within the cross section of the 3D image 70 that is located at the position notified by the MDU as the position of the second sensor position V2 in the 3D image 70. Based on the information notified by the MDU, the control unit 21 calculates the movement distance of the sensor 28 when it moves from the first point P1 to the second point P2. The control unit 21 converts the calculated movement distance into a distance on the X-ray image 80 according to the ratio calculated in S612. The control unit 21 stores in the storage unit 22 a position that is the converted distance away from the first position L1 specified in S611 as the second position L2, and a position in the three-dimensional image 70 of the second sensor position V2 as the other end position of the movement line 74, in association with each other. The direction in which the second position L2 exists relative to the first position L1 may be specified manually, or may be automatically determined by matching it with the position of the axis 27 of the catheter 26 on the X-ray image 80.

[0080] Steps S614 and S615 are similar to steps S603 and S604 shown in FIG. 8, respectively, and therefore will not be described here.

[0081] A third example of the detailed procedure of step S6 will be described with reference to FIG.

[0082] Steps S621 and S622 are similar to steps S601 and S602 shown in Fig. 8, and therefore will not be described here. Step S622 may be replaced with steps S612 and S613 shown in Fig. 11. In such a modification, the operation of manually specifying the second position L2 is not required.

[0083] In S623, in response to an operation of manually specifying at least one position on the X-ray image 80 that is different from the first position L1 and the second position L2, the control unit 21 sets the shape of the movement line 74 to a shape that corresponds to the first position L1, the second position L2, and the connecting line 82 that connects the at least one position. Specifically, the control unit 21 accepts, via the input unit 24, an operation of clicking or touching, with the input device 40, an arbitrary position on the shaft 27 of the catheter 26 other than the first position L1 and the second position L2 as a third position L3 on the X-ray image 80 displayed on the screen 51. The control unit 21 further accepts, via the input unit 24, an operation of clicking or touching, with the input device 40, an arbitrary position on the shaft 27 of the catheter 26 other than the first position L1, the second position L2, and the third position L3 as a fourth position L4 on the X-ray image 80 displayed on the screen 51. 13 , the control unit 21 places marks such as circles at a first position L1, a second position L2, a third position L3, and a fourth position L4 on the X-ray image 80 displayed on the screen 51, and draws a straight line connecting all of the placed marks as a connecting line 82. The control unit 21 may further accept, via the input unit 24, an operation of clicking or touching any position on the shaft 27 of the catheter 26 other than the first position L1, the second position L2, the third position L3, and the fourth position L4 with the input device 40 on the X-ray image 80 displayed on the screen 51. Alternatively, the control unit 21 may not accept an operation of clicking or touching the fourth position L4.

[0084] The control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 so that the position and length of the movement line 74 match the position and length of the connecting line 82. Specifically, the control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 so that the positions of both ends of the movement line 74 stored in the storage unit 22 match the first position L1 and second position L2 stored in the storage unit 22 in association with them.

[0085] The control unit 21 automatically changes the shape of the connecting line 82 each time a position other than the first position L1 and the second position L2 is specified, and changes the shape of the movement line 74 accordingly. Specifically, each time a position other than the first position L1 and the second position L2 is clicked or touched, the control unit 21 places a new mark such as a circle at the clicked or touched position on the X-ray image 80 displayed on the screen 51, and updates the connecting line 82 so that it passes through the newly placed mark. The control unit 21 then adjusts the shape of the movement line 74 so that it matches the updated shape of the connecting line 82.

[0086] After S623, when step S7 is executed, for example, the control unit 21 updates the screen 51 displayed on the display 50 via the output unit 25 as shown in FIG.

[0087] As described above, in the first to third examples, when the control unit 21 receives an operation by the user to manually change the shape of the movement line 74 corresponding to the virtual axis before generating the three-dimensional image 70 in S7, the control unit 21 adjusts the shape of the movement line 74 to the shape changed by that operation.

[0088] A fourth example of the detailed procedure of step S6 will be described with reference to FIG.

[0089] Steps S631 and S632 are similar to steps S601 and S602 shown in Fig. 8, and therefore will not be described here. Step S632 may be replaced with steps S612 and S613 shown in Fig. 11. In such a modification, the operation of manually specifying the second position L2 is not required.

[0090] In S633, the control unit 21 automatically detects an image of the shaft 27 of the catheter 26 from the X-ray image 80. A known image recognition technique can be used to detect the image of the shaft 27 of the catheter 26. Machine learning such as deep learning may also be used.

[0091] In S634, the control unit 21 sets the shape of the movement line 74 based on the image detected in S633. Specifically, the control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 with respect to the X-ray image 80 so that the positions of both ends of the movement line 74 stored in the storage unit 22 coincide with the first position L1 and second position L2 associated with them and stored in the storage unit 22. Then, the control unit 21 adjusts the shape of the movement line 74 so that it coincides with the shape of the image detected in S633.

[0092] A fifth example of the detailed procedure of step S6 will be described with reference to FIG.

[0093] In this example, step S632 in the fourth example is replaced with steps S612 and S613 shown in Fig. 11. Step S641 is the same as step S601 shown in Fig. 8, so a description thereof will be omitted. Steps S642 and S643 are the same as steps S612 and S613 shown in Fig. 11, respectively, so a description thereof will be omitted. Steps S644 and S645 are the same as steps S633 and S634 shown in Fig. 14, respectively, so a description thereof will be omitted.

[0094] As described above, in the fourth and fifth examples, the control unit 21 sets, as a virtual axis in the three-dimensional space, a movement line 74 that extends in a shape corresponding to the shape of the shaft 27 of the catheter 26 in the section from the first point P1 to the second point P2 from a position in the three-dimensional space corresponding to the first sensor position V1 identified in S5 for the two-dimensional image 60 obtained by the sensor 28 when the sensor 28 is located at the first point P1. When generating the three-dimensional image 70 in S7, the control unit 21 sets the shape of the three-dimensional image 70 to a shape in which the sensor position identified in S5 is located on the virtual axis.

[0095] In the first to fifth examples, instead of accepting an operation to manually specify the first position L1 when the sensor 28 is located at the first point P1, the control unit 21 may automatically detect the first position L1 when the sensor 28 is located at the first point P1 and associate the first position L1 with one end of the movement line 74. In such a modified example, the control unit 21 automatically detects the position at which the sensor 28 is located on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at the first point P1. Known image recognition technology can be used as a method for detecting the position at which the sensor 28 is located. Machine learning, such as deep learning, may also be used. The control unit 21 receives a notification from the MDU driving the catheter 26 via the communication unit 23 regarding the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is located at the first point P1. The control unit 21 specifies the first sensor position V1 specified in S5 in the cross section existing at the position notified by the MDU in the three-dimensional image 70 as the position of the first sensor position V1 in the three-dimensional image 70. The control unit 21 stores the automatically detected position as the first position L1 and the position of the first sensor position V1 in the three-dimensional image 70 as the position of one end of the movement line 74 in the memory unit 22 in association with each other.

[0096] In the first, third, and fifth examples, instead of accepting an operation to manually specify the second position L2 when the sensor 28 is at the second point P2, the control unit 21 may automatically detect the second position L2 when the sensor 28 is at the second point P2 and associate the second position L2 with the other end of the movement line 74. In such a modified example, the control unit 21 automatically detects the position at which the sensor 28 is imaged on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is at the second point P2. Known image recognition technology can be used as a method for detecting the position at which the sensor 28 is imaged. Machine learning, such as deep learning, may also be used. The control unit 21 receives notification from the MDU driving the catheter 26, via the communication unit 23, of the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is at the second point P2. The control unit 21 specifies the second sensor position V2 specified in S5 in the cross section existing at the position notified by the MDU in the three-dimensional image 70 as the position of the second sensor position V2 in the three-dimensional image 70. The control unit 21 stores the automatically detected position as the second position L2 and the position of the second sensor position V2 in the three-dimensional image 70 as the position of the other end of the movement line 74 in the memory unit 22 in association with each other.

[0097] A modified example of the operation shown in Fig. 2 will be described with reference to Fig. 19. With respect to steps S1A to S9A shown in Fig. 19, the description of the same processes as steps S1 to S9 shown in Fig. 2 will be omitted or simplified as appropriate.

[0098] In this modification, X-ray images are continuously taken during the pull-back operation, which is the operation of pulling the sensor 28 toward the user, and the position of the sensor 28 on the corresponding X-ray image is continuously captured each time a two-dimensional image is generated, i.e., for each frame. As a result, a curved three-dimensional image 70 can be constructed in real time.

[0099] In S1A, the control unit 21 acquires an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at the i-th point Pi in the biological lumen. Specifically, the control unit 21 receives, via the communication unit 23, the X-ray image captured when the sensor 28 is located at the i-th point Pi from the X-ray imaging device 30. This X-ray image need only include an image of the sensor 28, and may not include an image of the shaft 27 of the catheter 26. Note that i is assumed to be "1" when steps S1A to S7A are first executed, "2" when steps S1A to S7A are last executed, and "3" or greater when steps S1A to S7A are executed from the second time onwards (excluding the last time), and is incremented by one each time. For example, if steps S1A to S7A are executed a total of four times, i will have the values ​​"1," "3," "4," and "2," respectively. When i is "1", steps S3A, S4A, S6A, and S7A may be skipped. After S7A, steps S1A to S7A are repeatedly executed until an end operation is performed, such as pressing the end button or ending the pullback. When an end operation is performed, steps S8A and S9A are executed.

[0100] In S2A, the controller 21 acquires a two-dimensional image obtained by the sensor 28 when the sensor 28 is located at the i-th point Pi. Specifically, the controller 21 receives, via the communication unit 23, reflected wave information from the sensor 28 regarding reflected waves received by the sensor 28 when the sensor 28 is located at the first point P1. The controller 21 generates a two-dimensional cross-sectional image corresponding to the first point P1 by imaging the cross-sectional structure of the biological lumen based on the received reflected wave information. Alternatively, the controller 21 may receive, via the communication unit 23, from a separately provided IVUS device, a two-dimensional cross-sectional image generated based on the reflected wave information corresponding to the first point P1.

[0101] The step of S1A may be performed before S2A, may be performed simultaneously with S2A, or may be performed after S2A.

[0102] In S3A, the control unit 21 visualizes the three-dimensional structure of the biological lumen based on the two-dimensional image acquired in S2A and, if any, one or more two-dimensional images acquired earlier, to generate a three-dimensional image 70. Specifically, the control unit 21 generates the three-dimensional image 70 by stacking the cross-sectional image generated or received in S2A and, if any, one or more cross-sectional images generated or received earlier.

[0103] In S4A, the control unit 21 causes the display 50 to display the three-dimensional image 70 generated in S3A superimposed on the X-ray image acquired in S1A.

[0104] In S5A, the control unit 21 identifies the position of the sensor 28 as the sensor position for the two-dimensional image acquired in S2A. Specifically, the control unit 21 identifies the center position of the two-dimensional image acquired by the sensor 28 when the sensor 28 is located at the ith point Pi as the ith sensor position Vi.

[0105] In S6A, the control unit 21 sets at least a portion of the movement line 74 in the three-dimensional image 70 generated in S3A. The movement line 74 is a line extending from the position of the first sensor position V1 in the three-dimensional image 70 in a shape corresponding to the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen. The shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen is identified using the multiple X-ray images acquired in S1A until i becomes "2." If i is not "2," the shape of the axis 27 of the catheter 26 from the first point P1 to partway through the section is identified using the X-ray images acquired in S1A up to that point.

[0106] Specifically, the control unit 21 automatically detects an image of the sensor 28 from the X-ray image acquired in S1A. That is, the control unit 21 automatically detects the position where the sensor 28 is imaged on the X-ray image acquired in S1A. Known image recognition technology can be used to detect the position where the sensor 28 is imaged. Machine learning, such as deep learning, may also be used. The control unit 21 receives notification from the MDU driving the catheter 26 via the communication unit 23 regarding the position of the sensor 28 in the longitudinal axis direction of the catheter 26 when the sensor 28 is located at the ith point Pi. The control unit 21 identifies the ith sensor position Vi identified in S5 within the cross section located at the position notified by the MDU in the 3D image 70 as the position of the ith sensor position Vi in the 3D image 70. The control unit 21 stores the automatically detected position as the ith position Li and the position of the ith sensor position Vi in the 3D image 70 as the ith set position of the movement line 74 in the memory unit 22 in association with each other. The first set position of the moving line 74 corresponds to the position of one end of the moving line 74. The second set position of the moving line 74 corresponds to the position of the other end of the moving line 74.

[0107] In S7A, the control unit 21 adjusts the shape of the 3D image 70 generated in S3A to match the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen, which was identified using the multiple X-ray images acquired in S1A until i became "2." If i is not "2," the control unit 21 adjusts the shape of the 3D image 70 generated in S3A to match the shape of the axis 27 of the catheter 26 from the first point P1 to partway through the section, which was identified using the X-ray images acquired in S1A up to that point. Specifically, the control unit 21 adjusts the shape of the 3D image 70 generated in S3A by locating the sensor position identified in S5A up to that point on the portion of the movement line 74 set in S6A up to that point. More specifically, the control unit 21 adjusts the relative position and size at which the three-dimensional image 70 is displayed on the display 50, as well as the shape of the three-dimensional image 70, with respect to the X-ray image 80, so that each set position of the moving line 74 stored in the memory unit 22 matches the position on the X-ray image 80 displayed on the screen 51 that has already been associated with it and stored in the memory unit 22.

[0108] Steps S5A to S7A may be performed before S4A.

[0109] As described above, in this modification, the controller 21 associates a first position L1 corresponding to the first point P1 on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at a first point P1 within the body lumen with one end of the movement line 74. The controller 21 associates a second position L2 corresponding to the second point P2 on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at a second point P2 within the body lumen with the other end of the movement line 74. The controller 21 automatically detects an image of the sensor 28 from each of the multiple X-ray images captured by the X-ray imaging device 30 while the sensor 28 moves through the section from the first point P1 to the second point P2 within the body lumen. The controller 21 sets the shape of the movement line 74 based on the detected images.

[0110] Therefore, according to this embodiment, it is possible to construct a curved three-dimensional image 70 in real time. Although a corresponding X-ray image is required for each cross-sectional image, it is sufficient that at least the sensor 28 is visible in each X-ray image, and the axis 27 of the catheter 26 does not have to be visible in each X-ray image.

[0111] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks in the block diagram may be integrated, or one block may be divided. Instead of executing two or more steps in the flowchart chronologically as described, they may be executed in parallel or in a different order depending on the processing capacity of the device executing each step or as needed. Other modifications are possible without departing from the spirit of the present disclosure. For example, if the control unit 21 is to display only a three-dimensional image 70 that matches the actual blood vessel path on the display 50, the control unit 21 may not display the X-ray image 80 on the display 50 in S4.

[0112] REFERENCE SIGNS LIST 10 Image processing system 20 Image processing device 21 Control unit 22 Memory unit 23 Communication unit 24 Input unit 25 Output unit 26 Catheter 27 Shaft 28 Sensor 29 Guide wire 30 X-ray imaging device 40 Input device 50 Display 51 Screen 52 First area 53 Second area 54 Slider 60 Two-dimensional image 61 Human body model image 70 Three-dimensional image 71 Human body model image 72 Cross-sectional image 73 Camera icon 74 Movement line 75, 85 Blood vessels 76, 86 Branching section 80 X-ray image 82 Connecting line

Claims

1. An image processing device having a control unit that sets a virtual axis that exists in three-dimensional space based on the position of the axis of a catheter in each of multiple two-dimensional images contained in a plurality of two-dimensional images obtained by moving a sensor attached to a catheter inserted into a biological lumen along the axis of the catheter, visualizes the three-dimensional structure of the biological lumen based on the multiple two-dimensional images, and generates a three-dimensional image in the three-dimensional space having multiple cross sections that are spaced apart along the virtual axis and perpendicular to the virtual axis, sets a rotation angle for rotating the three-dimensional image, and adjusts the shape of the three-dimensional image by rotating each cross section contained in the multiple cross sections by the rotation angle around the position of the virtual axis in that cross section.

2. The image processing device according to claim 1, wherein the control unit, upon receiving an operation by a user to manually specify an angle, sets the rotation angle to the angle specified by the operation.

3. The image processing device according to claim 1, wherein the control unit automatically sets the rotation angle based on the three-dimensional image and an X-ray image taken by an X-ray imaging device that sees through the biological lumen.

4. The image processing device according to claim 1, wherein, when the control unit receives an operation by a user to manually change the shape of the virtual axis before generating the three-dimensional image, the control unit adjusts the shape of the virtual axis to the shape changed by the operation.

5. The image processing device described in claim 1, wherein the multiple two-dimensional images are obtained by moving the sensor along the axis of the catheter through a section from a first point to a second point within the biological lumen, and the control unit identifies the position of the axis of the catheter in each of the multiple two-dimensional images as the sensor position, sets a movement line in the three-dimensional space as the virtual axis, extending from a position in the three-dimensional space corresponding to the sensor position identified for the two-dimensional image obtained by the sensor when the sensor is located at the first point, in a shape corresponding to the shape of the axis of the catheter in the section, and when generating the three-dimensional image, sets the shape of the three-dimensional image to a shape in which the sensor position identified for each of the multiple two-dimensional images is located on the virtual axis.

6. The image processing device according to claim 5, wherein each cross section included in the plurality of cross sections is a cross section corresponding to a sensor position identified for each two-dimensional image included in the plurality of two-dimensional images.

7. The image processing device according to claim 1, wherein the control unit, when adjusting the shape of the three-dimensional image, displays the three-dimensional image on a display in a state where the three-dimensional image is superimposed on an X-ray image taken by an X-ray imaging device that sees through the body lumen.

8. The image processing device according to claim 7, wherein the control unit adjusts the shape of the three-dimensional image and then displays the three-dimensional image on the display superimposed on the X-ray image or another X-ray image taken by the X-ray imaging device.

9. An image processing system comprising: the image processing device according to claim 7 or 8; and the display.

10. An image processing method comprising: setting a virtual axis existing in three-dimensional space based on the position of the axis of a catheter in each of a plurality of two-dimensional images contained in the two-dimensional images obtained by moving a sensor attached to a catheter inserted into a biological lumen along the axis of the catheter; imaging the three-dimensional structure of the biological lumen based on the plurality of two-dimensional images to generate a three-dimensional image having a plurality of cross sections in the three-dimensional space that are spaced apart along the virtual axis and perpendicular to the virtual axis; setting a rotation angle for rotating the three-dimensional image; and adjusting the shape of the three-dimensional image by rotating each cross section contained in the plurality of cross sections by the rotation angle around the position of the virtual axis in each cross section.

11. An image processing program that causes a computer to perform the following operations: setting a virtual axis that exists in three-dimensional space based on the position of the axis of a catheter in each of multiple two-dimensional images contained in a plurality of two-dimensional images obtained by moving a sensor attached to a catheter inserted into a biological lumen along the axis of the catheter; imaging the three-dimensional structure of the biological lumen based on the multiple two-dimensional images to generate a three-dimensional image in the three-dimensional space having multiple cross sections that are spaced apart along the virtual axis and perpendicular to the virtual axis; setting a rotation angle for rotating the three-dimensional image; and adjusting the shape of the three-dimensional image by rotating each cross section contained in the multiple cross sections by the rotation angle around the position of the virtual axis in each cross section.

Images