Image processing device, image processing system, image processing method, and image processing program

Abstract

Provided is an image processing device that generates a three-dimensional image by imaging a three-dimensional structure of a living body lumen on the basis of a plurality of two-dimensional images obtained by moving a sensor, which is provided in a catheter inserted into the living body lumen, within a section from a first point to a second point in the living body lumen along the shaft of the catheter. The image processing device comprises a control part that adjusts the shape of the three-dimensional image according to the shape of the shaft of the catheter in the section, the shape being specified by using at least one X-ray image including an image of at least a portion of the catheter captured by an X-ray imaging device that sees through the living body lumen.

Description

Image processing device, image processing system, image processing method, and image processing program

[0001] The present disclosure relates to an image processing device, an image processing system, an image processing method, and an image processing program.

[0002] Patent Document 1 discloses a method for identifying the direction of a patient's ventral side in a three-dimensional image when generating and displaying a three-dimensional image of biological tissue based on a two-dimensional ultrasound image acquired by a transducer of an ultrasound catheter.

[0003] International Publication No. 2022 / 045182

[0004] During catheterization procedures such as stent graft placement, 3D CT images taken before surgery or DSA images taken during surgery can be merged with real-time X-ray images to reduce X-ray exposure or assist in catheter manipulation. "CT" stands for computed tomography. "DSA" stands for digital subtraction angiography. DSA images are created by subtracting a mask X-ray image taken before contrast administration from a contrast image taken after contrast administration. Merging 3D CT or DSA images with X-ray images allows for simultaneous confirmation of the detailed vascular shape and real-time status. However, when a stiff guidewire or catheter is inserted into a blood vessel, the vessel stretches, potentially creating a large discrepancy between the 3D CT image and the actual organ shape. X-ray exposure during 3D CT imaging can also be an issue. Because DSA images use a large amount of contrast agent, contrast-induced nephropathy can occur.

[0005] Therefore, instead of displaying 3D CT images taken before surgery, it is conceivable to display 3D images generated during surgery by a method that does not involve X-ray exposure, such as that disclosed in Patent Document 1. However, when blood vessels are tortuous, it may not be possible to reproduce the shape of the blood vessels in a 3D image simply by stacking 2D ultrasound images.

[0006] An object of the present disclosure is to make it easier to reproduce the shape of a biological lumen in a three-dimensional image when imaging the three-dimensional structure of a biological lumen to generate a three-dimensional image.

[0007] Some aspects of the present disclosure are set forth below.

[0008] [1] An image processing device that generates a three-dimensional image by imaging the three-dimensional structure of a biological lumen based on a plurality of two-dimensional images obtained by a sensor attached to a catheter inserted into the biological lumen moving along the axis of the catheter through a section from a first point to a second point within the biological lumen, the image processing device comprising: a control unit that adjusts the shape of the three-dimensional image to match the shape of the axis of the catheter in the section, which is identified using at least one X-ray image including an image of at least a portion of the catheter, taken by an X-ray imaging device that sees through the biological lumen.

[0009] [2] The image processing device described in [1], wherein the control unit identifies the position of the sensor as a sensor position for each of the plurality of two-dimensional images included in the plurality of two-dimensional images, sets a movement line in the three-dimensional image extending in a shape corresponding to the shape of the axis of the catheter in the section from the position in the three-dimensional image of the sensor position identified for the two-dimensional image obtained by the sensor when the sensor is located at the first point, and adjusts the shape of the three-dimensional image by arranging each identified sensor position on the set movement line.

[0010] [3] The image processing device described in [2], wherein the control unit: associates a first position corresponding to the first point on the at least one X-ray image with a position at one end of the movement line; associates a second position corresponding to the second point on the at least one X-ray image with a position at the other end of the movement line; sets a connecting line connecting the first position and the second position on the at least one X-ray image; and sets a shape of the movement line in response to an operation to manually change the shape of the set connecting line.

[0011] [4] The image processing device described in [2], wherein the control unit: associates a first position corresponding to the first point on the at least one X-ray image with the position of one end of the movement line; associates a second position corresponding to the second point on the at least one X-ray image with the position of the other end of the movement line; and sets the shape of the movement line to a shape corresponding to a connecting line connecting the first position, the second position, and the at least one position in response to an operation of manually specifying at least one position different from the first position and the second position on the at least one X-ray image.

[0012] [5] The image processing device described in [2], wherein the control unit: associates a first position corresponding to the first point on the at least one X-ray image with the position of one end of the movement line; associates a second position corresponding to the second point on the at least one X-ray image with the position of the other end of the movement line; automatically detects an image of the axis of the catheter from the at least one X-ray image; and sets the shape of the movement line based on the detected image.

[0013] [6] The image processing device described in [2], wherein the control unit: associates a first position corresponding to the first point on the at least one X-ray image with the position of one end of the movement line; associates a second position corresponding to the second point on the at least one X-ray image with the position of the other end of the movement line; automatically detects an image of the sensor from each of a plurality of X-ray images taken by the X-ray imaging device while the sensor moves through the section; and sets the shape of the movement line based on the detected images.

[0014] [7] The image processing device according to any one of [2] to [6], wherein the control unit, in response to an operation of manually specifying a first position corresponding to the first point on the at least one X-ray image when the sensor is present at the first point, associates the first position with a position at one end of the movement line.

[0015] [8] The image processing device described in [7], wherein the control unit corresponds the second position to the position of the other end of the movement line in response to an operation of manually specifying a second position corresponding to the second point on the at least one X-ray image when the sensor is present at the second point.

[0016] [9] The control unit acquires ratio information indicating the ratio of the dimensions of the at least one X-ray image to the dimensions of the three-dimensional image, and based on the acquired ratio information, associates a position on the at least one X-ray image that is a distance away from the first position corresponding to the distance of the movement line with the position of the other end of the movement line as a second position on the at least one X-ray image corresponding to the second point.

[0017]

[10] The image processing device described in any one of [2] to [6], wherein the control unit automatically detects a first position corresponding to the first point on the at least one X-ray image when the sensor is present at the first point, and associates the first position with the position of one end of the movement line.

[0018]

[11] The image processing device described in

[10] , wherein the control unit automatically detects a second position corresponding to the second point on the at least one X-ray image when the sensor is located at the second point, and associates the second position with the position of the other end of the movement line.

[0019]

[12] The image processing device according to any one of [2] to

[11] , wherein the control unit causes the three-dimensional image to be displayed on a display in a manner superimposed on the at least one X-ray image.

[0020]

[13] The image processing device according to

[12] , wherein the control unit updates the three-dimensional image in accordance with changes in the movement line while the movement line is being set.

[0021]

[14] An image processing system comprising: the image processing device according to any one of [1] to

[13] ; and a display that displays the three-dimensional image and the at least one X-ray image.

[0022]

[15] An image processing method for generating a three-dimensional image by imaging the three-dimensional structure of a biological lumen based on a plurality of two-dimensional images obtained by a sensor provided on a catheter inserted into the biological lumen moving along the axis of the catheter through a section from a first point to a second point within the biological lumen, the image processing method including a control unit adjusting the shape of the three-dimensional image to match the shape of the axis of the catheter in the section, which is identified using at least one X-ray image including an image of at least a part of the catheter, taken by an X-ray imaging device that sees through the biological lumen.

[0023]

[16] An image processing program that causes a computer to generate a three-dimensional image by imaging the three-dimensional structure of a biological lumen based on a plurality of two-dimensional images obtained by a sensor provided on a catheter inserted into the biological lumen moving along the axis of the catheter through a section from a first point to a second point within the biological lumen, and that causes the computer to perform operations including adjusting the shape of the three-dimensional image to match the shape of the axis of the catheter in the section, which is identified using at least one X-ray image including an image of at least a part of the catheter, taken by an X-ray imaging device that sees through the biological lumen.

[0024] According to the present disclosure, when imaging the three-dimensional structure of a biological lumen to generate a three-dimensional image, the shape of the biological lumen can be easily reproduced in the three-dimensional image.

[0025] FIG. 1 is a block diagram illustrating a configuration of an image processing system according to an embodiment of the present disclosure. FIG. 2 is a flowchart illustrating an operation of an image processing device according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of updating, in step S4, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of a three-dimensional image generated by an image processing device according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of shape adjustment, in step S7, of a three-dimensional image generated by an image processing device according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of updating, in step S7, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 8 is a flowchart illustrating a first example of detailed procedures of step S6. FIG. 9 is a diagram illustrating an example of updating, in step S603, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of updating, in step S604, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 11 is a flowchart illustrating a second example of detailed procedures of step S6. FIG. 12 is a flowchart illustrating a third example of detailed procedures of step S6. FIG. 13 is a diagram illustrating an example of updating, in step S623, of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 14 is a flowchart illustrating a fourth example of detailed procedures of step S6. FIG. 15 is a flowchart illustrating a fifth example of detailed procedures of step S6. FIG. 16 is a flowchart illustrating a modified example of the operation of an image processing device according to an embodiment of the present disclosure.

[0026] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0027] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0028] The configuration of an image processing system 10 according to this embodiment will be described with reference to FIG.

[0029] The image processing system 10 includes an image processing device 20, a sensor 28, an X-ray imaging device 30, an input device 40, and a display 50. The image processing device 20 is connected to the sensor 28, the X-ray imaging device 30, the input device 40, and the display 50 via a cable, a network, or wirelessly.

[0030] The image processing device 20 is, for example, a general-purpose computer such as a PC, a server computer such as a cloud server, or a dedicated computer. "PC" is an abbreviation for personal computer. The image processing device 20 may be installed in a medical facility such as a hospital, or may be installed in a facility separate from the medical facility, such as a data center.

[0031] The sensor 28 is, for example, an ultrasound transducer used in IVUS (intravascular ultrasound). The sensor 28 transmits ultrasound waves while moving along the axis 27 of a catheter 26 inserted into a body lumen such as a blood vessel, as shown in FIG. 3 , and receives reflected waves of the transmitted ultrasound waves.

[0032] The X-ray imaging device 30 is, for example, a C-arm. The X-ray imaging device 30 is installed in a medical facility and performs transillumination of a living body lumen.

[0033] The input device 40 is, for example, a pointing device such as a mouse, a keyboard, or a touch screen that is integrated with the display 50. The input device 40 is installed in a medical facility and is used by an operator such as a doctor or a clinical engineer to control the display of various information, including images, on the display 50.

[0034] The display 50 is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electroluminescent. The display 50 is installed in a medical facility and displays various information, including images, to an operator to support catheter surgery such as a stent graft placement procedure.

[0035] The configuration of an image processing apparatus 20 according to this embodiment will be described with reference to FIG.

[0036] The image processing device 20 includes a control unit 21 , a storage unit 22 , a communication unit 23 , an input unit 24 , and an output unit 25 .

[0037] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. An example of the programmable circuit is an FPGA. "FPGA" is an abbreviation for field-programmable gate array. An example of the dedicated circuit is an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 controls each part of the image processing device 20 and executes processing related to the operation of the image processing device 20.

[0038] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. Flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. Magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 22 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the image processing device 20 and data obtained by the operation of the image processing device 20 .

[0039] The communication unit 23 includes at least one communication module. The communication module is, for example, a module compatible with a wired LAN communication standard such as Ethernet (registered trademark) or a wireless LAN communication standard such as IEEE 802.11. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. The communication unit 23 receives data used in the operation of the image processing device 20 and transmits data obtained by the operation of the image processing device 20. The communication unit 23 is connected to the sensor 28 and the X-ray imaging device 30. Note that the communication unit 23 does not have to be a communication module as described above, as long as it can receive image signals from the sensor 28 and the X-ray imaging device 30.

[0040] The input unit 24 includes at least one input interface. The input interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface. The input unit 24 accepts an operation to input data used in the operation of the image processing device 20. The input unit 24 is connected to an input device 40.

[0041] The output unit 25 includes at least one output interface. The output interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). The output unit 25 outputs data obtained by the operation of the image processing device 20. The output unit 25 is connected to a display 50.

[0042] The functions of the image processing device 20 are realized by executing an image processing program according to this embodiment on a processor serving as the control unit 21. That is, the functions of the image processing device 20 are realized by software. The image processing program causes a computer to execute the operations of the image processing device 20, thereby causing the computer to function as the image processing device 20. That is, the computer functions as the image processing device 20 by executing the operations of the image processing device 20 in accordance with the image processing program.

[0043] The program can be stored on a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include flash memory, magnetic recording devices, optical disks, magneto-optical recording media, and ROMs. The program can be distributed by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs that store the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program can also be distributed by storing it in the storage of a server and transferring it from the server to another computer. The program can also be provided as a program product.

[0044] A computer temporarily stores a program stored on a portable medium or transferred from a server in its main storage device. The computer then reads the program stored in the main storage device with a processor and executes processing in accordance with the read program. The computer may also read the program directly from the portable medium and execute processing in accordance with the program. The computer may also execute processing in accordance with the received program each time a program is transferred from the server to the computer. Processing may also be executed using a so-called ASP-type service that realizes functions simply by issuing execution instructions and obtaining results, without transferring the program from the server to the computer. "ASP" is an abbreviation for application service provider. Programs include information used for processing by a computer that is equivalent to a program. For example, data that is not a direct instruction to a computer but has properties that define computer processing falls under the category of "equivalent to a program."

[0045] Some or all of the functions of the image processing device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, some or all of the functions of the image processing device 20 may be realized by hardware.

[0046] The operation of the image processing device 20 according to this embodiment will be described with reference to Fig. 2. The operation described below corresponds to the image processing method according to this embodiment. That is, the image processing method according to this embodiment includes steps S1 to S7 shown in Fig. 2.

[0047] In S1, the control unit 21 acquires one or more X-ray images taken by the X-ray imaging device 30. Specifically, the control unit 21 receives one or more X-ray images from the X-ray imaging device 30 via the communication unit 23.

[0048] In S2, the controller 21 acquires multiple two-dimensional images obtained by the sensor 28 while the sensor 28 moves through a section from the first point P1 to the second point P2 within the biological lumen. Specifically, the controller 21 receives, via the communication unit 23, reflected wave information from the sensor 28 regarding reflected waves received by the sensor 28 while the sensor 28 moves through a section from the first point P1 to the second point P2 within the biological lumen. The controller 21 generates multiple two-dimensional cross-sectional images by imaging the cross-sectional structure of the biological lumen based on the received reflected wave information. Alternatively, the controller 21 may receive, via the communication unit 23, multiple two-dimensional cross-sectional images generated by a separately provided IVUS device based on the reflected wave information from the IVUS device.

[0049] Step S1 may be executed before, during, or after S2. By the time both steps S1 and S2 are completed at the latest, the control unit 21 causes the display 50 to display a screen 51 such as that shown in FIG. 3 via the output unit 25.

[0050] The screen 51 includes a first area 52 and a second area 53 adjacent to the first area 52. In the example shown in FIG. 3 , the second area 53 is disposed on the right side of the first area 52, but the second area 53 may also be disposed on the left side of the first area 52.

[0051] The first area 52 displays the most recent two-dimensional image 60 from the multiple two-dimensional images acquired in S2 and a human body model image 61. The human body model image 61 is displayed so that its orientation is aligned with that of the two-dimensional image 60. In the example shown in FIG. 3 , the catheter 26 is inserted from the groin of the patient, with the upper side of the two-dimensional image 60 corresponding to the patient's abdomen and the lower side of the two-dimensional image 60 corresponding to the patient's back. Therefore, the human body model image 61 is displayed so that its bottom is visible, with its abdomen facing upward and its back facing downward. In the example shown in FIG. 3 , the human body model image 61 is positioned in the upper left corner of the two-dimensional image 60, but the human body model image 61 may be positioned anywhere near the two-dimensional image 60. The position of the human body model image 61 may be fixed or may be freely changeable.

[0052] A slider 54 is also displayed in the first area 52. The slider 54 indicates the position of the sensor 28 in the longitudinal direction of the catheter 26. The slider 54 is located at the end of the first area 52 that is closer to the second area 53. Therefore, in the example shown in FIG. 3 , the slider 54 is located at the right end of the first area 52.

[0053] The second area 53 displays the X-ray image 80 acquired in S1. The X-ray image 80 was acquired by the X-ray imaging device 30 when the axis 27 of the catheter 26 was present at least over the section from the first point P1 to the second point P2 within the biological lumen. The X-ray image 80 includes an image of at least a portion of the catheter 26, specifically, an image of the axis 27 of the catheter 26 and an image of the sensor 28. In the example shown in FIG. 3 , the X-ray image 80 further includes an image of the guidewire 29.

[0054] In S3, the control unit 21 visualizes the three-dimensional structure of the biological lumen based on the multiple two-dimensional images acquired in S2, and generates a three-dimensional image 70 as shown in Fig. 5. Specifically, the control unit 21 generates the three-dimensional image 70 by stacking the multiple cross-sectional images generated or received in S2.

[0055] In S4, the control unit 21 causes the three-dimensional image 70 generated in S3 to be superimposed on the X-ray image 80 acquired in S1 and displayed on the display 50. For example, the control unit 21 updates the screen 51 displayed on the display 50 via the output unit 25 as shown in FIG.

[0056] In the second area 53, the 3D image 70 generated in S3 and the human body model image 71 are displayed superimposed on the X-ray image 80 acquired in S1. In the example shown in FIG. 4, the angle of the 3D image 70 corresponds to the angle when the surgical subject is viewed from the front, the upper side of the 3D image 70 corresponds to the side where the surgical subject's head is located, and the lower side of the 3D image 70 corresponds to the side where the surgical subject's feet are located. Therefore, the human body model image 71 is displayed so that the front is visible, with the head facing up and the feet facing down. In the example shown in FIG. 4, the human body model image 71 is positioned in the upper left corner of the 3D image 70, but the human body model image 71 may be positioned anywhere near the 3D image 70. The position of the human body model image 71 may be fixed or may be arbitrarily changeable. The 3D image 70 may be partially cropped along the longitudinal axis of the catheter 26 so that the internal structure of the biological lumen can be observed, and the cropped portion may be hidden.

[0057] A cross-sectional image 72 is also displayed in the second area 53. The cross-sectional image 72 is a reduced version of the two-dimensional image 60 displayed in the first area 52. A camera icon 73 indicating the position of the viewpoint is placed on the edge of the cross-sectional image 72. This makes it easy to see from which direction the three-dimensional image 70 displayed in the second area 53 is viewed in the cross-sectional image 72.

[0058] The image display orientation in which the three-dimensional image 70 is displayed on the display 50 is adjusted manually or automatically so that it is viewed from a viewpoint corresponding to the imaging angle at which the X-ray image 80 was captured by the X-ray imaging device 30. The relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 have already been adjusted in the example shown in Fig. 4, but in reality they do not need to be adjusted yet. Fig. 4 shows a first position L1 corresponding to the first point P1 and a second position L2 corresponding to the second point P2 on the X-ray image 80, but the first position L1 and the second position L2 may be manually specified or automatically detected by S6.

[0059] In S5, the control unit 21 identifies the position of the sensor 28 as the sensor position for each of the multiple two-dimensional images acquired in S2. Specifically, the control unit 21 identifies the center position of the two-dimensional image 60 acquired by the sensor 28 when the sensor 28 is located at a first point P1 as a first sensor position V1. The control unit 21 identifies the center position of the two-dimensional image 60 acquired by the sensor 28 when the sensor 28 is located at a second point P2 as a second sensor position V2. The control unit 21 identifies the center positions of the two-dimensional image 60 acquired by the sensor 28 when the sensor 28 is located at a third point P3 and a fourth point P4, which are different from the first point P1 and the second point P2, as a third sensor position V3 and a fourth sensor position V4, respectively. The control unit 21 may further identify the center positions of the two-dimensional image 60 acquired by the sensor 28 when the sensor 28 is located at a point different from the first point P1, the second point P2, the third point P3, and the fourth point P4 as the sensor position. Alternatively, the control unit 21 does not need to identify the fourth sensor position V4.

[0060] In S6, the control unit 21 sets a movement line 74 as shown in FIG. 6 in the three-dimensional image 70 generated in S3. The movement line 74 is a line extending from the position of the first sensor position V1 identified in S5 in the three-dimensional image 70 in a shape corresponding to the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen. The shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen is identified using at least one X-ray image acquired in S1. Specifically, the shape of the axis 27 of the catheter 26 from the first position L1 to the second position L2 on the X-ray image 80 acquired in S1 is identified as the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen.

[0061] In S7, the control unit 21 adjusts the shape of the three-dimensional image 70 generated in S3 to match the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen, which is identified using at least one X-ray image. Specifically, as shown in Fig. 6, the control unit 21 adjusts the shape of the three-dimensional image 70 generated in S3 by arranging each sensor position identified in S5 on the movement line 74 set in S6. For example, as shown in Fig. 7, the control unit 21 updates the screen 51 displayed on the display 50 via the output unit 25.

[0062] Step S7 may be executed after S6, but is preferably executed simultaneously with S6. That is, it is preferable that the control unit 21 updates the three-dimensional image 70 in accordance with changes in the movement line 74 while the movement line 74 is being set.

[0063] To reduce the amount of calculation required to adjust the shape of the three-dimensional image 70 in S7, the control unit 21 preferably converts the voxel data of the three-dimensional image 70 into triangular or quadrangular mesh data when generating the three-dimensional image 70 in S3. Any method, such as the Marching Cubes method, Surface Nets, or Delaunay triangulation, can be used to convert the voxel data into mesh data. Any method, such as RMF or Frenet-Serret, can be used to deform the mesh data when adjusting the shape of the three-dimensional image 70. "RMF" is an abbreviation for rotation minimizing frames.

[0064] As a modification of this embodiment, the control unit 21 may cause the three-dimensional image 70 and the X-ray image 80 to be separately displayed on the display 50 in S4. As another modification, the control unit 21 may further acquire another X-ray image taken by the X-ray imaging device 30 at an angle different from that of the X-ray image 80 in S1, and further adjust the shape of the three-dimensional image 70 in the depth direction using the another X-ray image in S4 to S7.

[0065] As described above, in this embodiment, the control unit 21 generates a three-dimensional image 70 by imaging the three-dimensional structure of the biological lumen based on a plurality of two-dimensional images 60 obtained by moving the sensor 28 provided on the catheter 26 inserted into the biological lumen along the axis 27 of the catheter 26 through a section from the first point P1 to the second point P2 within the biological lumen. The control unit 21 adjusts the shape of the three-dimensional image 70 to match the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 within the biological lumen, which is identified using at least one X-ray image including an image of the sensor 28 taken by the X-ray imaging device 30 that transmits the biological lumen.

[0066] Therefore, according to this embodiment, when the three-dimensional structure of a biological lumen is visualized to generate a three-dimensional image 70, the shape of the biological lumen can be easily reproduced in the three-dimensional image 70. For example, even if the blood vessel is tortuous, the shape of the blood vessel can be easily reproduced in the three-dimensional image 70.

[0067] A first example of the detailed procedure of step S6 will be described with reference to FIG.

[0068] In S601, the control unit 21 associates the first position L1 with one end of the movement line 74 in response to an operation of manually specifying the first position L1 when the sensor 28 is located at the first point P1. Specifically, the control unit 21 accepts, via the input unit 24, an operation of clicking or touching, using the input device 40, a position at which the sensor 28 appears on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at the first point P1. The control unit 21 receives, via the communication unit 23, notification from the MDU driving the catheter 26 regarding the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is located at the first point P1. "MDU" is an abbreviation for motor drive unit. The control unit 21 identifies the first sensor position V1 identified in S5 within the cross section located at the position notified by the MDU in the 3D image 70 as the position of the first sensor position V1 in the 3D image 70. The control unit 21 stores the clicked or touched position as the first position L1 and the position of the first sensor position V1 in the three-dimensional image 70 as the position of one end of the movement line 74 in the memory unit 22 in association with each other.

[0069] In S602, the control unit 21 associates the second position L2 with the other end of the movement line 74 in response to an operation of manually specifying the second position L2 when the sensor 28 is located at the second point P2. Specifically, the control unit 21 accepts, via the input unit 24, an operation of clicking or touching, using the input device 40, the position at which the sensor 28 appears on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at the second point P2. The control unit 21 receives, via the communication unit 23, a notification from the MDU driving the catheter 26 regarding the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is located at the second point P2. The control unit 21 identifies the second sensor position V2 identified in S5 within the cross section located at the position notified by the MDU in the 3D image 70 as the position of the second sensor position V2 in the 3D image 70. The control unit 21 stores the clicked or touched position as the second position L2 and the position of the second sensor position V2 in the three-dimensional image 70 as the other end position of the movement line 74 in the memory unit 22 in association with each other.

[0070] In S603, the control unit 21 sets a connecting line 82 on the X-ray image 80 as shown in FIG. 9 . The connecting line 82 is a line connecting the first position L1 and the second position L2. For example, as shown in FIG. 9 , the control unit 21 places marks such as circles at the first position L1 and the second position L2 on the X-ray image 80 displayed on the screen 51, and draws a straight line connecting the placed marks as the connecting line 82. In the example shown in FIG. 9 , the X-ray image 80 was taken when the sensor 28 was located at the first point P1. However, the X-ray image 80 may be taken at any time as long as the axis 27 of the catheter 26 was located at least over the section from the first point P1 to the second point P2 within the biological lumen.

[0071] The control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 so that the position and length of the movement line 74 match the position and length of the connecting line 82. Specifically, the control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 so that the positions of both ends of the movement line 74 stored in the storage unit 22 match the first position L1 and second position L2 stored in the storage unit 22 in association with them.

[0072] In S604, the control unit 21 sets the shape of the movement line 74 in response to an operation to manually change the shape of the connecting line 82 set in S603. Specifically, the control unit 21 accepts an operation to bend the connecting line 82 by dragging any position on the connecting line 82 other than the first position L1 and the second position L2 with the input device 40 on the X-ray image 80 displayed on the screen 51 via the input unit 24. Every time the connecting line 82 is bent, the control unit 21 adjusts the shape of the movement line 74 so that it matches the bent shape of the connecting line 82.

[0073] After S604, when step S7 is executed, for example, the control unit 21 updates the screen 51 displayed on the display 50 via the output unit 25 as shown in FIG.

[0074] A second example of the detailed procedure of step S6 will be described with reference to FIG.

[0075] Step S611 is the same as step S601 shown in FIG. 8, and therefore a description thereof will be omitted.

[0076] In S612, the control unit 21 acquires ratio information indicating the ratio of the dimensions of the X-ray image 80 to the dimensions of the three-dimensional image 70. Specifically, when the actual scale of the X-ray image 80 is known, the control unit 21 calculates the ratio of the distance on the X-ray image 80 to the movement distance of the sensor 28 as the ratio information.

[0077] In S613, the control unit 21 associates a position on the X-ray image 80 that is a distance away from the first position L1 corresponding to the distance of the movement line 74 as a second position L2, with the other end of the movement line 74, based on the ratio information acquired in S612. Specifically, the control unit 21 receives notification from the MDU driving the catheter 26 via the communication unit 23 regarding the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is located at the second point P2. The control unit 21 identifies the second sensor position V2 identified in S5 within the cross section of the 3D image 70 that is located at the position notified by the MDU as the position of the second sensor position V2 in the 3D image 70. Based on the information notified by the MDU, the control unit 21 calculates the movement distance of the sensor 28 when it moves from the first point P1 to the second point P2. The control unit 21 converts the calculated movement distance into a distance on the X-ray image 80 according to the ratio calculated in S612. The control unit 21 stores in the storage unit 22 a position that is the converted distance away from the first position L1 specified in S611 as the second position L2, and a position in the three-dimensional image 70 of the second sensor position V2 as the other end position of the movement line 74, in association with each other. The direction in which the second position L2 exists relative to the first position L1 may be specified manually, or may be automatically determined by matching it with the position of the axis 27 of the catheter 26 on the X-ray image 80.

[0078] Steps S614 and S615 are similar to steps S603 and S604 shown in FIG. 8, respectively, and therefore will not be described here.

[0079] A third example of the detailed procedure of step S6 will be described with reference to FIG.

[0080] Steps S621 and S622 are similar to steps S601 and S602 shown in Fig. 8, and therefore will not be described here. Step S622 may be replaced with steps S612 and S613 shown in Fig. 11. In such a modification, the operation of manually specifying the second position L2 is not required.

[0081] In S623, in response to an operation of manually specifying at least one position on the X-ray image 80 that is different from the first position L1 and the second position L2, the control unit 21 sets the shape of the movement line 74 to a shape that corresponds to the first position L1, the second position L2, and the connecting line 82 that connects the at least one position. Specifically, the control unit 21 accepts, via the input unit 24, an operation of clicking or touching, with the input device 40, an arbitrary position on the shaft 27 of the catheter 26 other than the first position L1 and the second position L2 as a third position L3 on the X-ray image 80 displayed on the screen 51. The control unit 21 further accepts, via the input unit 24, an operation of clicking or touching, with the input device 40, an arbitrary position on the shaft 27 of the catheter 26 other than the first position L1, the second position L2, and the third position L3 as a fourth position L4 on the X-ray image 80 displayed on the screen 51. 13 , the control unit 21 places marks such as circles at a first position L1, a second position L2, a third position L3, and a fourth position L4 on the X-ray image 80 displayed on the screen 51, and draws a straight line connecting all of the placed marks as a connecting line 82. The control unit 21 may further accept, via the input unit 24, an operation of clicking or touching any position on the shaft 27 of the catheter 26 other than the first position L1, the second position L2, the third position L3, and the fourth position L4 with the input device 40 on the X-ray image 80 displayed on the screen 51. Alternatively, the control unit 21 may not accept an operation of clicking or touching the fourth position L4.

[0082] The control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 so that the position and length of the movement line 74 match the position and length of the connecting line 82. Specifically, the control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 relative to the X-ray image 80 so that the positions of both ends of the movement line 74 stored in the storage unit 22 match the first position L1 and second position L2 stored in the storage unit 22 in association with them.

[0083] The control unit 21 automatically changes the shape of the connecting line 82 each time a position other than the first position L1 and the second position L2 is specified, and changes the shape of the movement line 74 accordingly. Specifically, each time a position other than the first position L1 and the second position L2 is clicked or touched, the control unit 21 places a new mark such as a circle at the clicked or touched position on the X-ray image 80 displayed on the screen 51, and updates the connecting line 82 so that it passes through the newly placed mark. The control unit 21 then adjusts the shape of the movement line 74 so that it matches the updated shape of the connecting line 82.

[0084] After S623, when step S7 is executed, for example, the control unit 21 updates the screen 51 displayed on the display 50 via the output unit 25 as shown in FIG.

[0085] A fourth example of the detailed procedure of step S6 will be described with reference to FIG.

[0086] Steps S631 and S632 are similar to steps S601 and S602 shown in Fig. 8, and therefore will not be described here. Step S632 may be replaced with steps S612 and S613 shown in Fig. 11. In such a modification, the operation of manually specifying the second position L2 is not required.

[0087] In S633, the control unit 21 automatically detects an image of the shaft 27 of the catheter 26 from the X-ray image 80. A known image recognition technique can be used to detect the image of the shaft 27 of the catheter 26. Machine learning such as deep learning may also be used.

[0088] In S634, the control unit 21 sets the shape of the movement line 74 based on the image detected in S633. Specifically, the control unit 21 adjusts the relative position and size of the three-dimensional image 70 displayed on the display 50 with respect to the X-ray image 80 so that the positions of both ends of the movement line 74 stored in the storage unit 22 coincide with the first position L1 and second position L2 associated with them and stored in the storage unit 22. Then, the control unit 21 adjusts the shape of the movement line 74 so that it coincides with the shape of the image detected in S633.

[0089] A fifth example of the detailed procedure of step S6 will be described with reference to FIG.

[0090] In this example, step S632 in the fourth example is replaced with steps S612 and S613 shown in Fig. 11. Step S641 is the same as step S601 shown in Fig. 8, so a description thereof will be omitted. Steps S642 and S643 are the same as steps S612 and S613 shown in Fig. 11, respectively, so a description thereof will be omitted. Steps S644 and S645 are the same as steps S633 and S634 shown in Fig. 14, respectively, so a description thereof will be omitted.

[0091] In the first to fifth examples, instead of accepting an operation to manually specify the first position L1 when the sensor 28 is located at the first point P1, the control unit 21 may automatically detect the first position L1 when the sensor 28 is located at the first point P1 and associate the first position L1 with one end of the movement line 74. In such a modified example, the control unit 21 automatically detects the position at which the sensor 28 is located on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at the first point P1. Known image recognition technology can be used as a method for detecting the position at which the sensor 28 is located. Machine learning, such as deep learning, may also be used. The control unit 21 receives a notification from the MDU driving the catheter 26 via the communication unit 23 regarding the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is located at the first point P1. The control unit 21 specifies the first sensor position V1 specified in S5 in the cross section existing at the position notified by the MDU in the three-dimensional image 70 as the position of the first sensor position V1 in the three-dimensional image 70. The control unit 21 stores the automatically detected position as the first position L1 and the position of the first sensor position V1 in the three-dimensional image 70 as the position of one end of the movement line 74 in the memory unit 22 in association with each other.

[0092] In the first, third, and fifth examples, instead of accepting an operation to manually specify the second position L2 when the sensor 28 is at the second point P2, the control unit 21 may automatically detect the second position L2 when the sensor 28 is at the second point P2 and associate the second position L2 with the other end of the movement line 74. In such a modified example, the control unit 21 automatically detects the position at which the sensor 28 is imaged on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is at the second point P2. Known image recognition technology can be used as a method for detecting the position at which the sensor 28 is imaged. Machine learning, such as deep learning, may also be used. The control unit 21 receives notification from the MDU driving the catheter 26, via the communication unit 23, of the position of the sensor 28 in the longitudinal direction of the catheter 26 when the sensor 28 is at the second point P2. The control unit 21 specifies the second sensor position V2 specified in S5 in the cross section existing at the position notified by the MDU in the three-dimensional image 70 as the position of the second sensor position V2 in the three-dimensional image 70. The control unit 21 stores the automatically detected position as the second position L2 and the position of the second sensor position V2 in the three-dimensional image 70 as the position of the other end of the movement line 74 in the memory unit 22 in association with each other.

[0093] A modified example of the operation shown in Fig. 2 will be described with reference to Fig. 16. With regard to steps S1A to S7A shown in Fig. 16, the description of the same processes as steps S1 to S7 shown in Fig. 2 will be omitted or simplified as appropriate.

[0094] In this modification, X-ray images are continuously taken during the pull-back operation, which is the operation of pulling the sensor 28 toward the user, and the position of the sensor 28 on the corresponding X-ray image is continuously captured each time a two-dimensional image is generated, i.e., for each frame. As a result, a curved three-dimensional image 70 can be constructed in real time.

[0095] In the following description, i is set to "1" when steps S1A to S7A are executed for the first time, "2" when steps S1A to S7A are executed for the last time, and "3" or greater when steps S1A to S7A are executed for the second or subsequent time (excluding the last time), and is incremented by one each time. For example, if steps S1A to S7A are executed a total of four times, i takes the values ​​"1," "3," "4," and "2" in that order. When i is "1," steps S3A, S4A, S6A, and S7A may be skipped. After S7A, steps S1A and onward are repeatedly executed until an end operation is performed, such as pressing the end button or ending the pullback.

[0096] In S1A, the control unit 21 acquires an X-ray image taken by the X-ray imaging device 30 when the sensor 28 is located at the i-th point Pi in the living body lumen. Specifically, the control unit 21 receives the X-ray image taken when the sensor 28 is located at the i-th point Pi from the X-ray imaging device 30 via the communication unit 23. This X-ray image needs to include at least an image of the sensor 28, and does not need to include an image of the shaft 27 of the catheter 26.

[0097] In S2A, the controller 21 acquires a two-dimensional image obtained by the sensor 28 when the sensor 28 is located at the i-th point Pi. Specifically, the controller 21 receives, via the communication unit 23, reflected wave information from the sensor 28 regarding reflected waves received by the sensor 28 when the sensor 28 is located at the first point P1. The controller 21 generates a two-dimensional cross-sectional image corresponding to the first point P1 by imaging the cross-sectional structure of the biological lumen based on the received reflected wave information. Alternatively, the controller 21 may receive, via the communication unit 23, from a separately provided IVUS device, a two-dimensional cross-sectional image generated based on the reflected wave information corresponding to the first point P1.

[0098] The step of S1A may be performed before S2A, may be performed simultaneously with S2A, or may be performed after S2A.

[0099] In S3A, the control unit 21 visualizes the three-dimensional structure of the biological lumen based on the two-dimensional image acquired in S2A and, if any, one or more two-dimensional images acquired earlier, to generate a three-dimensional image 70. Specifically, the control unit 21 generates the three-dimensional image 70 by stacking the cross-sectional image generated or received in S2A and, if any, one or more cross-sectional images generated or received earlier.

[0100] In S4A, the control unit 21 causes the display 50 to display the three-dimensional image 70 generated in S3A superimposed on the X-ray image acquired in S1A.

[0101] In S5A, the control unit 21 identifies the position of the sensor 28 as the sensor position for the two-dimensional image acquired in S2A. Specifically, the control unit 21 identifies the center position of the two-dimensional image acquired by the sensor 28 when the sensor 28 is located at the ith point Pi as the ith sensor position Vi.

[0102] In S6A, the control unit 21 sets at least a portion of the movement line 74 in the three-dimensional image 70 generated in S3A. The movement line 74 is a line extending from the position of the first sensor position V1 in the three-dimensional image 70 in a shape corresponding to the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen. The shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen is identified using the multiple X-ray images acquired in S1A until i becomes "2." If i is not "2," the shape of the axis 27 of the catheter 26 from the first point P1 to partway through the section is identified using the X-ray images acquired in S1A up to that point.

[0103] Specifically, the control unit 21 automatically detects an image of the sensor 28 from the X-ray image acquired in S1A. That is, the control unit 21 automatically detects the position where the sensor 28 is imaged on the X-ray image acquired in S1A. Known image recognition technology can be used to detect the position where the sensor 28 is imaged. Machine learning, such as deep learning, may also be used. The control unit 21 receives notification from the MDU driving the catheter 26 via the communication unit 23 regarding the position of the sensor 28 in the longitudinal axis direction of the catheter 26 when the sensor 28 is located at the ith point Pi. The control unit 21 identifies the ith sensor position Vi identified in S5 within the cross section located at the position notified by the MDU in the 3D image 70 as the position of the ith sensor position Vi in the 3D image 70. The control unit 21 stores the automatically detected position as the ith position Li and the position of the ith sensor position Vi in the 3D image 70 as the ith set position of the movement line 74 in the memory unit 22 in association with each other. The first set position of the moving line 74 corresponds to the position of one end of the moving line 74. The second set position of the moving line 74 corresponds to the position of the other end of the moving line 74.

[0104] In S7A, the control unit 21 adjusts the shape of the 3D image 70 generated in S3A to match the shape of the axis 27 of the catheter 26 in the section from the first point P1 to the second point P2 in the biological lumen, which was identified using the multiple X-ray images acquired in S1A until i became "2." If i is not "2," the control unit 21 adjusts the shape of the 3D image 70 generated in S3A to match the shape of the axis 27 of the catheter 26 from the first point P1 to partway through the section, which was identified using the X-ray images acquired in S1A up to that point. Specifically, the control unit 21 adjusts the shape of the 3D image 70 generated in S3A by locating the sensor position identified in S5A up to that point on the portion of the movement line 74 set in S6A up to that point. More specifically, the control unit 21 adjusts the relative position and size at which the three-dimensional image 70 is displayed on the display 50, as well as the shape of the three-dimensional image 70, with respect to the X-ray image 80, so that each set position of the moving line 74 stored in the memory unit 22 matches the position on the X-ray image 80 displayed on the screen 51 that has already been associated with it and stored in the memory unit 22.

[0105] Steps S5A to S7A may be performed before S4A.

[0106] As described above, in this modification, the controller 21 associates a first position L1 corresponding to the first point P1 on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at a first point P1 within the body lumen with one end of the movement line 74. The controller 21 associates a second position L2 corresponding to the second point P2 on an X-ray image captured by the X-ray imaging device 30 when the sensor 28 is located at a second point P2 within the body lumen with the other end of the movement line 74. The controller 21 automatically detects an image of the sensor 28 from each of the multiple X-ray images captured by the X-ray imaging device 30 while the sensor 28 moves through the section from the first point P1 to the second point P2 within the body lumen. The controller 21 sets the shape of the movement line 74 based on the detected images.

[0107] Therefore, according to this embodiment, it is possible to construct a curved three-dimensional image 70 in real time. Although a corresponding X-ray image is required for each cross-sectional image, it is sufficient that at least the sensor 28 is visible in each X-ray image, and the axis 27 of the catheter 26 does not have to be visible.

[0108] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks in the block diagram may be integrated, or one block may be divided. Instead of executing two or more steps in the flowchart chronologically as described, they may be executed in parallel or in a different order depending on the processing capacity of the device executing each step or as needed. Other modifications are possible without departing from the spirit of the present disclosure. For example, if the control unit 21 is to display only a three-dimensional image 70 that matches the actual blood vessel path on the display 50, the control unit 21 may not display the X-ray image 80 on the display 50 in S4.

[0109] REFERENCE SIGNS LIST 10 Image processing system 20 Image processing device 21 Control unit 22 Memory unit 23 Communication unit 24 Input unit 25 Output unit 26 Catheter 27 Shaft 28 Sensor 29 Guide wire 30 X-ray imaging device 40 Input device 50 Display 51 Screen 52 First area 53 Second area 54 Slider 60 Two-dimensional image 61 Human body model image 70 Three-dimensional image 71 Human body model image 72 Cross-sectional image 73 Camera icon 74 Movement line 80 X-ray image 82 Connecting line

Claims

1. An image processing device that generates a three-dimensional image by imaging the three-dimensional structure of a biological lumen based on multiple two-dimensional images obtained by a sensor attached to a catheter inserted into the biological lumen moving along the axis of the catheter from a first point to a second point within the biological lumen, and the image processing device is equipped with a control unit that adjusts the shape of the three-dimensional image to match the shape of the axis of the catheter in the section identified using at least one X-ray image including an image of at least a portion of the catheter taken by an X-ray imaging device that visualizes the biological lumen.

2. The image processing device described in claim 1, wherein the control unit identifies the position of the sensor as a sensor position for each of the multiple two-dimensional images included in the multiple two-dimensional images, sets a movement line in the three-dimensional image extending from a position in the three-dimensional image of the sensor position identified for the two-dimensional image obtained by the sensor when the sensor is located at the first point, in a shape corresponding to the shape of the catheter axis in the section, and adjusts the shape of the three-dimensional image by arranging each identified sensor position on the set movement line.

3. The image processing device described in claim 2, wherein the control unit: associates a first position on the at least one X-ray image corresponding to the first point with the position of one end of the movement line; associates a second position on the at least one X-ray image corresponding to the second point with the position of the other end of the movement line; sets a connecting line connecting the first position and the second position on the at least one X-ray image; and sets a shape of the movement line in response to an operation of manually changing the shape of the set connecting line.

4. The image processing device of claim 2, wherein the control unit: associates a first position on the at least one X-ray image corresponding to the first point with a position at one end of the movement line; associates a second position on the at least one X-ray image corresponding to the second point with a position at the other end of the movement line; and sets a shape of the movement line to a shape corresponding to a connecting line connecting the first position, the second position, and the at least one position in response to an operation of manually specifying at least one position different from the first position and the second position on the at least one X-ray image.

5. The image processing device described in claim 2, wherein the control unit: associates a first position on the at least one X-ray image corresponding to the first point with the position of one end of the line of movement; associates a second position on the at least one X-ray image corresponding to the second point with the position of the other end of the line of movement; automatically detects an image of the catheter axis from the at least one X-ray image; and sets the shape of the line of movement based on the detected image.

6. The image processing device described in claim 2, wherein the control unit: associates a first position on the at least one X-ray image corresponding to the first point with the position of one end of the line of movement; associates a second position on the at least one X-ray image corresponding to the second point with the position of the other end of the line of movement; automatically detects an image of the sensor from each of a plurality of X-ray images taken by the X-ray imaging device while the sensor moves through the section; and sets a shape of the line of movement based on the detected images.

7. The image processing device of claim 2, wherein the control unit corresponds the first position to a position at one end of the movement line in response to an operation of manually specifying a first position corresponding to the first point on the at least one X-ray image when the sensor is present at the first point.

8. The image processing device according to claim 7, wherein the control unit corresponds the second position to the position of the other end of the movement line in response to an operation of manually specifying a second position corresponding to the second point on the at least one X-ray image when the sensor is present at the second point.

9. The image processing device described in claim 7, wherein the control unit acquires ratio information indicating a ratio of a dimension of the at least one X-ray image to a dimension of the three-dimensional image, and based on the acquired ratio information, corresponds a position on the at least one X-ray image that is a distance away from the first position corresponding to the distance of the movement line to the other end position of the movement line as a second position on the at least one X-ray image corresponding to the second point.

10. An image processing device as described in claim 2, wherein the control unit automatically detects a first position corresponding to the first point on the at least one X-ray image when the sensor is present at the first point, and associates the first position with a position at one end of the movement line.

11. The image processing device according to claim 10, wherein the control unit automatically detects a second position corresponding to the second point on the at least one X-ray image when the sensor is located at the second point, and associates the second position with the position of the other end of the movement line.

12. The image processing device according to claim 2, wherein the control unit causes the three-dimensional image to be displayed on a display in a state where the three-dimensional image is superimposed on the at least one X-ray image.

13. The image processing device according to claim 12, wherein the control unit updates the three-dimensional image in response to a change in the movement line while the movement line is being set.

14. An image processing system comprising: an image processing device according to any one of claims 1 to 13; and a display for displaying the three-dimensional image and the at least one X-ray image.

15. An image processing method for generating a three-dimensional image by imaging the three-dimensional structure of a biological lumen based on a plurality of two-dimensional images obtained by a sensor attached to a catheter inserted into the biological lumen moving along the axis of the catheter through a section from a first point to a second point within the biological lumen, the image processing method including a control unit adjusting the shape of the three-dimensional image to match the shape of the axis of the catheter in the section identified using at least one X-ray image including an image of at least a portion of the catheter, taken by an X-ray imaging device that sees through the biological lumen.

16. An image processing program that causes a computer to generate a three-dimensional image by imaging the three-dimensional structure of a biological lumen based on multiple two-dimensional images obtained by a sensor attached to a catheter inserted into a biological lumen moving along the axis of the catheter along a section from a first point to a second point within the biological lumen, and that causes the computer to perform operations including adjusting the shape of the three-dimensional image to match the shape of the axis of the catheter in the section, which is identified using at least one X-ray image including an image of at least a portion of the catheter, taken by an X-ray imaging device that visualizes the biological lumen.