Image processing device, image processing system, image processing method, and image processing program

Abstract

Provided is an image processing device for generating a three-dimensional image by imaging a three-dimensional structure of a biological lumen on the basis of tomographic information obtained by a sensor moving in the biological lumen, and for displaying the generated three-dimensional image on a display, the image processing device comprising a control unit for detecting a branch lumen branching from the biological lumen on the basis of the tomographic information or the three-dimensional image, calculating, as a branching angle with respect to a reference direction, the position at which the detected branch lumen branches in the circumferential direction of the biological lumen, and displaying, on the display along with the three-dimensional image, angle information related to the calculated branching angle and position information indicating the branching position at which the branch lumen branches from the biological lumen in the three-dimensional image.

Description

Image processing device, image processing system, image processing method, and image processing program

[0001] The present disclosure relates to an image processing device, an image processing system, an image processing method, and an image processing program.

[0002] Patent Document 1 discloses a method for identifying the direction of a patient's ventral side in a three-dimensional image of biological tissue that is generated and displayed based on a two-dimensional ultrasound image acquired by a transducer of an ultrasound catheter.

[0003] International Publication No. 2022 / 045182

[0004] During catheter-based stent graft placement procedures, it is necessary to place the stent graft in a position that does not block branching blood vessels. Therefore, it is important to inject contrast into the blood vessels and take X-rays at an angle that allows accurate observation of the branching locations of major blood vessels. However, the branching locations of major blood vessels vary from patient to patient. Traditionally, surgeons have adjusted the imaging angle by repeatedly injecting contrast and taking X-rays. However, the more contrast injections and X-rays are performed, the greater the amount of contrast used and X-ray exposure, and the longer the procedure takes. This can result in a greater burden on the patient.

[0005] An object of the present disclosure is to reduce the burden on the living body when checking the branching position of a biological lumen.

[0006] Some aspects of the present disclosure are set forth below.

[0007] [1] An image processing device that visualizes the three-dimensional structure of a biological lumen based on tomographic information obtained by a sensor moving within the biological lumen to generate a three-dimensional image and displays the generated three-dimensional image on a display, the image processing device comprising: a control unit that detects branch lumens branching from the biological lumen based on the tomographic information or the three-dimensional image; calculates the position at which the detected branch lumen branches in the circumferential direction of the biological lumen as a branch angle relative to a reference direction, or calculates the image display direction at which the three-dimensional image is displayed on the display in the circumferential direction of the three-dimensional image as a display angle relative to the front direction of the living body in the three-dimensional image; and displays angle information related to the calculated branch angle or display angle and position information indicating the branch position at which the branch lumen branches from the biological lumen in the three-dimensional image on the display together with the three-dimensional image.

[0008] [2] The image processing device described in [1], wherein the control unit acquires an X-ray image taken by an X-ray imaging device that sees through the biological lumen, adjusts the image display orientation so that the acquired X-ray image can be seen from a viewpoint corresponding to the imaging angle at which the X-ray image was taken by the X-ray imaging device, and then displays the three-dimensional image on the display by superimposing it on the X-ray image.

[0009] [3] The image processing device according to [2], wherein the control unit changes the image display orientation in accordance with a change in the shooting angle.

[0010] [4] The image processing device according to [1], wherein the control unit updates the angle information in response to an operation to manually change the image display orientation.

[0011] [5] The image processing device according to any one of [1] to [4], wherein the control unit calculates the branch angle using the image display orientation as the reference direction, and causes the display to display the calculated value of the branch angle as the angle information.

[0012] [6] The image processing device according to any one of [1] to [4], wherein the control unit calculates the branch angle using a direction perpendicular to the image display orientation as the reference direction, and causes the display to display the calculated value of the branch angle as the angle information.

[0013] [7] The image processing device according to any one of [1] to [4], wherein the control unit calculates the branching angle using a direction perpendicular to an imaging angle at which an X-ray image is taken by an X-ray imaging device that sees through the biological lumen as the reference direction, and causes the calculated value of the branching angle to be displayed on the display as the angle information.

[0014] [8] The image processing device according to any one of [1] to [4], wherein the control unit calculates the branching angle, and causes the display to display, as the angle information, a recommended value of the imaging angle at which an X-ray image is taken by an X-ray imaging device that sees through the biological lumen, which is calculated from the branching angle.

[0015] [9] The image processing device according to any one of [1] to [8], wherein the control unit calculates the branch angle, and when the branch angle is within a predetermined range with respect to the image display orientation, highlights the angle information or the position information on the display.

[0016]

[10] The image processing device described in any of [1] to [8], wherein the control unit calculates the branching angle, and when the branching angle is within a predetermined range of an imaging angle at which an X-ray image is taken by an X-ray imaging device that sees through the biological lumen, highlights the angle information or the position information on the display.

[0017]

[11] The image processing device according to any one of [1] to

[10] , wherein the control unit calculates the branch angle, and sets a color selected based on the branch angle as the angle information as a position display color for displaying the position information.

[0018]

[12] The image processing device according to any one of [1] to

[11] , wherein the control unit calculates the branch angle, and causes the display to display, as the angle information, a figure facing a direction determined by the branch angle.

[0019]

[13] The image processing device according to any one of [1] to [4], wherein the control unit calculates the display angle, and causes the display to display, as the angle information, information indicating whether the image display orientation is shifted to the left or right relative to the forward direction of the living body, and the magnitude of the shift.

[0020]

[14] The image processing device according to any one of [1] to

[13] , wherein the control unit causes the display to display, as the position information, a figure to be placed at the branch position.

[0021]

[15] An image processing system comprising: the image processing device according to any one of [1] to

[14] ; and the display.

[0022]

[16] An image processing method for imaging the three-dimensional structure of a biological lumen based on tomographic information obtained by a sensor moving within the biological lumen to generate a three-dimensional image, and displaying the generated three-dimensional image on a display, the image processing method comprising: detecting a branch lumen branching from the biological lumen based on the tomographic information or the three-dimensional image; calculating a position at which the detected branch lumen branches in a circumferential direction of the biological lumen as a branching angle with respect to a reference direction, or calculating an image display direction at which the three-dimensional image is displayed on the display in a circumferential direction of the three-dimensional image as a display angle with respect to a front direction of the living body in the three-dimensional image; and displaying angle information related to the calculated branching angle or display angle, and position information indicating the branching position at which the branch lumen branches from the biological lumen in the three-dimensional image, together with the three-dimensional image, on the display.

[0023]

[17] An image processing program that causes a computer that visualizes the three-dimensional structure of a biological lumen based on tomographic information obtained by a sensor moving within the biological lumen to generate a three-dimensional image and displays the generated three-dimensional image on a display to perform operations including: detecting branch lumens branching from the biological lumen based on the tomographic information or the three-dimensional image; calculating a branching position of the detected branch lumen in the circumferential direction of the biological lumen as a branching angle with respect to a reference direction, or calculating an image display direction in the circumferential direction of the three-dimensional image at which the three-dimensional image is displayed on the display as a display angle with respect to the front direction of the living body in the three-dimensional image; and displaying, together with the three-dimensional image, angle information related to the calculated branching angle or display angle, and position information indicating the branching position at which the branch lumen branches from the biological lumen in the three-dimensional image on the display.

[0024] According to the present disclosure, it is possible to reduce the burden on the living body when checking the branching position of a biological lumen.

[0025] 1 is a block diagram showing a configuration of an image processing system according to an embodiment of the present disclosure. FIG. 2 is a flowchart showing the operation of an image processing device according to an embodiment of the present disclosure. FIG. 3 is a diagram showing an example of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 4 is a diagram showing a definition of a branching angle according to an embodiment of the present disclosure. FIG. 5 is a diagram showing an example of a cross-sectional image according to an embodiment of the present disclosure. FIG. 6 is a diagram showing an example of a three-dimensional image according to an embodiment of the present disclosure. FIG. 7 is a diagram showing another example of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 8 is a diagram showing a definition of left and right shooting angles according to an embodiment of the present disclosure. FIG. 9 is a diagram showing yet another example of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 10 is a diagram showing yet another example of a screen displayed on a display according to an embodiment of the present disclosure. FIG. 11 is a diagram showing yet another example of a screen displayed on a display according to an embodiment of the present disclosure.

[0026] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0027] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0028] The configuration of an image processing system 10 according to this embodiment will be described with reference to FIG.

[0029] The image processing system 10 includes an image processing device 20, a sensor 28, an X-ray imaging device 30, an input device 40, and a display 50. The image processing device 20 is connected to the sensor 28, the X-ray imaging device 30, the input device 40, and the display 50 via a cable, a network, or wirelessly.

[0030] The image processing device 20 is, for example, a general-purpose computer such as a PC, a server computer such as a cloud server, or a dedicated computer. "PC" is an abbreviation for personal computer. The image processing device 20 may be installed in a medical facility such as a hospital, or may be installed in a facility separate from the medical facility, such as a data center.

[0031] The sensor 28 is, for example, an ultrasound transducer used in IVUS (intravascular ultrasound). The sensor 28 transmits ultrasound waves while moving through a body lumen such as a blood vessel, and receives reflected waves of the transmitted ultrasound waves.

[0032] The X-ray imaging device 30 is, for example, a C-arm. The X-ray imaging device 30 is installed in a medical facility and performs transillumination of a living body lumen.

[0033] The input device 40 is, for example, a pointing device such as a mouse, a keyboard, or a touch screen that is integrated with the display 50. The input device 40 is installed in a medical facility and is used by an operator such as a doctor or a clinical engineer to control the display of various information, including images, on the display 50.

[0034] The display 50 is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electroluminescent. The display 50 is installed in a medical facility and displays various information, including images, to an operator to support catheter surgery such as a stent graft placement procedure.

[0035] The configuration of an image processing apparatus 20 according to this embodiment will be described with reference to FIG.

[0036] The image processing device 20 includes a control unit 21 , a storage unit 22 , a communication unit 23 , an input unit 24 , and an output unit 25 .

[0037] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. An example of the programmable circuit is an FPGA. "FPGA" is an abbreviation for field-programmable gate array. An example of the dedicated circuit is an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 controls each part of the image processing device 20 and executes processing related to the operation of the image processing device 20.

[0038] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. Flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. Magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 22 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores information used in the operation of the image processing device 20 and information obtained by the operation of the image processing device 20 .

[0039] The communication unit 23 includes at least one communication module. The communication module is, for example, a module that complies with a wired LAN communication standard such as Ethernet (registered trademark) or a wireless LAN communication standard such as IEEE 802.11. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. The communication unit 23 receives information used in the operation of the image processing device 20 and transmits information obtained by the operation of the image processing device 20. The communication unit 23 is connected to the sensor 28 and the X-ray imaging device 30.

[0040] The input unit 24 includes at least one input interface. The input interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface. The input unit 24 accepts an operation to input information used in the operation of the image processing device 20. The input unit 24 is connected to an input device 40.

[0041] The output unit 25 includes at least one output interface. The output interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). The output unit 25 outputs information obtained by the operation of the image processing device 20. The output unit 25 is connected to a display 50.

[0042] The functions of the image processing device 20 are realized by executing an image processing program according to this embodiment on a processor serving as the control unit 21. That is, the functions of the image processing device 20 are realized by software. The image processing program causes a computer to execute the operations of the image processing device 20, thereby causing the computer to function as the image processing device 20. That is, the computer functions as the image processing device 20 by executing the operations of the image processing device 20 in accordance with the image processing program.

[0043] The program can be stored on a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include flash memory, magnetic recording devices, optical disks, magneto-optical recording media, and ROMs. The program can be distributed by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs that store the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program can also be distributed by storing it in the storage of a server and transferring it from the server to another computer. The program can also be provided as a program product.

[0044] A computer temporarily stores a program stored on a portable medium or transferred from a server in its main storage device. The computer then reads the program stored in the main storage device with a processor and executes processing in accordance with the read program. The computer may also read the program directly from the portable medium and execute processing in accordance with the program. The computer may also execute processing in accordance with the received program each time a program is transferred from the server to the computer. Processing may also be executed using a so-called ASP-type service that realizes functions simply by issuing execution instructions and obtaining results, without transferring the program from the server to the computer. "ASP" is an abbreviation for application service provider. Programs include information used for processing by a computer that is equivalent to a program. For example, data that is not a direct instruction to a computer but has properties that define computer processing falls under the category of "equivalent to a program."

[0045] Some or all of the functions of the image processing device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, some or all of the functions of the image processing device 20 may be realized by hardware.

[0046] The operation of the image processing device 20 according to this embodiment will be described with reference to Fig. 2. The operation described below corresponds to the image processing method according to this embodiment. That is, the image processing method according to this embodiment includes steps S1 to S5 shown in Fig. 2.

[0047] In S1, the control unit 21 acquires tomographic information obtained by the sensor 28 moving inside the biological lumen. Specifically, the control unit 21 receives, via the communication unit 23, reflected wave information regarding the reflected wave received by the sensor 28 from the sensor 28 as tomographic information.

[0048] In S2, the control unit 21 visualizes the three-dimensional structure of the biological lumen based on the tomographic information acquired in S1, and generates a three-dimensional image 70 such as those shown in Figures 3, 7, 9, and 11 to 16. Specifically, the control unit 21 visualizes the cross-sectional structure of the biological lumen based on the tomographic information received in S1, thereby generating a plurality of cross-sectional images. The control unit 21 then stacks the generated cross-sectional images to generate the three-dimensional image 70.

[0049] In S3, the control unit 21 detects branch lumens branching from the biological lumen based on the tomographic information acquired in S1 or the three-dimensional image 70 generated in S2. For example, when the biological lumen is a major blood vessel such as the abdominal aorta or its lumen, the branch lumens are branch blood vessels such as the celiac artery, superior mesenteric artery, renal artery, or inferior mesenteric artery or their lumen. Specifically, the control unit 21 analyzes the multiple cross-sectional images generated in S2 or the three-dimensional image 70 generated in S2 to detect the branch lumens. Known image recognition technology can be used as a method for analyzing the multiple cross-sectional images or the three-dimensional image 70 to detect the branch lumens. Machine learning such as deep learning may also be used.

[0050] In S4, the control unit 21 calculates a branching angle relative to the reference direction, or a display angle relative to the front direction of the living body in the three-dimensional image 70 generated in S2. The branching angle refers to the position at which the branch lumen detected in S3 branches in the circumferential direction of the living body lumen, expressed as an angle relative to the reference direction. Specific examples of branching angles will be described later. The display angle refers to the image display direction in which the three-dimensional image 70 is displayed on the display 50, expressed as an angle relative to the front direction of the living body in the three-dimensional image 70, in the circumferential direction of the three-dimensional image 70. Specific examples of display angles will also be described later.

[0051] In S5, the control unit 21 displays the three-dimensional image 70 generated in S2 on the display 50. The control unit 21 displays angle information 74 related to the branching angle or display angle calculated in S4 and position information 75 indicating the branching position at which the branch lumen branches off from the biological lumen in the three-dimensional image 70 generated in S2, together with the three-dimensional image 70, on the display 50. For example, the control unit 21 displays screens 51 such as those shown in FIGS. 3, 7, 9, and 11 to 16 on the display 50.

[0052] The screen 51 includes a first area 52 and a second area 53 adjacent to the first area 52. In the examples shown in Figures 3, 7, 9, and 11 to 16, the second area 53 is arranged to the right of the first area 52, but the second area 53 may also be arranged to the left of the first area 52.

[0053] The first area 52 displays a two-dimensional image 60 and a human body model image 61. The two-dimensional image 60 corresponds to the most recent cross-sectional image among the multiple cross-sectional images generated in S2. The human body model image 61 is displayed so that its orientation is aligned with that of the two-dimensional image 60. In the examples shown in FIGS. 3, 7, 9, and 11 to 16, a catheter 26 equipped with a sensor 28 is inserted from the groin of the patient, with the upper side of the two-dimensional image 60 corresponding to the patient's abdomen and the lower side of the two-dimensional image 60 corresponding to the patient's back. Therefore, the human body model image 61 is displayed so that its bottom is visible, with its abdomen facing up and its back facing down. In the examples shown in FIGS. 3, 7, 9, and 11 to 16, the human body model image 61 is positioned in the upper left corner of the two-dimensional image 60, but the human body model image 61 may be positioned anywhere near the two-dimensional image 60. The position of the human body model image 61 may be fixed or may be freely changeable.

[0054] A slider 54 is also displayed in the first area 52. The slider 54 indicates the position of the sensor 28 in the longitudinal direction of the catheter 26. The slider 54 is located at the end of the first area 52 closer to the second area 53. Therefore, in the examples shown in FIGS. 3, 7, 9, and 11 to 16, the slider 54 is located at the right end of the first area 52.

[0055] The second area 53 displays a three-dimensional image 70 and a human body model image 71. In the examples shown in FIGS. 3 , 7 , 9 , and 11 to 16 , the angle of the three-dimensional image 70 corresponds to the angle at which the surgical subject is viewed from the front, the upper side of the three-dimensional image 70 corresponds to the side where the surgical subject's head is located, and the lower side of the three-dimensional image 70 corresponds to the side where the surgical subject's feet are located. Therefore, the human body model image 71 is displayed so that the front is visible, with the head facing up and the feet facing down. In the examples shown in FIGS. 3 , 7 , 9 , and 11 to 16 , the human body model image 71 is positioned in the upper left corner of the three-dimensional image 70, but the human body model image 71 may be positioned anywhere near the three-dimensional image 70. The position of the human body model image 71 may be fixed or may be arbitrarily changeable. The three-dimensional image 70 may be cropped along the longitudinal axis of the catheter 26 so that the internal structure of the biological lumen can be observed, and the cropped portion may be hidden.

[0056] A cross-sectional image 72 is also displayed in the second area 53. The cross-sectional image 72 is a reduced version of the two-dimensional image 60 displayed in the first area 52. A camera icon 73 indicating the position of the viewpoint is placed on the edge of the cross-sectional image 72. This makes it easy to see from which direction the three-dimensional image 70 displayed in the second area 53 is viewed in the cross-sectional image 72.

[0057] The second area 53 further displays angle information 74 and position information 75. In the examples shown in FIGS. 3 , 7 , 9 , and 11 to 13 , information related to the branching angles of five branch lumens is displayed as the angle information 74, and information indicating five branch positions is displayed as the position information 75. In the example shown in FIG. 14 , information related to the branching angles of two branch lumens is displayed as the angle information 74, and information indicating two branch positions is displayed as the position information 75. In the example shown in FIG. 15 , information related to the display angle is displayed as the angle information 74, and information indicating five branch positions is displayed as the position information 75. In the example shown in FIG. 16 , information related to the display angle is displayed as the angle information 74, and information indicating two branch positions is displayed as the position information 75.

[0058] As a modified example of this embodiment, the control unit 21 may acquire X-ray images captured by an X-ray imaging device 30 that provides a transillumination of a living body lumen. Specifically, the control unit 21 may receive the X-ray images from the X-ray imaging device 30 via the communication unit 23. In this modified example, the control unit 21 identifies the orientation of a marker provided on the catheter 26 in the received X-ray image and the orientation of the marker in the tomographic information (two-dimensional ultrasound image) on which the three-dimensional image 70 is based, and determines a predetermined orientation of the living body (e.g., the front of the living body) in the three-dimensional image 70 by associating them. Furthermore, when receiving the X-ray images from the X-ray imaging device 30, the control unit 21 also receives the imaging angle at which the X-ray images were captured (in this embodiment, the positioning angle of the C-arm at the time of imaging). Based on the recognized predetermined orientation of the living body in the three-dimensional image 70 (e.g., the front of the living body) and the received imaging angle, the control unit 21 adjusts the image display orientation so that the acquired X-ray image is viewed from a viewpoint corresponding to the imaging angle at which the X-ray image was captured by the X-ray imaging device 30, and displays the three-dimensional image 70 on the display 50, superimposed on the X-ray image. For example, the three-dimensional image 70 and the human body model image 71 are displayed in the second area 53 of the screen 51, superimposed on the X-ray image. The control unit 21 may automatically change the image display orientation in accordance with changes in the imaging angle. In this modification, the control unit 21 may accept the imaging angle at which the X-ray image is captured as an input from the surgeon (operator) rather than receiving it from the X-ray imaging device 30. Alternatively, the control unit 21 may receive an X-ray image captured before receiving the imaging angle and use the X-ray image to identify the orientation of the marker provided on the catheter 26.

[0059] The details of steps S4 and S5 in the example shown in FIG. 3 will be described below.

[0060] In S4, the control unit 21 calculates the branching angle using the image display orientation as the reference direction. If five branching vessels are detected as branching lumens in S3, five calculated branching angles are obtained in S4. For example, when the three-dimensional image 70 is displayed on the display 50 so that the front of the living body is visible, as shown in FIG. 4, the abdomen is 0° and the back is 180° when viewed from the feet of the surgical patient. In the example shown in FIG. 3, the five calculated branching angles are "0°," "10°," "90°," "200°," and "240°." A calculated branching angle of "0°" means that the corresponding branching vessel branches off from the main vessel at the center of the abdominal side of the surgical patient. A calculated branching angle of "10°" means that the corresponding branching vessel branches off from the main vessel at a position 10° to the left of the center of the abdominal side of the surgical patient. A calculated branching angle of "90°" means that the corresponding branch vessel branches off from the main vessel at a position 90° to the left of the center of the patient's abdomen. A calculated branching angle of "200°" means that the corresponding branch vessel branches off from the main vessel at a position 20° to the right of the center of the patient's back. A calculated branching angle of "240°" means that the corresponding branch vessel branches off from the main vessel at a position 60° to the right of the center of the patient's back.

[0061] As a specific method for calculating the branch angle, various methods can be used as follows.

[0062] As a first method, the branching angle may be calculated directly from the multiple cross-sectional images generated in S2. In this example, as shown in FIG. 5 , when the controller 21 detects a unique branched lumen that is continuous in the pullback direction of the catheter 26 in S3, it identifies, from the multiple cross-sectional images, a cross-sectional image 80 that includes a branching center 81, which is the center of the detected branched lumen in a plane extending in the width direction and the pullback direction. The controller 21 calculates a center of gravity 82 of the biological lumen into which the catheter 26 is inserted in the identified cross-sectional image 80. The controller 21 calculates a branching vector 83 that points from the center of gravity 82 to the branching center 81 in the identified cross-sectional image 80. The controller 21 calculates the angle of the calculated branching vector 83 with respect to the front direction of the living body as the branching angle. In this example, if the cross-sectional image 80 is tilted with respect to the transverse plane of the living body, an error may occur in the calculated value of the branching angle. However, if the living body lumen is the abdominal aorta, the cross-sectional image 80 will not be tilted significantly, and only a negligible error will occur.

[0063] As a second method, the branching angle may be calculated taking into account the tilt of the multiple cross-sectional images generated in S2. In this example, as shown in FIG. 6 , the control unit 21 calculates the center of gravity of the biological lumen into which the catheter 26 is inserted in each of the multiple cross-sectional images and identifies a center of gravity line 76 connecting the centers of gravity along the pullback direction. When the control unit 21 detects a unique branch lumen that is continuous in the pullback direction of the catheter 26 in S3, the control unit 21 identifies, from the multiple cross-sectional images, a cross-sectional image 80 that includes a branch center 81, which is the center of the detected branch lumen in a plane extending in the width direction and the pullback direction. The control unit 21 calculates a branch vector 83 that points from the center of gravity 82 to the branch center 81 in the identified cross-sectional image 80. The control unit 21 calculates a tangent vector 77 of the center of gravity line 76 at the center of gravity 82 of the identified cross-sectional image 80. The control unit 21 calculates a rotation matrix from the cross-sectional plane of the biological body to the plane 78 of the identified cross-sectional image 80 based on the calculated tangent vector 77, thereby calculating the tilt of the cross-sectional image 80 with respect to the cross-sectional plane of the biological body. Based on the calculated rotation matrix, the control unit 21 converts the calculated branch vector 83 into a three-dimensional vector in the three-dimensional image 70. The control unit 21 calculates a projected branch vector by projecting the three-dimensional vector onto a cross-section of the living body. The control unit 21 calculates the angle of the calculated projected branch vector with respect to the front direction of the living body as the branch angle. In this example, even if the cross-sectional image 80 is tilted with respect to the cross-section of the living body, a more accurate calculated value of the branch angle can be obtained.

[0064] As a third method, the branching angle may be calculated based on the three-dimensional image 70 generated in S2. In this example, the forward direction of the organism is defined as the x direction, the direction perpendicular to the forward direction of the organism in a cross section of the organism is defined as the y direction, and the pullback direction is defined as the z direction. The control unit 21 identifies the x, y, and z coordinates of the branching position in the three-dimensional image 70. The control unit 21 projects the x and y coordinates of the branching position onto the x-y plane and calculates a branching vector from the origin on the x-y plane. The control unit 21 calculates the angle of the calculated branching vector with respect to the x direction as the branching angle. In this example, the three-dimensional image 70 is used, so a more accurate branching angle calculation value can be obtained.

[0065] In S5, the control unit 21 causes the display 50 to display the calculated values ​​of the branch angles as angle information 74. As a result, as shown in FIG. 3 , the calculated values ​​of the branch angles "0°," "10°," "90°," "200°," and "240°" are displayed near the corresponding branch positions on the screen 51. The control unit 21 causes the display 50 to display, as position information 75, a figure to be placed at the branch position. In the example shown in FIG. 3 , a mark such as a circle is placed at each of the five branch positions.

[0066] The control unit 21 may update the angle information 74 in response to a manual operation to change the image display orientation. For example, when the 3D image 70 is rotated so that the right side of the living body is visible, the abdomen is 90° and the back is 270° when viewed from the feet of the surgical patient. As a result, the updated values ​​of the five branch angles are "90°," "100°," "180°," "290°," and "330°." The updated values ​​of the branch angles "90°," "100°," "180°," "290°," and "330°" are then displayed on the screen 51 near the corresponding branch positions.

[0067] When the branch angle is within a predetermined range with respect to the image display orientation, the control unit 21 may highlight the angle information 74 or the position information 75 on the display 50. For example, when the "predetermined range" is 85° or more and 95° or less, or 265° or more and 275° or less, the calculated value of the branch angle "90°" may be highlighted in a conspicuous color, size, or font. Alternatively, a mark placed at the branch position corresponding to the calculated value of the branch angle "90°" may be highlighted in a conspicuous color, size, or shape, or may flash.

[0068] The control unit 21 may highlight the angle information 74 or the position information 75 on the display 50 when the branching angle is within a predetermined range of the imaging angle at which the X-ray image is captured by the X-ray imaging device 30. For example, when the imaging angle is 0° and matches the image display orientation, and the "predetermined range" is 85° to 95° or 265° to 275°, the calculated branching angle value "90°" is highlighted with a conspicuous color, size, or font. Alternatively, a mark located at the branching position corresponding to the calculated branching angle value "90°" may be highlighted with a conspicuous color, size, or shape, or may flash. In this modification, the control unit 21 determines whether the branching angle is within a predetermined range of the imaging angle by receiving the imaging angle at which the X-ray image is captured from the X-ray imaging device 30 or by accepting it as input from the surgeon (operator).

[0069] The control unit 21 may set a color selected according to the branch angle as the angle information 74 as the position display color for displaying the position information 75. For example, if blue is selected when the branch angle is 85° or more and 95° or less, or 265° or more and 275° or less, and red is selected otherwise, then the mark placed at the branch position corresponding to the calculated value of the branch angle "90°" is set to blue, and the other marks are set to red.

[0070] The control unit 21 may display, on the display 50, a graphic that faces a direction determined by the branching angle, as the angle information 74. For example, an arrow, a triangle, a false blood vessel, or a long rod may be arranged so as to face radially outward from each of the five branching positions on the three-dimensional image 70.

[0071] A modification of the example shown in FIG. 3 will now be described with reference to FIG.

[0072] In S5, the control unit 21 further displays, as angle information 74, a recommended value of the imaging angle at which an X-ray image is captured by the X-ray imaging device 30, which is calculated from the branch angle, on the display 50. For example, when the X-ray imaging device 30 is a C-arm, as shown in FIG. 8 , LAO 0° or RAO 0° is when the detector of the C-arm is directly above the abdomen of the surgical patient, LAO 90° is when the detector is rotated 90° to the left of the patient, and RAO 90° is when the detector is rotated 90° to the right of the patient. The detector is also called an image intensifier (II). "II" is an abbreviation for image intensifier. "LAO" is an abbreviation for left anterior oblique. "RAO" is an abbreviation for right anterior oblique. In the example shown in FIG. 7 , the calculated values ​​of the five branching angles are assumed to be "0°," "10°," "90°," "200°," and "240°," the same as in the example shown in FIG. 3 . Then, from these calculated values ​​of the five branching angles, five recommended imaging angles are assumed to be "LAO 90°," "RAO 80°," "LAO 0°," "RAO 70°," and "RAO 30°." These five recommended imaging angles indicate imaging angles at which the positions at which the corresponding branching vessels branch can be observed from directly beside the imaging angles. As shown in FIG. 7 , the recommended imaging angles "LAO 90°," "RAO 80°," "LAO 0°," "RAO 70°," and "RAO 30°" are displayed on the screen 51 near the corresponding branching positions along with the calculated values ​​of the branching angles.

[0073] A modification of the example shown in FIG. 7 will now be described with reference to FIG.

[0074] In S5, the control unit 21 displays on the display 50, as angle information 74, not only the recommended imaging angle in the left-right direction of the surgical patient but also the recommended imaging angle in the up-down direction of the surgical patient. For example, when the X-ray imaging device 30 is a C-arm, as shown in Fig. 10, a state in which the detector of the C-arm is directly above the abdomen of the surgical patient is CRANIAL 0° or CAUDAL 0°, a state in which the detector is rotated 90° upward of the patient is CRANIAL 90°, and a state in which the detector is rotated 90° downward of the patient is CAUDAL 90°. In the example shown in Fig. 9, the recommended values ​​of five imaging angles are "CAUDAL 20°," "CRANIAL 10°," "CRANIAL 0°," "CRANIAL 40°," and "CAUDAL 60°." These five recommended imaging angles indicate imaging angles at which the positions at which the corresponding branching vessels branch can be observed from directly beside them. As shown in Fig. 9, the recommended values ​​for the upper and lower imaging angles, "CAUDAL 20°," "CRANIAL 10°," "CRANIAL 0°," "CRANIAL 40°," and "CAUDAL 60°," are displayed on the screen 51 near the corresponding branching positions, along with the calculated values ​​of the branching angles and the recommended values ​​for the left and right imaging angles.

[0075] Just as the recommended values ​​for the horizontal imaging angles are calculated from the horizontal branching angles, the recommended values ​​for the vertical imaging angles can also be calculated from the vertical branching angles. When the second method described above is used as a specific method for calculating the branching angles, the control unit 21 calculates a projected branching vector by projecting a three-dimensional vector onto the sagittal plane of the living body. The control unit 21 calculates the angle of the calculated projected branching vector relative to the front direction of the living body as the vertical branching angle. When the third method described above is used as a specific method for calculating the branching angles, the control unit 21 projects the xz coordinates of the branching position onto the xz plane and calculates a branching vector from the origin on the xz plane. The control unit 21 calculates the angle of the calculated branching vector relative to the x direction as the vertical branching angle. In either case, the control unit 21 can calculate the recommended values ​​for the vertical imaging angles from the calculated vertical branching angles.

[0076] Another modification of the example shown in FIG. 7 will be described with reference to FIG.

[0077] 11 , on the screen 51, only the recommended imaging angle values ​​"LAO 90°," "RAO 80°," "LAO 0°," "RAO 70°," and "RAO 30°" may be displayed near the corresponding branch positions as the angle information 74. In other words, the calculated values ​​of the branch angles may not be displayed.

[0078] A modification of the example shown in FIG. 11 will be described with reference to FIG.

[0079] 12 , on the screen 51, recommended imaging angle ranges "LAO 85-90°," "RAO 85-90°," "RAO 75-85°," "LAO 0-5°," "RAO 0-5°," "RAO 65-75°," and "RAO 25-35°" may be displayed near the corresponding branch positions as angle information 74, instead of the recommended imaging angle values ​​"LAO 90°," "RAO 80°," "LAO 0°," "RAO 70°," and "RAO 30°." In this example, it can be seen that setting the imaging angle to RAO 85 makes it easier to observe two branch positions simultaneously.

[0080] The details of steps S4 and S5 in the example shown in FIG. 13 will be described below, focusing on the differences from the example shown in FIG.

[0081] In S4, the control unit 21 calculates the branching angle using a direction perpendicular to the image display orientation as the reference direction. If five branching vessels are detected as branching lumens in S3, five calculated branching angles are obtained in S4. For example, when the three-dimensional image 70 is displayed on the display 50 so that the front of the living body is visible, the abdomen is 90° and the back is −90° as viewed from the feet of the surgical patient. In the example shown in FIG. 13, the five calculated branching angles are “90°,” “80°,” “0°,” “−70°,” and “−30°.” A calculated branching angle of “90°” means that the corresponding branching vessel branches off from the main vessel at the center of the abdominal side of the surgical patient. A calculated branching angle of “80°” means that the corresponding branching vessel branches off from the main vessel at a position 10° to the left of the center of the abdominal side of the surgical patient. A calculated branching angle of "0°" means that the corresponding branch vessel branches off from the main vessel at a position 90° to the left of the center of the surgical patient's abdomen. A calculated branching angle of "-70°" means that the corresponding branch vessel branches off from the main vessel at a position 20° to the right of the center of the surgical patient's back. A calculated branching angle of "-30°" means that the corresponding branch vessel branches off from the main vessel at a position 60° to the right of the center of the surgical patient's back.

[0082] In S5, the control unit 21 causes the display 50 to display the calculated values ​​of the branch angles as angle information 74. As a result, as shown in Fig. 13, the calculated values ​​of the branch angles "90°", "80°", "0°", "-70°", and "-30°" are displayed on the screen 51 near the corresponding branch positions.

[0083] In the example shown in Figure 13, it is easy to understand how many degrees counterclockwise rotation is required to orient each branch position to the side. When the branch position is on the dorsal side, applying a minus sign makes it easy to distinguish from when the branch position is on the ventral side.

[0084] As a modification of the example shown in Fig. 13 , in S4, the control unit 21 may calculate the branching angle using a direction orthogonal to the imaging angle at which the X-ray image is captured by the X-ray imaging device 30 as the reference direction. For example, when the imaging angle is LAO 0° or RAO 0°, the calculated value of the branching angle is the same as that of the example shown in Fig. 13 . In this modification, the control unit 21 receives the imaging angle at which the X-ray image is captured from the X-ray imaging device 30 or accepts it as an input from the surgeon (operator), and thereby calculates the reference direction orthogonal to the imaging angle.

[0085] Referring to FIG. 14, a further modification of the example shown in FIG. 7 will be described.

[0086] In the example shown in FIG. 14 , two branch vessels are detected as branch lumens, and two calculated values ​​of branch angles, "100°" and "290°," are obtained. Then, from these calculated values ​​of branch angles, two recommended imaging angles, "LAO 10°" and "LAO 20°," are obtained. The calculated value of the branch angle, "100°," means that the corresponding branch vessel branches off from the main vessel at a position 80° to the left of the center of the patient's back. The calculated value of the branch angle, "290°," means that the corresponding branch vessel branches off from the main vessel at a position 70° to the right of the center of the patient's abdomen. The recommended values ​​of the two imaging angles indicate imaging angles at which the branching position of the corresponding branch vessel can be observed from directly to the side. As shown in FIG. 14 , on the screen 51, calculated branch angle values ​​"100°" and "290°" and recommended imaging angle values ​​"LAO 10°" and "LAO 20°" are displayed near the corresponding branch positions as angle information 74. Furthermore, instead of marks such as circles, arrows are displayed at each of the two branch positions as position information 75. The arrows may be positioned so as to point radially outward from each of the two branch positions on the three-dimensional image 70. This makes it easier to intuitively understand where the branch positions are located. The arrows may be positioned so as to extend from the center of the branch position, from a position slightly above the branch position, or from a position slightly below the branch position.

[0087] The details of steps S4 and S5 in the example shown in FIG. 15 will be described below.

[0088] In S4, the control unit 21 calculates the display angle. For example, when the three-dimensional image 70 is displayed on the display 50 so that the front of the living body is visible, the calculated value of the display angle is "0°."

[0089] In S5, the control unit 21 causes the display 50 to display, as angle information 74, information indicating whether the image display orientation is shifted left or right relative to the front direction of the living body and the magnitude of the shift. In the example shown in FIG. 15 , the display 50 displays the angle at which the 3D image 70 displayed on the screen 51 corresponds to the imaging angle at which the X-ray image is captured by the X-ray imaging device 30. Therefore, "LAO 0°" is displayed as the angle information 74. The control unit 21 causes the display 50 to display, as position information 75, a figure to be placed at the branch position. In the example shown in FIG. 15 , a mark such as a circle is placed at each of the five branch positions detected in S3.

[0090] The control unit 21 may update the angle information 74 in response to a manual operation to change the image display orientation. For example, when the three-dimensional image 70 is rotated so that the right side of the living body is visible, the updated display angle is "270°." Then, "RAO 90°" is displayed on the screen 51 as the angle information 74.

[0091] A modification of the example shown in FIG. 15 will be described with reference to FIG.

[0092] In the example shown in FIG. 16 , two branch blood vessels are detected as branch lumens. As shown in FIG. 16 , instead of marks such as circles, arrows are placed at each of the two branch positions on the screen 51 as position information 75. The arrows may be placed so as to point radially outward from each of the two branch positions on the three-dimensional image 70. This makes it easier to intuitively understand where the branch positions are located. The arrows may be placed so as to extend from the center of the branch position, from a position slightly above the branch position, or from a position slightly below the branch position.

[0093] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks shown in the block diagrams may be integrated, or one block may be divided. Two or more steps shown in the flowcharts may be executed in parallel or in a different order, instead of being executed in chronological order as described, depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible within the scope of the present disclosure.

[0094] REFERENCE SIGNS LIST 10 Image processing system 20 Image processing device 21 Control unit 22 Memory unit 23 Communication unit 24 Input unit 25 Output unit 26 Catheter 28 Sensor 30 X-ray imaging device 40 Input device 50 Display 51 Screen 52 First area 53 Second area 54 Slider 60 Two-dimensional image 61 Human body model image 70 Three-dimensional image 71 Human body model image 72 Cross-sectional image 73 Camera icon 74 Angle information 75 Position information 76 Center of gravity line 77 Tangent vector 78 Plane 80 Cross-sectional image 81 Branch center 82 Center of gravity 83 Branch vector

Claims

1. An image processing device that visualizes the three-dimensional structure of a biological lumen based on tomographic information obtained by a sensor moving within the biological lumen, generates a three-dimensional image, and displays the generated three-dimensional image on a display, comprising: a control unit that detects a branch lumen branching off from the biological lumen based on the tomographic information or the three-dimensional image; calculates a position at which the detected branch lumen branches in the circumferential direction of the biological lumen as a branching angle relative to a reference direction, or calculates an image display orientation at which the three-dimensional image is displayed on the display in the circumferential direction of the three-dimensional image as a display angle relative to the front direction of the organism in the three-dimensional image; and displays, on the display together with the three-dimensional image, angle information related to the calculated branching angle or display angle, and position information indicating the branching position at which the branch lumen branches off from the biological lumen in the three-dimensional image.

2. The image processing device according to claim 1, wherein the control unit acquires an X-ray image taken by an X-ray imaging device that sees through the biological lumen, adjusts the image display orientation so that the acquired X-ray image can be seen from a viewpoint corresponding to the shooting angle at which the X-ray image was taken by the X-ray imaging device, and then superimposes the three-dimensional image on the X-ray image and displays it on the display.

3. The image processing device according to claim 2, wherein the control unit changes the image display orientation in accordance with the change in the shooting angle.

4. The image processing device according to claim 1, wherein the control unit updates the angle information in response to an operation for manually changing the image display orientation.

5. The image processing device according to claim 1, wherein the control unit calculates the branching angle using the image display orientation as the reference direction, and causes the display to display the calculated value of the branching angle as the angle information.

6. The image processing device according to claim 1, wherein the control unit calculates the branching angle using a direction perpendicular to the image display orientation as the reference direction, and causes the display to display the calculated value of the branching angle as the angle information.

7. The image processing device according to claim 1, wherein the control unit calculates the branching angle using a direction perpendicular to the shooting angle at which an X-ray image is taken by an X-ray imaging device that sees through the biological lumen as the reference direction, and causes the calculated value of the branching angle to be displayed on the display as the angle information.

8. The image processing device according to claim 1, wherein the control unit calculates the branching angle and causes the display to display, as the angle information, a recommended value of the shooting angle at which an X-ray image is taken by an X-ray imaging device that sees through the biological lumen, which is calculated from the branching angle.

9. The image processing device according to claim 1, wherein the control unit calculates the branching angle, and when the branching angle is within a predetermined range with respect to the image display orientation, causes the angle information or the position information to be highlighted on the display.

10. The image processing device of claim 1, wherein the control unit calculates the branching angle, and when the branching angle is within a predetermined range of an imaging angle at which an X-ray image is taken by an X-ray imaging device that visualizes the biological lumen, highlights the angle information or the position information on the display.

11. The image processing device according to claim 1, wherein the control unit calculates the branching angle, and sets a color selected based on the branching angle as the angle information as a position display color in which the position information is displayed.

12. The image processing device according to claim 1, wherein the control unit calculates the branching angle, and causes the display to display, as the angle information, a figure facing in a direction determined by the branching angle.

13. The image processing device according to claim 1, wherein the control unit calculates the display angle, and causes the display to display, as the angle information, information indicating whether the image display orientation is shifted to the left or right relative to the forward direction of the living body, and the magnitude of the shift.

14. The image processing device according to claim 1, wherein the control unit causes the display to display, as the position information, a figure to be placed at the branch position.

15. An image processing system comprising: an image processing device according to any one of claims 1 to 14; and a display.

16. An image processing method for imaging the three-dimensional structure of a biological lumen based on tomographic information obtained by a sensor moving within the biological lumen to generate a three-dimensional image and displaying the generated three-dimensional image on a display, comprising: detecting a branch lumen branching off from the biological lumen based on the tomographic information or the three-dimensional image; calculating a position at which the detected branch lumen branches off in the circumferential direction of the biological lumen as a branching angle relative to a reference direction, or calculating an image display orientation at which the three-dimensional image is displayed on the display in the circumferential direction of the three-dimensional image as a display angle relative to the front direction of the organism in the three-dimensional image; and displaying, together with the three-dimensional image, angle information related to the calculated branching angle or display angle, and position information indicating the branching position at which the branch lumen branches off from the biological lumen in the three-dimensional image on the display.

17. An image processing program that causes a computer to perform operations including: imaging the three-dimensional structure of a biological lumen based on tomographic information obtained by a sensor moving within the biological lumen to generate a three-dimensional image, and displaying the generated three-dimensional image on a display; detecting a branch lumen branching from the biological lumen based on the tomographic information or the three-dimensional image; calculating a position at which the detected branch lumen branches in the circumferential direction of the biological lumen as a branching angle relative to a reference direction, or calculating an image display direction at which the three-dimensional image is displayed on the display in the circumferential direction of the three-dimensional image as a display angle relative to the front direction of the organism in the three-dimensional image; and displaying, together with the three-dimensional image, angle information related to the calculated branching angle or display angle, and position information indicating the branching position at which the branch lumen branches from the biological lumen in the three-dimensional image on the display.

Images