Visualization system, method for controlling visualization system, visualization device. program, and medical device system

Abstract

To more appropriately visualize a contact state of a balloon with a target site in real time on the basis of image information acquired by imaging the balloon with a camera.SOLUTION: A visualization system that outputs information to be used in balloon catheter treatment on an organism includes a visualization device which outputs visualization information for visualizing a contact state of a balloon on the basis of a camera image acquired by using a camera to image a balloon of a balloon catheter in a contact state in which the balloon is inserted into an organ and pressed against an inner surface of the organ.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a visualization system, a control method for a visualization system, a visualization device, a program, and a medical equipment system. [Background technology]

[0002] In recent years, medical technology has become more sophisticated and complex, and accordingly, the skills required of physicians, for example, have also become more sophisticated and diverse.

[0003] Among such medical technologies, for example, catheter ablation therapy is a treatment method in which a target site within the body is ablated using a catheter inserted into the body. For example, by destroying the target site through ablation, diseases such as arrhythmia caused by atrial fibrillation, endometriosis, and cancer are treated. A balloon catheter having a balloon at its distal end is known as a catheter used in catheter ablation therapy.

[0004] When the balloon catheter is inserted into the body, the balloon is deflated and stretched in the longitudinal direction of the balloon catheter. Once the balloon catheter is inserted into the body, a liquid is supplied to the balloon catheter. The balloon expands as the liquid is supplied. The surface temperature of the balloon is controlled by adjusting the temperature of the liquid inside the balloon. By bringing the balloon, which has been adjusted to a predetermined surface temperature, into contact with a circumferential target region, for example, a connection site of a vein to the atrium, the circumferential target region can be ablated in one go.

[0005] In ablation treatment using such a balloon catheter, it is important to control the surface temperature of the balloon, the contact of the balloon with the target site, the ablation time, and the like.

[0006] Here, as a conventional technology for assisting users in performing medical procedures as described above, a technology has been proposed in which a medical image of a patient's blood vessels is acquired, and when this medical image is input, a trained model outputs balloon information related to the balloon to be used in the patient's intravascular treatment, and the acquired medical image is input into a computer to execute a process of outputting the balloon information (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 2021 / 193024 Summary of the Invention [Problem to be solved by the invention]

[0008] In the above-mentioned conventional technology, the user can select the balloon to be used using balloon information output from a computer that indicates the shape or type of balloon to be used for intravascular treatment.

[0009] However, in the above-mentioned conventional technology, the balloon information output from the computer indicates information for selecting the balloon to be used, and there is a problem that, for example, in balloon catheter treatment, it is not possible to support the user's medical procedure by visualizing the contact state of the balloon in real time.

[0010] The present invention is based on this finding of the inventors, and aims to visualize the contact state of a balloon in real time based on image information obtained by capturing an image of the balloon with a camera. [Means for solving the problem]

[0011] A visualization system according to an embodiment of one aspect of the present invention includes a visualization device that outputs visualization information for visualizing a contact state of a balloon, based on a camera image captured by a camera of a balloon of a balloon catheter that has been inserted into an organ of a living body and is pressed against and in contact with the inner surface of the organ, wherein the balloon has a plurality of markers arranged on it, and the camera image is an image captured by the camera of an area of ​​the balloon where the plurality of markers are arranged. The visualization device acquires a camera image captured by the camera of an area of ​​the balloon of the balloon catheter that includes at least a portion of the markers, analyzes the camera image to obtain a marker image by extracting the marker area where the plurality of markers are located, obtains deformation data from the marker image by using an estimation model for estimating the shape of the balloon from the marker image, and generates and outputs visualization information for visualizing the contact state of the balloon, thereby outputting information for use in balloon catheter treatment of a living body.

[0012] In the visualization system, the visualization device may perform a process on the camera image to remove projection distortion of the camera that captured the camera image before acquiring the marker image.

[0013] In the visualization system, the plurality of markers may be arranged on the balloon at least in an area that is to come into contact with an organ of the living body.

[0014] In the visualization system, when some of the multiple markers are missing from the camera image, the visualization device may analyze the camera image to obtain the marker image by extracting the marker region where the markers included in the camera image are located, use the estimation model to obtain deformation data that estimates deformation of the shape of the balloon from the marker image with some of the markers missing, and generate and output visualization information that visualizes the contact state of the balloon based on the deformation data.

[0015] In the visualization system, the camera is positioned on the balloon catheter so as to capture images of the inside of the balloon, and the presence of an obstruction between the camera and the markers positioned on the balloon and / or a change in the relative position of the camera on the balloon catheter may cause some of the multiple markers to be missing from the camera image captured by the camera.

[0016] In the visualization system, the balloon catheter includes the balloon, an outer cylindrical shaft connected to a base end of the balloon, and an inner cylindrical shaft connected to a tip of the balloon opposite the base end, the inner cylindrical shaft passing through the outer cylindrical shaft and extending into the balloon, and the camera is positioned in the balloon catheter so as to capture an image of the inside of the balloon from the base end side, and the shielding object may be a portion of the inner cylindrical shaft located inside the balloon.

[0017] In the visualization system, the camera may be disposed within the outer cylindrical shaft.

[0018] In the visualization system, the plurality of markers may be arranged on the balloon at least around the tip.

[0019] In the visualization system, the plurality of markers may be arranged on an inner surface of the balloon.

[0020] In the visualization system, the visualization device may include an input unit for inputting image data of the camera image, a processing unit that acquires the camera image based on the image data input by the input unit, analyzes the camera image to acquire a marker image by extracting a marker region where the multiple markers are located, acquires deformation data that estimates deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image, and generates visualization information that visualizes the contact state of the balloon based on the deformation data, and an output unit that outputs the visualization information acquired by the processing unit.

[0021] In the visualization system, the visualization device may include a storage unit that stores the estimation model in advance, and the processing unit of the visualization device may read the estimation model from the storage unit.

[0022] In the visualization system, the processing unit of the visualization device may store at least one of the deformation data, the visualization information, the deformation data, and / or information for generating the deformation data in the memory unit of the visualization device.

[0023] In the visualization system, the camera image may be an image of at least a region on the inner surface of the inside of the balloon where the plurality of markers are arranged, captured from inside the balloon.

[0024] The visualization system may further include an estimation model generation device that generates the estimation model for estimating the shape of the balloon from the marker image by machine learning using a neural network.

[0025] In the visualization system, the visualization device and the estimation model generation device may be configured as an integrated unit, and the estimation model generation device may be constructed as software in the visualization device that generates the estimation model.

[0026] In the visualization system, the estimation model generation device may use at least one of the deformed data, the visualization information, the deformed data, and / or information for generating the deformed data for machine learning using the neural network.

[0027] In the visualization system, the visualization information may include an image, text information, and / or audio information indicating the contact state of the balloon.

[0028] In the visualization system, the visualization information may include an image, text information and / or audio information indicating the shape of the balloon.

[0029] A control method for a visualization system according to an embodiment of one aspect of the present invention is a control method for a visualization system that outputs information to be used in balloon catheter treatment of a living organism, the control method including: a visualization device that outputs visualization information for visualizing a contact state of a balloon, based on a camera image captured by a camera of a balloon of a balloon catheter that has been inserted into an organ of the living organism and is pressed against and in contact with an inner surface of the organ; the balloon has a plurality of markers arranged on it, and the camera image is an image captured by the camera of an area of ​​the balloon where the plurality of markers are arranged; the visualization device acquires a camera image captured by the camera of an area of ​​the balloon of the balloon catheter that includes at least a portion of the plurality of markers; the visualization device analyzes the camera image to acquire a marker image extracting a marker area where the plurality of markers are located; the visualization device acquires deformation data that estimates deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image; and the visualization device generates and outputs visualization information that visualizes the contact state of the balloon, based on the deformation data.

[0030] A visualization device according to an embodiment of one aspect of the present invention outputs visualization information for visualizing a contact state of a balloon, based on a camera image captured by a camera of a balloon of a balloon catheter that has been inserted into an organ of a living body and is pressed against and in contact with the inner surface of the organ, wherein the balloon has a plurality of markers arranged thereon, and the camera image is an image captured by the endoscopic camera of an area of ​​the balloon where the plurality of markers are arranged. The visualization device acquires a camera image captured by the endoscopic camera of an area of ​​the balloon of the balloon catheter that includes at least a portion of the plurality of markers, analyzes the camera image to obtain a marker image by extracting a marker area where the plurality of markers are located, acquires deformation data that estimates deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image, and generates and outputs visualization information for visualizing the contact state of the balloon, based on the deformation data.

[0031] The visualization device may include an input unit for inputting image data of the camera image, a processing unit that acquires the camera image based on the image data input by the input unit, analyzes the camera image to acquire a marker image by extracting a marker region where the multiple markers are located, acquires deformation data that estimates deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image, and generates visualization information that visualizes the contact state of the balloon based on the deformation data, and an output unit that outputs the visualization information acquired by the processing unit.

[0032] A program according to an embodiment of one aspect of the present invention is a program executed by a visualization device including a computer and configured to output visualization information for visualizing a contact state of a balloon of a balloon catheter inserted into an organ of a living body and pressed against and in contact with the inner surface of the organ, based on a camera image captured by a camera of the balloon, wherein the balloon has a plurality of markers arranged thereon, and the camera image is an image captured by the camera of an area of ​​the balloon where the plurality of markers are arranged. The visualization device acquires the camera image captured by the camera of an area of ​​the balloon of the balloon catheter that includes at least a portion of the markers, analyzes the camera image to obtain a marker image by extracting the marker area where the plurality of markers are located, obtains deformation data from the marker image by using an estimation model for estimating the shape of the balloon from the marker image, and generates and outputs visualization information for visualizing the contact state of the balloon based on the deformation data.

[0033] A medical device system according to an embodiment of one aspect of the present invention is a medical device system used for balloon catheter treatment on a living organism, the medical device system performing the balloon catheter treatment comprising: a balloon catheter including a balloon and a camera that images the balloon; and a visualization system that outputs information to be used in the balloon catheter treatment on the living organism based on a camera image captured by the camera, the visualization system outputting visualization information for visualizing a contact state of the balloon based on a camera image captured by the camera of the balloon of the balloon catheter inserted into an organ of the living organism and pressed against and in contact with an inner surface of the organ. a visualization device that outputs a visualization image of a region of the balloon where the markers are arranged, the visualization device having a plurality of markers arranged on the balloon, the visualization device acquiring a camera image of a region of the balloon of the balloon catheter that includes at least a portion of the markers, analyzing the camera image to acquire a marker image by extracting a marker region where the markers are located, acquiring deformation data that estimates deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image, and generating and outputting visualization information that visualizes the contact state of the balloon based on the deformation data.

[0034] The medical device system may further include an estimation model generation device that generates the estimation model for estimating the shape of the balloon from the marker image by machine learning using a neural network.

[0035] In the medical equipment system, the visualization device and the estimation model generation device may be configured as an integrated unit, and the estimation model generation device may be constructed as software in the visualization device that generates the estimation model.

[0036] In the medical device system, when some of the multiple markers are missing from the camera image, the visualization device may analyze the camera image to obtain the marker image by extracting the marker area where the markers included in the camera image are located, use the estimation model to obtain deformation data that estimates the deformation of the shape of the balloon 25 from the marker image with some of the markers missing, and generate and output visualization information that visualizes the contact state of the balloon based on the deformation data.

[0037] In the medical device system, the camera is positioned on the balloon catheter so as to capture images of the inside of the balloon, and if an obstruction exists between the camera and the markers positioned on the balloon and / or the relative position of the camera on the balloon catheter changes, some of the multiple markers may be missing from the camera image captured by the camera.

[0038] In the medical device system, the balloon catheter includes the balloon, an outer cylindrical shaft connected to a base end of the balloon, and an inner cylindrical shaft connected to a tip of the balloon opposite the base end, the inner cylindrical shaft passing through the outer cylindrical shaft and extending into the balloon, and the camera is positioned in the balloon catheter so as to capture an image of the inside of the balloon from the base end side, and the shielding object may be a portion of the inner cylindrical shaft located inside the balloon.

[0039] In the medical device system, the camera may be disposed within the outer cylindrical shaft of the balloon catheter. [Effects of the Invention]

[0040] According to the present invention, the contact state of the balloon can be visualized in real time. [Brief explanation of the drawings]

[0041] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a medical device system including a visualization system including a visualization device and an estimation model generation device according to an embodiment, and a balloon catheter system applied to balloon catheter treatment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the visualization device illustrated in FIG. [Figure 3] FIG. 3 is a diagram showing an example of the configuration of the balloon catheter system shown in FIG. [Figure 4A] FIG. 4A is a diagram showing an example of the configuration of the vicinity of the balloon of the balloon catheter in an inflated state in region Z of FIG. [Figure 4B] FIG. 4B is a diagram showing an example of the configuration of the vicinity of the balloon of the balloon catheter in a deflated state in region Z of FIG. [Figure 5] FIG. 5 is a cross-sectional view taken along line IV-IV in FIG. 5A. [Figure 6A] FIG. 6A is a diagram showing an example of a configuration focusing on a balloon on which a plurality of markers are arranged of a balloon catheter. [Figure 6B] FIG. 6B is a diagram showing another example of a configuration focusing on a balloon on which a plurality of markers are arranged of a balloon catheter. [Figure 7] FIG. 7 is a diagram for explaining an example of a learning stage of the control method of the visualization system shown in FIG. [Figure 8A] FIG. 8A is a diagram showing an example of a bird's-eye view of a three-dimensional configuration of a virtual balloon that constitutes a tactile sensor constructed in a simulation environment. [Figure 8B] FIG. 8B is a diagram showing an example of a bird's-eye view of a three-dimensional configuration of a plurality of markers arranged in a virtual balloon that constitutes a tactile sensor constructed in a simulation environment. [Figure 9] FIG. 9 is a diagram showing an example of a simulated camera image capturing marker regions of a plurality of markers placed on a three-dimensional virtual balloon constructed in a simulation environment. [Figure 10]FIG. 10 is a diagram showing an example of a first data set of camera images of virtual balloon markers collected by simulation and deformed mesh data corresponding to the camera images. [Figure 11] FIG. 11 is a diagram showing an example of a second data set obtained by multiplying the first data set shown in FIG. 10 by changing the camera orientation in the simulation environment. [Figure 12] FIG. 12 is a diagram illustrating a process of training an estimation model to obtain a trained estimation model. [Figure 13] FIG. 13 is a flow diagram showing an example of the visualization stage of the control method for the visualization system (visualization device) shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0042] An embodiment of the present disclosure will be described below with reference to the drawings. In the drawings accompanying this specification, the scale and aspect ratios have been appropriately changed and exaggerated from those of the actual objects for ease of understanding. Configurations shown in some drawings may be omitted in other drawings. The scale and aspect ratios may differ between the drawings.

[0043] In this specification, terms that specify shapes, geometric conditions, and their degrees, such as "parallel," "orthogonal," and "identical," as well as values ​​of lengths and angles, are not limited to their strict meanings, but are interpreted to include a range within which similar functions can be expected.

[0044] In the following embodiments, a medical device system for use in assisting a user in a medical procedure will be described.

[0045] The present invention also relates to a medical device system, particularly a visualization system, which is applied to a balloon catheter inserted into an organ of a living body (e.g., a human, an animal, etc.) and pressed against the inner surface of the organ. As an example of this balloon catheter, a balloon catheter used in balloon catheter treatment for arrhythmia such as atrial fibrillation will be described, but the invention can also be applied to balloon catheters used to treat other conditions such as endometriosis and cancer.

[0046] Here, Fig. 1 is a diagram showing an example of the configuration of a medical device system including a visualization system including a visualization device and an estimation model generation device according to an embodiment, and a balloon catheter system applied to balloon catheter treatment. Also, Fig. 2 is a diagram showing an example of the configuration of the visualization device shown in Fig. 1.

[0047] [Medical Device Systems] For example, as shown in FIG. 1, a medical device system 1000 according to this embodiment is a system used for balloon catheter treatment of a living body.

[0048] As shown in FIG. 1, for example, this medical device system 1000 is a balloon catheter system 10 for performing balloon catheter treatment, and includes the balloon catheter system 10 equipped with a balloon catheter 15 including a balloon 25 and a camera 55 for capturing images of the balloon 25, and a visualization system 300 that outputs information for use in balloon catheter treatment of the living body based on the camera image captured by the camera 55.

[0049] In addition, in the balloon catheter system 10 applied to this medical device system 1000, for example, as shown in FIG. 1, the balloon catheter 15 includes a balloon 25 on which a plurality of markers P are arranged, an outer cylindrical shaft 30 connected to the base end 25b of the balloon 25, and an inner cylindrical shaft 35 connected to the tip 25a opposite the base end 25b of the balloon 25, the inner cylindrical shaft 35 passing through the outer cylindrical shaft 30 and extending into the balloon 25.

[0050] In this embodiment, the camera 55 is arranged, for example, in the balloon catheter 15, so as to capture an image of the inside of the balloon 25 from the proximal end 25b side. This camera 55 is configured to output a camera image of the balloon 25. Note that, in the example of this embodiment, the camera 55 is described as a camera that acquires a camera image, but this is not limited to this, and any camera having a similar function that can capture the camera image can be applied.

[0051] An example of a more specific configuration of this balloon catheter system 10 will be described later.

[0052] [Visualization System] A visualization system 300 according to this embodiment is configured to output information for use in balloon catheter treatment of a living body. The visualization system 300 includes, for example, a visualization device 100 and an estimation model generation device 200, as shown in FIG.

[0053] 1, the visualization device 100 and the estimation model generation device 200 are depicted as separate components, but the visualization device 100 and the estimation model generation device 200 may be integrated into one device. In this case, the estimation model generation device 200 may be configured as software that generates the estimation model in the visualization device 100, for example.

[0054] [Estimation model generator] The estimation model generating device 200 generates an estimation model for estimating the shape of the balloon 25 from a marker image based on a camera image of the balloon 25, for example, as shown in FIG. 1, by machine learning using a neural network.

[0055] In this embodiment, the estimation model generation device 200 generates an estimation model using image data including an area of ​​a plurality of markers placed on a virtual balloon constructed in a simulation environment and the shape of the virtual balloon as training data.

[0056] Here, for example, there is a certain relationship between the contact state of the balloon 25 with the inner surface (target site) of an organ of a living body inserted into the organ and the shape of the balloon 25. Furthermore, when the shape of the balloon 25 changes, the arrangement of the multiple markers P arranged on the balloon 25 also changes. Therefore, it is considered possible to estimate the shape of the balloon 25 from a camera image capturing the multiple markers P. Then, based on information about the estimated shape of the balloon 25, it is also considered possible to visualize the contact state of the balloon 25 as described above.

[0057] Based on these considerations, it can be said that there is a certain correlation between a camera image capturing a plurality of markers P placed on a balloon 25 and the shape of the balloon 25. From this, it can be considered that, for example, as described below, in a simulation environment, it is possible to generate an estimation model that estimates the shape of a balloon from a camera image by machine learning using a known machine learning algorithm such as a neural network, using image data of a plurality of markers based on a camera image capturing a plurality of markers of a virtual balloon and the shape of the virtual balloon (mesh shape data) as training data.

[0058] In this embodiment, the contact state of the balloon with the inner surface of the organ is evaluated and visualized based on the shape of the balloon estimated using the estimation model from a camera image capturing multiple markers on the balloon that is in contact with the inner surface of the organ.

[0059] [Visualization device] Furthermore, as described above, the visualization device 100 is configured to output visualization information for visualizing the contact state of the balloon 25 using the estimation model generated by the estimation model generation device 200 based on a camera image captured by the camera 55 of the balloon 25 of the balloon catheter 15 inserted into an organ (visceral organ) of the living body and pressed against and in contact with the inner surface of the organ.

[0060] 1, a plurality of markers P are arranged on the balloon 25, as will be described later. The camera image is an image captured by the camera 55 of the area of ​​the balloon 25 where the plurality of markers P are arranged.

[0061] In particular, the multiple markers P are arranged in at least an area of ​​the balloon 25 that is to come into contact with the organ of the living body. This allows the markers P, which are deformed when the balloon 25 comes into contact with the organ, to be set to displace. That is, by capturing the displacement of the markers P, it is possible to more appropriately estimate the change in the shape of the balloon 25 and visualize the contact state of the balloon 25 with the organ.

[0062] More specifically, for example, the visualization device 100 is configured to read the estimation model generated by the estimation model generation device 200.

[0063] The visualization device 100 is configured to acquire a camera image obtained by the camera 55 capturing an area including at least some of the markers P of the balloon 25 of the balloon catheter 15.

[0064] The visualization device 100 then analyzes the camera image and acquires a marker image by extracting a marker region where a plurality of markers P are located. In particular, before acquiring the marker image, the visualization device 100 performs a process on the camera image to remove projection distortion of the camera 55 that captured the camera image.

[0065] The visualization device 100 then uses an estimation model to estimate the shape of the balloon 25 from the marker image based on the camera image, and obtains deformation data that estimates the deformation of the shape of the balloon 25 from the marker image.

[0066] Then, the visualization device 100 generates and outputs visualization information that visualizes the contact state of the balloon 25 based on the deformation data.

[0067] Note that this visualization information may include, for example, text information and / or audio information in addition to or instead of an image (e.g., mesh data) showing the contact state of the balloon 25. Furthermore, this visualization information may include an image (e.g., mesh data) showing the shape of the balloon 25, and text information and / or audio information showing the shape of the balloon 25.

[0068] As described above, in this embodiment, the camera 55 is disposed in the balloon catheter 15 so as to capture an image of the interior of the balloon 25 from the proximal end 25b. That is, the camera image is an image of an area on the inner surface of the balloon 25 where at least a plurality of markers P are disposed, captured from inside the balloon 25. An obstruction may exist between the camera 55 and some of the plurality of markers P disposed on the balloon 25. In this case, for example, the portion of the inner cylindrical shaft 35 located inside the balloon 25 may be the obstruction.

[0069] For example, if an obstruction exists between the camera 55 and the marker P placed on the balloon 25 and / or the relative position of the camera 55 on the balloon catheter changes, some of the multiple markers may be missing from the camera image captured by the camera 55.

[0070] However, even in such a case where some of the markers P are missing from the camera image, the visualization device 100 analyzes the camera image and acquires a marker image by extracting a marker region where the markers P included in the camera image are located. The visualization device 100 then uses an estimation model to acquire deformation data that estimates the deformation of the shape of the balloon 25 from the marker image where some of the markers P are missing. The visualization device 100 then generates and outputs visualization information that visualizes the contact state of the balloon 25 based on the deformation data.

[0071] Here, the visualization device 100 includes an input unit IN, a processing unit Y, an output unit D, and a storage unit M, as shown in FIG.

[0072] An input unit IN of the visualization device 100 is configured to receive image data of a camera image acquired by the camera 55 of the balloon catheter system 10. The input unit IN also receives data of an estimation model generated by the estimation model generation device 200. If the estimation model generation device 200 is integrated with the visualization device 100, the estimation model will be generated by a processing unit Y of the visualization device 100.

[0073] In addition, the processing unit Y of the visualization device 100 acquires a camera image based on image data input by the input unit IN, analyzes the camera image to acquire (calculate) a marker image by extracting a marker area where multiple markers P are located, acquires (calculates) deformation data that estimates the deformation of the shape of the balloon 25 from the marker image using an estimation model for estimating the shape of the balloon 25, and generates visualization information that visualizes the contact state of the balloon 25 based on the deformation data.

[0074] In addition, the processing unit Y of the visualization device 100 is configured to store at least one of the acquired deformation data, visualization information, deformation data, and / or information for generating the deformation data in the memory unit M of the visualization device 100.

[0075] The estimation model generating device 200 may use at least one of the deformation data, visualization information, deformation data, and / or information for generating the deformation data in machine learning using a neural network to generate the estimation model.

[0076] As described above, the visualization device 100 and the estimation model generation device 200 may be integrated. In this case, the estimation model generation device 200 may be implemented as software that generates the estimation model in the processing unit Y of the visualization device 100 shown in FIG.

[0077] The output unit D outputs the visualization information acquired by the processing unit. The output unit D is, for example, a display device that displays the visualization information externally.

[0078] Note that, when the output unit D is a display device, the visualization information is, for example, an image (e.g., mesh data) showing the contact state of the balloon 25. However, as described above, the visualization information may include, for example, text information and / or audio information in addition to or instead of the image (e.g., mesh data) showing the contact state of the balloon 25. Furthermore, the visualization information may include an image (e.g., mesh data) showing the shape of the balloon 25, and text information and / or audio information showing the shape of the balloon 25.

[0079] The storage unit M also stores a program for processing executed by the processing unit Y of the visualization device 100, an estimation model generated by the estimation model generation device 200, and data (image data, etc.) required for the processing. The storage unit M is configured to temporarily store the transformed data generated by the processing unit Y, visualization information, the transformed data, and / or information for generating the transformed data. The storage unit M is, for example, a non-volatile memory (readable storage medium) such as a NAND flash memory.

[0080] The above-mentioned program is a program executed by the visualization device 100 that includes at least a computer (processing unit Y), and causes the computer (processing unit Y) to execute the following process: acquire a camera image based on image data input by the input unit IN; analyze the camera image to acquire (calculate) a marker image that extracts a marker area where multiple markers P are located; acquire (calculate) deformation data that estimates the deformation of the shape of the balloon 25 from the marker image using an estimation model for estimating the shape of the balloon 25; and generate visualization information that visualizes the contact state of the balloon 25 based on the deformation data.

[0081] [Balloon catheter system] Here, a specific configuration of the balloon catheter system described above will be described. FIG. 3 is a diagram showing an example of the configuration of the balloon catheter system 10 shown in FIG. 1. FIG. 4A is a diagram showing an example of the configuration of the vicinity of the balloon of a balloon catheter in an inflated state, in region Z of FIG. 3. FIG. 4B is a diagram showing an example of the configuration of the vicinity of the balloon of a balloon catheter in a deflated state, in region Z of FIG. 3. FIG. 5 is a cross-sectional view taken along line IV-IV of FIG. 5A. FIG. 6A is a diagram showing an example of the configuration focusing on a balloon of a balloon catheter on which multiple markers are arranged. FIG. 6B is a diagram showing another example of the configuration focusing on a balloon of a balloon catheter on which multiple markers are arranged. Note that the multiple markers P of the balloon 25 are not shown in FIGS. 3, 4A, and 4B.

[0082] 3, 4A, and 4B, a balloon catheter system 10 that is applied to the visualization system 300 and is used for balloon catheter treatment of a living body will be described.

[0083] 3, the balloon catheter system 10 includes a balloon catheter 15, a control device 70, and a stirring device 75 connected to the balloon catheter 15. The balloon catheter 15 also includes a catheter body 20 having a longitudinal direction LD, and a handle 50 connected to the proximal end of the catheter body 20.

[0084] 4A, the catheter main body 20 has a balloon 25, an outer cylindrical shaft 30 connected to the proximal end 25b of the balloon 25, an inner cylindrical shaft 35 connected to the distal end 25a of the balloon 25, and a heating member 40 disposed within the balloon 25. The inner cylindrical shaft 35 passes through the outer cylindrical shaft 30 and extends into the balloon 25. A liquid delivery path LP that communicates with the inside of the balloon 25 is formed between the outer cylindrical shaft 30 and the inner cylindrical shaft 35. The heating member 40 heats the liquid within the balloon 25.

[0085] The longitudinal direction LD of the catheter main body 20 is specified as the direction in which the central axes of the outer cylindrical shaft 30 and the inner cylindrical shaft 35 extending from the outer cylindrical shaft 30 extend. In this specification, the "distal" side used with respect to each component of the balloon catheter 15 and the catheter main body 20 means the side away from the operator (surgeon) of the handle 50 and the balloon catheter 15 along the longitudinal direction LD of the catheter main body 20, or in other words, the tip side. In addition, the "proximal" side used with respect to each component of the balloon catheter 15 and the catheter main body 20 means the side close to the operator (surgeon) of the handle 50 and the balloon catheter 15 along the longitudinal direction LD of the catheter main body 20, or in other words, the base end side.

[0086] The balloon catheter system 10 and the balloon catheter 15 will be described in further detail below. First, the catheter body 20 of the balloon catheter 15 will be described in detail. As described above, the catheter body 20 of the balloon catheter 15 includes the balloon 25, the outer cylindrical shaft 30, the inner cylindrical shaft 35, the heating element 40, the temperature sensor 45, and the camera 55.

[0087] Of these, the outer cylindrical shaft 30 and the inner cylindrical shaft 35 are both configured in a tubular, typically cylindrical, shape. Therefore, the outer cylindrical shaft 30 and the inner cylindrical shaft 35 each form a lumen as an internal space. For example, a guide wire (not shown) is inserted into the lumen formed by the inner cylindrical shaft 35. The inner cylindrical shaft 35 is inserted into the lumen formed by the outer cylindrical shaft 30. That is, the outer cylindrical shaft 30 and the inner cylindrical shaft 35 have a double-tube shaft configuration. The inner diameter of the outer cylindrical shaft 30 is larger than the outer diameter of the inner cylindrical shaft 35. Therefore, a lumen remains between the outer cylindrical shaft 30 and the inner cylindrical shaft 35. This lumen between the outer cylindrical shaft 30 and the inner cylindrical shaft 35 forms a liquid delivery path LP. As shown in FIG. 4A , the liquid delivery path LP is connected to the balloon 25. The liquid delivery path LP also extends into the handle 50.

[0088] A balloon 25 is connected to the outer cylindrical shaft 30 and the inner cylindrical shaft 35. The balloon 25 is formed so as to be expandable by filling it with a liquid and contractable by discharging the liquid. The balloon 25 preferably has a shape that can fit the target site (e.g., a blood vessel) to be treated. As an example, a spherical shape with a diameter of 15 mm to 40 mm can be adopted as the shape of the balloon 25 that fits the pulmonary vein junction of the left atrium. Here, the spherical shape includes a perfect sphere, an oblate spheroid, and an elongated spheroid, and also includes a nearly spherical shape.

[0089] 4A and 4B, in the illustrated catheter main body 20, the distal end (tip) 25a of the balloon 25 is fixed to the distal end (tip) 35a of the inner cylindrical shaft 35. The proximal end (base end) 25b of the balloon 25 is fixed to the distal end (tip) 30a of the outer cylindrical shaft 30. The balloon 25 can be connected to the outer cylindrical shaft 30 and the inner cylindrical shaft 35 by bonding or thermal welding.

[0090] 6A and 6B, multiple markers P are arranged on the balloon 25. In the example of FIGS. 6A and 6B, the external appearance of the balloon 25 is shown, and the multiple markers P are visible from the outside of the balloon 25, but it is sufficient that at least the multiple markers P are arranged on the inner surface of the balloon 25 so that they can be imaged by the camera 55. In this case, as the shape of the balloon 25 changes, it is possible to image the change in position of the multiple markers P by the camera 55.

[0091] As described above, the markers P are arranged at least in the area of ​​the balloon 25 that is to come into contact with the organ of the living body. For example, in the examples of Fig. 6A and Fig. 6B, the markers P are arranged at least around the tip 25a of the balloon 25. However, the markers P may be arranged over the entire balloon 25.

[0092] Furthermore, the multiple markers P are arranged on the inner surface of the balloon 25 so that they can be imaged by the camera 55. However, the multiple markers P may be arranged on the outside of the balloon 25 as long as they can be imaged by the camera 55, for example, when the balloon 25 is transparent.

[0093] By arranging the multiple markers P in this way, it is possible to set the markers P to be displaced when the balloon 25 comes into contact with an organ. That is, by capturing the displacement of the markers P, it is possible to more appropriately estimate the change in the shape of the balloon 25 and visualize the contact state of the balloon 25 with the organ.

[0094] As shown in FIGS. 4A and 4B , the balloon 25 connected to the outer and inner cylindrical shafts 30 and 35 deforms as the outer and inner cylindrical shafts 30 and 35 move relative to each other in the longitudinal direction LD. In the illustrated example, the relative movement of the outer and inner cylindrical shafts 30 and 35 adjusts the size of the balloon 25 in the longitudinal direction LD. As shown in FIG. 4B , the inner cylindrical shaft 35 moves distally relative to the outer cylindrical shaft 30 in the longitudinal direction LD, stretching the balloon 25 in the longitudinal direction LD and placing it in a more tense state. In the illustrated example, the distal movement range of the inner cylindrical shaft 35 relative to the outer cylindrical shaft 30 in the longitudinal direction LD is restricted by the balloon 25. As the inner cylindrical shaft 35 moves proximally relative to the outer cylindrical shaft 30 from the state shown in FIG. 4B , the balloon 25 becomes relaxed. The relaxed balloon 25 can be inflated by introducing a liquid into the balloon 25, as shown in FIG. 4A . That is, the size of the balloon 25 in the longitudinal direction LD can be adjusted by moving the outer cylindrical shaft 30 and the inner cylindrical shaft 35 relative to each other.

[0095] Furthermore, the heating element 40 is disposed within the balloon 25. The heating element 40 is a member for heating the liquid filled within the balloon 25. As an example, a nichrome wire that generates heat through electrical resistance can be used as the heating element 40. As another example of the heating element 40, a coil electrode 41 can be used, as shown in FIGS. 4A and 4B. By applying high-frequency current to the heating element 40 serving as the coil electrode 41, a high-frequency current flows between the coil electrode 41 and an externally disposed counter electrode 77 (FIG. 3), and the liquid located between the coil electrode 41 and the counter electrode 77 generates Joule heat. The counter electrode 77 is disposed, for example, on the back of the patient.

[0096] 4A and 4B, the coil electrode 41 is provided on the inner cylindrical shaft 35 extending inside the balloon 25. The coil electrode 41 may be formed by a conductive wire wound around the inner cylindrical shaft 35. The coil electrode 41 is electrically connected to a wiring 42 for high-frequency current application. The wiring 42 extends to the handle 50 through the liquid transfer path LP, which serves as a lumen between the outer cylindrical shaft 30 and the inner cylindrical shaft 35.

[0097] The diameter of the coil electrode 41 and the wiring 42 may be 0.1 mm or more and 1 mm or less, or 0.1 mm or more and 0.4 mm or less. Examples of conductive materials that may be used to form the coil electrode 41 and the wiring 42 include copper, silver, gold, platinum, and alloys thereof. As shown in FIG. 5, the wiring 42 may include a conductive linear portion such as a metal wire and an insulating coating that covers the conductive linear portion. The insulating coating can prevent short circuits in the wiring 42. Examples of materials that may be used to form the insulating coating include fluoropolymers.

[0098] The temperature sensor 45 acquires information about the temperature of the liquid. The temperature sensor 45 has a heat-sensing part 46 disposed in the liquid transfer path LP located between the outer cylindrical shaft 30 and the inner cylindrical shaft 35.

[0099] For the purpose of determining the surface temperature of the balloon 25 with high accuracy, the preferred length DX along the longitudinal direction LD of the outer tubular shaft from the distal end 30a of the outer tubular shaft 30 to the heat-sensing portion 46 of the temperature sensor 45 strictly depends on the amount of liquid supplied and discharged by the stirring device 75 described below.

[0100] 4A and 4B, the temperature sensor 45 typically has a heat-sensing part 46 disposed in the liquid feed path LP and a lead wire 47 electrically connected to the heat-sensing part 46. In a temperature sensor 45 that is a thermocouple, the part where dissimilar metals are connected forms the heat-sensing part 46. In a temperature sensor 45 that is a thermistor, a ceramic element forms the heat-sensing part 46. The lead wire 47 extends to the handle 50 through the liquid feed path LP, which is a lumen between the outer cylindrical shaft 30 and the inner cylindrical shaft 35.

[0101] In the illustrated example, the temperature sensor 45 is attached to the inner cylindrical shaft 35. As shown in Figures 4A and 4B, the lead wires 47 of the temperature sensor 45 are fixed, thereby attaching the temperature sensor 45 to the inner cylindrical shaft 35. The heat-sensing part 46 is spaced apart from both the outer cylindrical shaft 30 and the inner cylindrical shaft 35.

[0102] The camera 55 is also configured to capture an image of the balloon 25 of the balloon catheter 15. In particular, in this embodiment, for example, as shown in FIGS. 4A and 4B , the camera 55 is disposed on the balloon catheter 15 so as to capture an image of the interior of the balloon 25. The camera 55 is disposed on the balloon catheter 15 so as to capture an image of the interior of the balloon 25 from the proximal end 25b side. An obstruction may exist between the camera 55 and some of the multiple markers P disposed on the balloon 25. In this case, for example, the portion of the inner cylindrical shaft 35 located inside the balloon 25 may be the obstruction.

[0103] However, as described above, even when some of the markers P are missing from the camera image, the visualization device 100 analyzes the camera image and acquires a marker image by extracting a marker region where the markers P included in the camera image are located. The visualization device 100 then uses an estimation model to acquire deformation data that estimates the deformation of the shape of the balloon 25 from the marker image in which some of the markers P are missing. The visualization device 100 then generates and outputs visualization information that visualizes the contact state of the balloon 25 based on the deformation data.

[0104] The handle 50 is a part that is held by the user (operator) while using the balloon catheter system 10, as shown in FIG.

[0105] The handle 50 shown in Fig. 3 has a first handle section 51 and a second handle section 52 that are slidable relative to each other. The first handle section (front handle section) 51 is connected to the outer cylindrical shaft 30 of the catheter main body 20. The second handle section (rear handle section) 52 is connected to the inner cylindrical shaft 35 of the catheter main body 20. By moving the second handle section 52 relative to the first handle section 51, the inner cylindrical shaft 35 can be moved relative to the outer cylindrical shaft 30.

[0106] As shown in FIG. 3, the handle 50 also serves as a connection point between the balloon catheter 15 and other devices included in the balloon catheter system 10.

[0107] First, a connector 56 extends from the second handle portion 52. This connector 56 electrically connects the wiring 42 of the catheter main body 20 and the lead wire 47 of the temperature sensor 45 to the external control device 70. The connector 56 extends from one of multiple branch portions 52a provided on the second handle portion 52.

[0108] The second handle portion 52 has branches 52b and 52c in addition to the branch 52a to which the connector 56 is connected. These branches 52b and 52c function as portions for supplying liquid to the lumen, which is the internal space of the inner cylindrical shaft 35, and as portions from which a guidewire inserted into the lumen of the inner cylindrical shaft 35 extends. During cardiac ablation treatment, a small amount of saline solution, approximately 100 ml per hour, is typically discharged into the body through the lumen of the inner cylindrical shaft 35. Discharging the saline solution effectively prevents blood from flowing back into the lumen of the inner cylindrical shaft 35.

[0109] As shown in Fig. 3, an extension tube 57 extends from the first handle portion 51. This extension tube 57 connects the liquid feed path LP of the catheter main body 20 to an external supply device 74 or agitator 75. The extension tube 57 extends from a branch portion 51a provided in the first handle portion 51. The extension tube 57 is connected to the supply device 74 and the agitator 75 via a valve 58. In the illustrated example, by operating the valve 58, it is possible to select whether the liquid feed path LP is connected to either the supply device 74 or the agitator 75. A three-way stopcock can be used as the valve 58.

[0110] Next, the devices that constitute the balloon catheter system 10 together with the balloon catheter 15 described above, specifically the control device 70, the supply device 74, and the stirring device 75, will be described.

[0111] The illustrated control device 70 is electrically connected to the coil electrode 41 via wiring 42. The control device 70 has a high-frequency current control unit 70A that controls the application of high-frequency current to the coil electrode 41. In the illustrated example, the output from the heating member 40 is adjusted by controlling the application of high-frequency current to the coil electrode 41 with the high-frequency current control unit 70A. The high-frequency current control unit 70A can control the application of high-frequency current to the coil electrode 41 based on the surface temperature of the balloon 25 determined by a temperature calculation unit 70B (described later), or in accordance with preset processing, or in accordance with input from the user.

[0112] The control device 70 is also electrically connected to the lead wire 47 of the temperature sensor 45. The control device 70 has a temperature calculation unit 70B that calculates information related to the temperature acquired by the inner cylindrical shaft 35. The temperature calculation unit 70B calculates the liquid temperature in the liquid transfer path LP based on the information related to the temperature acquired by the temperature sensor 45, and further estimates the surface temperature of the balloon 25 based on the calculated liquid temperature. The temperature calculation unit 70B may display the identified surface temperature of the balloon 25 on the display unit 71.

[0113] Furthermore, the control device 70 has an agitator control section 70C that controls the agitator 75. The agitator control section 70C may be configured to display the control conditions of the agitator 75 on the display section 71.

[0114] The supply device 74 also supplies liquid into the liquid transfer path LP. By supplying liquid from the supply device 74 to the balloon 25 via the liquid transfer path LP, the balloon 25 can be inflated as shown in FIG. 4A. On the other hand, by discharging liquid from the balloon 25 via the liquid transfer path LP from the supply device 74, the balloon 25 can be deflated. The liquid supplied into the liquid transfer path LP can typically be physiological saline. As shown in the figure, a syringe can be used as the supply device 74. However, a pump or the like can also be used as the supply device 74.

[0115] The agitator 75 is provided to agitate the liquid in the balloon 25. By agitating the liquid in the balloon 25, the heat supplied to the inside of the balloon 25 can be dispersed or uniformized, thereby adjusting the surface temperature of the balloon 25. The agitator 75 repeatedly supplies liquid to the liquid feed path LP and discharges liquid from the liquid feed path LP.

[0116] [Visualization system control method] Here, a control method for the visualization system according to this embodiment having the above-described configuration and functions will be described.

[0117] [Learning stage] First, an example of a learning stage of the control method of the visualization system 300 shown in Fig. 1 will be described, particularly as the operation of the estimation model generation device 200. Fig. 7 is a diagram for explaining an example of a learning stage of the control method of the visualization system shown in Fig. 1.

[0118] First, for example, as shown in FIG. 7, the estimation model generating device 200 creates a three-dimensional model to be used in a simulation for generating an estimation model (for example, a flexible body simulation environment using Nvidia PhysX5 as a physics engine) (step S1 in FIG. 7).

[0119] Next, the estimation model generating device 200 constructs a tactile sensor in the simulation environment and determines constraint conditions and physical constants of the virtual balloon, which is a flexible part that deforms (step S2 in FIG. 7).

[0120] Here, Fig. 8A is a diagram showing an example of a bird's-eye view of the three-dimensional configuration of a virtual balloon that constitutes a tactile sensor constructed in a simulation environment. Fig. 8B is a diagram showing an example of a bird's-eye view of the three-dimensional configuration of multiple markers placed on a virtual balloon that constitutes a tactile sensor constructed in a simulation environment. Fig. 9 is a diagram showing an example of a simulated camera image capturing the marker areas of multiple markers placed on a three-dimensional virtual balloon constructed in a simulation environment.

[0121] For example, the "constraint conditions" for the virtual balloon shown in Figure 8A are the fixation of the spatial coordinates of the distal end 25a and the proximal end 25b. Furthermore, because the balloon in the simulation is a virtual balloon, the conditions are set based on the physical constants of a specific balloon. Specifically, the virtual balloon is set with a predetermined Young's modulus and Poisson's ratio as conditions.

[0122] A plurality of markers P as shown in Fig. 8B are placed on the virtual balloon as shown in Fig. 8A. Then, a camera image of the marker regions of the plurality of markers placed on the three-dimensional virtual balloon constructed in the simulation environment is acquired, for example, from the base end 25b side (Fig. 9).

[0123] Next, the estimation model generating device 200 collects data pairs of deformation data of the virtual balloon relating to various contact situations and camera images corresponding to the shapes of the deformed virtual balloons (step S3 in FIG. 7).

[0124] 10 is a diagram showing an example of a first data set DX of camera images of virtual balloon markers collected by simulation and deformed mesh data corresponding to these camera images. As shown in FIG. 10, the estimation model generating device 200 acquires, in a simulation, a first data set DX of camera images of virtual balloon markers (marker images) and deformed mesh data corresponding to these camera images.

[0125] For example, a specific example of an input condition for deforming a virtual balloon is defined by the size of a sphere that is to be brought into contact with the virtual balloon and the coordinates of that sphere. Therefore, the input condition does not define pressure, but defines the direction of deformation based on the coordinate axes.

[0126] Next, the estimation model generating device 200 performs processing such as shaping the collected data to construct a data set (step S4 in FIG. 7).

[0127] Here, Fig. 11 is a diagram showing an example of a second data set that is augmented by changing the orientation of the camera in the simulation environment with respect to the first data set shown in Fig. 10. For example, as shown in Fig. 11, the estimation model generation device 200 acquires the second data set DY by changing the orientation of the camera in the simulation environment with respect to the first data set DX.

[0128] Next, the estimation model generating device 200 trains an estimation model that estimates the deformation of the balloon based on the camera image (step S5 in FIG. 7).

[0129] 12 is a diagram illustrating a process of training an estimation model to obtain a trained estimation model. For example, as shown in FIG. 12, the estimation model generation device 200 uses, as training data, image data (training data) of multiple markers based on camera images capturing multiple markers of a virtual balloon and the shape of the virtual balloon (mesh data) as training data in a simulation environment, and generates a trained estimation model that estimates the shape of a balloon from camera images by machine learning using a neural network so as to minimize an error by comparing the estimation result (mesh data) with the training data.

[0130] Next, the estimation model generation device 200 stores the parameters of the trained estimation model, the learning curve, etc. (Step S6 in FIG. 7). Then, this trained estimation model is stored in, for example, the storage unit M of the visualization device 100.

[0131] Through the above flow, the estimation model generating device 200 generates an estimation model for estimating the shape of the balloon 25 from a marker image based on a camera image capturing the balloon 25 through machine learning, and stores the model in the visualization device 100.

[0132] [Visualization stage] Next, an example of the visualization stage of the control method for the visualization system 300 shown in Fig. 1 will be described, particularly as the operation of the visualization device 100. Fig. 13 is a flow diagram showing an example of the control method for the visualization system (visualization device) shown in Fig. 1. Note that the processing shown in Fig. 13 is mainly performed by the processing unit Y of the visualization device 100 executing a program that has been prepared in advance and stored in the memory unit M.

[0133] First, for example, in response to a user operation, the processing unit Y of the visualization device 100 reads the estimation model trained by the estimation model generation device 200 in the learning stage from the storage unit M of the visualization device 100 (step S11 in FIG. 13).

[0134] Next, the processing unit Y of the visualization device 100 acquires a camera image captured by the camera 55 of an area including at least some of the markers on the balloon 25 of the balloon catheter (step S12 in FIG. 13).

[0135] Next, before acquiring a marker image from the camera image, the processing unit Y of the visualization device 100 performs processing on the camera image to remove projection distortion of the camera 55 that captured the camera image (step S13 in FIG. 13).

[0136] Next, the processing unit Y of the visualization device 100 analyzes the camera image and acquires a marker image in which a marker region where a plurality of markers P are located is extracted (step S14 in FIG. 13).

[0137] In step S14, the processing unit Y of the visualization device 100 may acquire a marker image by extracting a marker area where multiple markers P are located from the camera image, using a model that has learned to extract markers of a certain color by machine learning a GAN model (generative adversarial network).

[0138] Next, the processing unit Y of the visualization device 100 uses the estimation model read out to estimate the shape of the balloon 25 from the extracted marker image to obtain deformation data that estimates the deformation of the shape (three-dimensional shape) of the balloon 25 from the marker image (step S15 in Figure 13).

[0139] Next, the processing unit Y of the visualization device 100 generates and outputs visualization information that visualizes the contact state of the balloon 25 based on the deformation data (step S16 in FIG. 13).

[0140] As described above, even when some of the markers P are missing from the camera image, the visualization device 100 analyzes the camera image and acquires a marker image by extracting a marker region where the markers P included in the camera image are located. The visualization device 100 then uses an estimation model to acquire deformation data that estimates deformation of the shape of the balloon 25 from the marker image in which some of the markers P are missing. The visualization device 100 then generates and outputs visualization information that visualizes the contact state of the balloon 25 based on the deformation data.

[0141] Next, for example, the processing unit Y of the visualization device 100 determines whether or not to complete the visualization process in response to an operational input from the user (step S17 in FIG. 13). In this step S17, if the processing unit Y of the visualization device 100 does not complete the visualization process in response to the user's operational input, it returns to the above-mentioned step S12 and executes the visualization process from step S12 to step S16 again.

[0142] On the other hand, in this step S17, when the visualization process is completed in response to the user's operation input, the processing unit Y of the visualization device 100 stores necessary information such as deformed mesh data in the memory unit M of the visualization device 100 (step S18 in Figure 13).

[0143] The processing unit Y of the visualization device 100 stores at least one of the deformation data, visualization information, the deformation data, and / or information for generating the deformation data in the storage unit M of the visualization device 100 (step S18 in FIG. 13).

[0144] As described above, when the output unit D is a display device, the visualization information is, for example, an image (e.g., mesh data) showing the contact state of the balloon 25. However, the visualization information may include, for example, text information and / or audio information in addition to or instead of the image (e.g., mesh data) showing the contact state of the balloon 25. The visualization information may also include an image (e.g., mesh data) showing the shape of the balloon 25 (including the amount of deformation from the initial state), and text information and / or audio information showing the shape of the balloon 25.

[0145] For example, by referring to the visualization information output from the medical device system 1000 in this manner, the user can properly recognize the contact state in which the balloon 25 of the balloon catheter system 10 is in contact with the inner surface of the internal organ of a living body.

[0146] As described above, the visualization system 300 according to this embodiment can more appropriately visualize the contact state of the balloon with the target site in real time based on the image information obtained by capturing an image of the balloon with a camera.

[0147] This allows, for example, a doctor who is the user, to more appropriately perform catheter ablation treatment by referring to the visualization information output by the visualization system.

[0148] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents. [Industrial Applicability]

[0149] This embodiment can be used in a balloon catheter system and a balloon catheter for treating arrhythmia such as atrial fibrillation, endometriosis, cancer, and the like. [Explanation of symbols]

[0150] 1000···Medical device system, 10···Balloon catheter system, 100···Visualization device, 200···Estimation model generation device, 300···Visualization system

Claims

1. a visualization device that outputs visualization information for visualizing the contact state of a balloon of a balloon catheter inserted into an organ of a living body and pressed against and in contact with the inner surface of the organ, based on a camera image captured by the camera; The balloon has a plurality of markers disposed thereon, the camera image is an image captured by the camera of an area of ​​the balloon where the plurality of markers are arranged, The visualization device includes: acquiring a camera image of an area including at least a portion of the plurality of markers on the balloon of the balloon catheter, the camera capturing the image; analyzing the camera image to obtain a marker image by extracting a marker region in which the plurality of markers are located; acquiring deformation data obtained by estimating deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image; generating and outputting visualization information for visualizing the contact state of the balloon based on the deformation data; A visualization system that outputs information for use in balloon catheter treatment of living bodies.

2. The visualization system according to claim 1 , wherein the visualization device performs processing on the camera image to remove projection distortion of a camera that captured the camera image before acquiring the marker image.

3. The visualization system according to claim 1 , wherein the plurality of markers are arranged on the balloon at least in an area that is to come into contact with the organ of the living body.

4. When the camera image is an image in which some of the plurality of markers are missing, the visualization device analyzes the camera image to obtain the marker image by extracting the marker regions in which the markers included in the camera image are located; using the estimation model to obtain deformation data that estimates deformation of the shape of the balloon from the marker image in which some of the markers are missing; The visualization system according to claim 1 or 2, further comprising: generating and outputting visualization information for visualizing the contact state of the balloon based on the deformation data.

5. the camera is positioned on the balloon catheter to image the interior of the balloon; 5. The visualization system of claim 4, wherein some of the plurality of markers are missing from the camera image captured by the camera due to an obstruction between the camera and the markers placed on the balloon and / or a change in the relative position of the camera on the balloon catheter.

6. The balloon catheter The balloon; an outer cylindrical shaft connected to the proximal end of the balloon; an inner cylindrical shaft connected to a tip end of the balloon opposite the base end, the inner cylindrical shaft passing through the outer cylindrical shaft and extending into the balloon; the camera is disposed in the balloon catheter so as to capture an image of the inside of the balloon from the proximal end side; The visualization system of claim 5 , wherein the obstruction is a portion of the inner cylindrical shaft located inside the balloon.

7. The visualization device is an input unit for inputting image data of the camera image; a processing unit that acquires the camera image based on the image data input by the input unit, analyzes the camera image to acquire a marker image by extracting a marker region where the plurality of markers are located, acquires deformation data that estimates deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon, and generates visualization information that visualizes the contact state of the balloon based on the deformation data; The visualization system according to claim 1 , further comprising: an output unit that outputs the visualization information acquired by the processing unit.

8. the visualization device includes a storage unit that stores the estimation model in advance; The visualization system according to claim 7 , wherein the processing unit of the visualization device reads the estimation model from the storage unit.

9. 9. The visualization system according to claim 8, wherein the processing unit of the visualization device stores at least one of the deformation data, the visualization information, the deformation data, and / or information for generating the deformation data in the storage unit of the visualization device.

10. an estimation model generation device that generates the estimation model for estimating the shape of the balloon from the marker image by machine learning using a neural network; The visualization system according to any one of claims 1 to 9.

11. 11. The visualization system according to claim 10, wherein the visualization device and the estimation model generation device are configured integrally, and the estimation model generation device is constructed as software in the visualization device that generates the estimation model.

12. 11. The visualization system according to claim 10, wherein the estimation model generation device uses at least one of the deformed data, the visualization information, the deformed data, and / or information for generating the deformed data for machine learning using the neural network.

13. The visualization system according to any one of claims 1 to 12, wherein the visualization information includes an image, text information, and / or audio information indicating the contact state of the balloon, and / or an image, text information, and / or audio information indicating the shape of the balloon.

14. a visualization device that outputs visualization information for visualizing the contact state of a balloon of a balloon catheter inserted into an organ of a living body and pressed against and in contact with the inner surface of the organ, based on a camera image captured by the camera; The balloon has a plurality of markers disposed thereon, the camera image is an image captured by the camera of an area of ​​the balloon where the plurality of markers are arranged, The visualization device acquires a camera image captured by the camera of an area including at least a portion of the plurality of markers on the balloon of the balloon catheter; The visualization device analyzes the camera image to obtain a marker image by extracting a marker region where the plurality of markers are located; The visualization device acquires deformation data obtained by estimating deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image; A control method for a visualization system that outputs information to be used in balloon catheter treatment of a living body, wherein the visualization device generates and outputs visualization information that visualizes the contact state of the balloon based on the deformation data.

15. 1. A visualization device that outputs visualization information for visualizing a contact state of a balloon of a balloon catheter inserted into an organ of a living body and pressed against and in contact with an inner surface of the organ, based on a camera image captured by the camera, the visualization device comprising: The balloon has a plurality of markers disposed thereon, the camera image is an image captured by the camera of an area of ​​the balloon where the plurality of markers are arranged, The visualization device includes: acquiring a camera image of an area including at least a portion of the plurality of markers on the balloon of the balloon catheter, the camera capturing the image; analyzing the camera image to obtain a marker image by extracting a marker region in which the plurality of markers are located; acquiring deformation data obtained by estimating deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image; A visualization device that generates and outputs visualization information that visualizes the contact state of the balloon based on the deformation data.

16. The visualization device is an input unit for inputting image data of the camera image; a processing unit that acquires the camera image based on the image data input by the input unit, analyzes the camera image to acquire a marker image by extracting a marker region where the plurality of markers are located, acquires deformation data that estimates deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon, and generates visualization information that visualizes the contact state of the balloon based on the deformation data; an output unit that outputs the visualization information acquired by the processing unit; The visualization device of claim 15, comprising:

17. A program executed by a visualization device including a computer, the program outputting visualization information for visualizing a contact state of a balloon of a balloon catheter inserted into an organ of a living body and pressed against and in contact with the inner surface of the organ, based on a camera image captured by the camera, the program comprising: The balloon has a plurality of markers disposed thereon, the camera image is an image captured by the camera of an area of ​​the balloon where the plurality of markers are arranged, The visualization device acquiring a camera image of an area including at least a portion of the plurality of markers on the balloon of the balloon catheter, the camera capturing the image; analyzing the camera image to obtain a marker image by extracting a marker region in which the plurality of markers are located; acquiring deformation data obtained by estimating deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image; and generating and outputting visualization information for visualizing the contact state of the balloon based on the deformation data.

18. a balloon catheter system including a balloon catheter including a balloon and a camera for imaging the balloon; a visualization system that outputs information to be used in balloon catheter treatment of a living body based on the camera image captured by the camera; Equipped with The visualization system includes: a visualization device that outputs visualization information for visualizing a contact state of the balloon based on a camera image of the balloon of the balloon catheter that is inserted into an organ of the living body and pressed against and in contact with an inner surface of the organ, The balloon has a plurality of markers disposed thereon, the camera image is an image captured by the camera of an area of ​​the balloon where the plurality of markers are arranged, The visualization device includes: acquiring a camera image of an area including at least a portion of the plurality of markers on the balloon of the balloon catheter, the camera capturing the image; analyzing the camera image to obtain a marker image by extracting a marker region in which the plurality of markers are located; acquiring deformation data obtained by estimating deformation of the shape of the balloon from the marker image using an estimation model for estimating the shape of the balloon from the marker image; A medical device system used in balloon catheter treatment of a living body, which generates and outputs visualization information that visualizes the contact state of the balloon based on the deformation data.

19. The medical device system according to claim 18 , further comprising an estimation model generation device that generates the estimation model for estimating the shape of the balloon from the marker image by machine learning using a neural network.

20. 20. The medical device system according to claim 19, wherein the visualization device and the estimation model generation device are configured as an integrated unit, and the estimation model generation device is constructed as software in the visualization device that generates the estimation model.

21. The visualization device includes: When some of the plurality of markers are missing from the camera image, the camera image is analyzed to extract the marker region in which the markers included in the camera image are located, and the marker image is acquired; using the estimation model to obtain deformation data that estimates deformation of the shape of the balloon from the marker image in which some of the markers are missing; The medical device system according to any one of claims 18 to 20, wherein visualization information for visualizing the contact state of the balloon is generated and output based on the deformation data.

22. the camera is disposed on the balloon catheter so as to capture an image of the interior of the balloon; 21. The medical device system of claim 20, wherein some of the plurality of markers are missing from the camera image captured by the camera due to an obstruction between the camera and the markers placed on the balloon and / or a change in the relative position of the camera on the balloon catheter.

23. The balloon catheter The balloon; an outer cylindrical shaft connected to the proximal end of the balloon; an inner cylindrical shaft connected to a tip end of the balloon opposite the base end, the inner cylindrical shaft passing through the outer cylindrical shaft and extending into the balloon; the camera is disposed in the balloon catheter so as to capture an image of the inside of the balloon from the proximal end side; The medical device system of claim 22 , wherein the shield is a portion of the inner cylindrical shaft located inside the balloon.

Images