Blood vessel demarcation discernment system, blood vessel demarcation discernment method, and program

Abstract

A blood vessel demarcation discernment system according to the present invention recognizes a three-dimensional structure of an abdominal surface of a patient, generates a depth image of the abdominal

Description

Blood vessel classification discrimination system, blood vessel classification discrimination method, and program

[0001] The present invention relates to a blood vessel classification system, a blood vessel classification method, and a program. This application claims priority to Japanese Patent Application No. 2022-062934, filed on April 5, 2022, the contents of which are incorporated herein by reference.

[0002] Resuscitative endovascular balloon occlusion of the aorta (REBOA) is a critical care procedure that uses a balloon catheter to stop bleeding and control bleeding volume in hemodynamically unstable cases such as traumatic bleeding. In REBOA, a balloon is placed in the aorta using a catheter. Unlike conventional catheter treatments, REBOA can be performed in situations where X-ray fluoroscopy, ultrasound, and other devices cannot be used. In recent years, REBOA has attracted attention as an effective procedure for not only traumatic bleeding but also non-traumatic bleeding. In REBOA, perfusion of non-bleeding areas can be selectively preserved by adjusting Zones 1 to 3 of the aorta where the balloon is placed (Zones 1 to 3 of the aorta are divided according to the branching positions of the aorta). Zones 1 to 3 of the aorta are described, for example, in Non-Patent Document 1.

[0003] Under circumstances where an X-ray fluoroscopy device, an ultrasound device, etc. is available and the catheter and Zones 1 to 3 of the aorta can be visualized using the X-ray fluoroscopy device, ultrasound device, etc., the position of the catheter relative to Zones 1 to 3 of the aorta is adjusted while checking the position of the catheter and the positions of Zones 1 to 3 of the aorta. For example, in an emergency, REBOA may be performed even under circumstances where an X-ray fluoroscopy device, ultrasound device, etc. cannot be used and Zones 1 to 3 of the aorta cannot be visualized using an X-ray fluoroscopy device, ultrasound device, etc. In such cases, conventionally, the surgeon had to perform REBOA without being able to grasp Zones 1 to 3 of the patient's aorta.

[0004] Markus Harboe Olsen et al. “Standardized distances for placement of REBOA in patients with aortic stenosis” Scientific Reports volume 10 Article number: 13410 (2020)Takeshi Takata, Susumu Nakabayashi, Hiroshi Kondo, Masayoshi Yamamoto, Shigeru Furui, Kenshiro Shiraishi, Takenori Kobayashi, Hiroshi Oba, Takahide Okamoto & Jun'ichi Kotoku “Mixed Reality Visualization of Radiation Dose for Health Professionals and Patients in Interventional Radiology” Journal of Medical Systems 45:38 (2021)Emily Rae, Andras Lasso, Matthew S. Holden, Evelyn Morin, Ron Levy & Gabor Fichtinger “Neurosurgical burr hole placement using the Microsoft HoloLens” Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling

[0005] In view of the above, an object of the present invention is to provide a blood vessel division discrimination system, a blood vessel division discrimination method, and a program that can easily identify Zone 1, Zone 2, and Zone 3 of a patient's aorta even in a situation where X-ray fluoroscopy or an ultrasound device cannot be used, and can support the realization of REBOA.

[0006] One aspect of the present invention includes a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface; a depth image generation unit that generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device; a blood vessel division discrimination device that uses a deep learning model to discriminate the aortic division of the patient whose three-dimensional structure of the abdominal surface has been recognized by the three-dimensional structure recognition device; and a training dataset generation unit that generates a training dataset to be used for training the deep learning model, wherein before training of the deep learning model is performed, the training dataset generation unit generates abdominal CT (Computed Tomography) images, abdominal MRI (Magnetic Resonance Imaging) images, and abdominal MRA (Magnetic Resonance Imaging) images of a patient different from the patient whose aortic division is to be discriminated by the blood vessel division discrimination device. a training data set indicating a correspondence relationship between each pixel in the training depth image and a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, or any other segment, based on the abdominal CT image, the abdominal MRI image, or the abdominal MRA image; and After the deep learning model has been trained using a deep learning model, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface, the depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device, and the blood vessel classification device uses the trained deep learning model to estimate whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit corresponds to the first blood vessel classification, the second blood vessel classification, the third blood vessel classification, or one of the other classifications.

[0007] One aspect of the present invention is a vascular division discrimination method for a vascular division discrimination system comprising: a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface; a depth image generation unit that generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device; a vascular division discrimination device that uses a deep learning model to discriminate the aortic division of the patient whose three-dimensional structure of the abdominal surface has been recognized by the three-dimensional structure recognition device; and a training dataset generation unit that generates a training dataset to be used for training the deep learning model, in which, before training of the deep learning model, the training dataset generation unit generates a training three-dimensional structure of the abdominal surface from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic division is to be discriminated by the vascular division discrimination device, generates a training depth image from the three-dimensional structure of the abdominal surface for training, and calculates a correlation between each pixel in the training depth image and an aortic division based on any of the abdominal CT images, abdominal MRI images, and abdominal MRA images. a three-dimensional structure recognition step in which the three-dimensional structure recognition device recognizes a three-dimensional structure of the patient's abdominal surface after the deep learning model has been trained using the training dataset generated in the training dataset generation step; a depth image generation step in which the depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized in the three-dimensional structure recognition step; and a blood vessel division discrimination step in which the blood vessel division discrimination device uses the trained deep learning model to estimate whether each pixel in the depth image of the patient's abdominal surface generated in the depth image generation step corresponds to the first blood vessel division, the second blood vessel division, the third blood vessel division, or one of the other divisions.

[0008] One aspect of the present invention is a program for causing a computer constituting a blood vessel division discrimination device included in a blood vessel division discrimination system, the blood vessel division discrimination system including a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface, a depth image generation unit that generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device, and a training dataset generation unit that generates a training dataset to be used for training a deep learning model, to execute a learning step of training a deep learning model using the training dataset generated by the training dataset generation unit, and a blood vessel division discrimination step, in which the training dataset generation unit generates a training three-dimensional structure of the abdominal surface from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from a patient whose aortic division is to be discriminated by the blood vessel division discrimination device, generates a training depth image from the training three-dimensional structure of the abdominal surface, and generates a training depth image from the abdominal and a depth image generation unit that generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device. In the vascular classification step, the program uses the trained deep learning model to estimate whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit corresponds to the first vascular segment, the second vascular segment, the third vascular segment, or one of the other segments.

[0009] One aspect of the present invention includes a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface, a blood vessel segment discrimination device that uses a deep learning model to discriminate the aortic segment of a patient whose abdominal surface three-dimensional structure has been recognized by the three-dimensional structure recognition device, and a training dataset generation unit that generates a training dataset to be used for training the deep learning model, wherein before training of the deep learning model is performed, the training dataset generation unit generates a training three-dimensional structure of the abdominal surface from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic segment is to be discriminated by the blood vessel segment discrimination device, and generates the training three-dimensional structure of the abdominal surface based on any of the abdominal CT images, abdominal MRI images, and abdominal MRA images. a training dataset indicating a correspondence between each point on the three-dimensional structure of the abdominal surface of the patient and a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, or another segment; after the deep learning model is trained using the training dataset generated by the training dataset generation unit, the three-dimensional structure recognition device recognizes the three-dimensional structure of the abdominal surface of the patient; and the vascular segment discrimination device uses the trained deep learning model to estimate whether each point on the three-dimensional structure of the abdominal surface of the patient recognized by the three-dimensional structure recognition device corresponds to the first vascular segment, the second vascular segment, the third vascular segment, or one of the other segments.

[0010] One aspect of the present invention is a blood vessel division discrimination method for a blood vessel division discrimination system including a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface, a blood vessel division discrimination device that uses a deep learning model to discriminate the aortic division of a patient whose three-dimensional structure of the abdominal surface has been recognized by the three-dimensional structure recognition device, and a training dataset generation unit that generates a training dataset to be used for training the deep learning model, in which, before training of the deep learning model, the training dataset generation unit generates a training three-dimensional structure of the abdominal surface from any one of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic division is to be discriminated by the blood vessel division discrimination device, and calculates a correlation between each point on the three-dimensional structure of the abdominal surface for training and the Zo of the aorta based on any one of the abdominal CT images, the abdominal MRI images, and the abdominal MRA images. a three-dimensional structure recognition step in which, after the deep learning model has been trained using the training dataset generated in the training dataset generation step, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface; and a blood vessel division discrimination step in which the blood vessel division discrimination device uses the trained deep learning model to estimate whether each point on the three-dimensional structure of the patient's abdominal surface recognized in the three-dimensional structure recognition step corresponds to the first blood vessel division, the second blood vessel division, the third blood vessel division, or one of the other divisions.

[0011] One aspect of the present invention is a program for causing a computer constituting a blood vessel division discrimination device included in a blood vessel division discrimination system, the system including a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface, and a training dataset generation unit that generates a training dataset used for training a deep learning model, to execute a learning step of training a deep learning model using the training dataset generated by the training dataset generation unit, and a blood vessel division discrimination step, in which the training dataset generation unit generates a three-dimensional structure of the abdominal surface for learning from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from a patient whose aortic division is to be discriminated by the blood vessel division discrimination device, and and the abdominal MRA image, generate the learning data set indicating a correspondence between each point on the three-dimensional structure of the abdominal surface used for learning and one of a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, and another segment; after the learning step is performed, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface; and in the vascular segment determination step, use is made of the trained deep learning model to estimate whether each point on the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or another segment.

[0012] According to the present invention, it is possible to provide a blood vessel segmentation discrimination system, a blood vessel segmentation discrimination method, and a program that can easily identify Zone 1, Zone 2, and Zone 3 of a patient's aorta even in a situation where X-ray fluoroscopy or an ultrasound device cannot be used, and can support the realization of REBOA.

[0013] 1 is a diagram illustrating an example of the blood vessel division discrimination system 1 according to the first embodiment. FIG. 2 is a diagram illustrating an example of processing executed by the training dataset generation unit 13. FIG. 3 is a flowchart illustrating an example of processing executed in the blood vessel division discrimination system 1 according to the first embodiment. FIG. 4 is a diagram conceptually illustrating training of a semantic segmentation model using a training dataset in the inventor's research. FIG. 5 is a diagram conceptually illustrating validation (Test) of a trained semantic segmentation model in the inventor's research. FIG. 6 is a diagram illustrating the results of evaluating, using the Dice coefficient and the Jaccard coefficient, the similarity between Zone 1, Zone 2, and Zone 3 estimated using a trained semantic segmentation model in the inventor's research. FIG. 7 is a diagram illustrating the position error of the boundary between Zone 1 and Zone 2, the boundary between Zone 2 and Zone 3, and the like, estimated using a trained semantic segmentation model in the inventor's research. FIG. 8 is a diagram illustrating four examples in which the position error of the aortic division estimated using a trained semantic segmentation model in the inventor's research was less than 10 mm. FIG. 1 is a diagram showing four examples in which the position error of the aortic segment estimated using a trained semantic segmentation model in the inventor's research is greater than 20 mm. FIG. 2 is a diagram for conceptually explaining a technology for projecting the Zone estimation results onto the patient's body surface in real time using HoloLens 2. FIG. 3 is a diagram showing an example of a blood vessel segmentation discrimination system 2 according to a third embodiment. FIG. 4 is a flowchart for explaining an example of processing executed in the blood vessel segmentation discrimination system 2 according to the third embodiment.

[0014] Hereinafter, embodiments of a blood vessel division discrimination system, a blood vessel division discrimination method, and a program according to the present invention will be described with reference to the drawings.

[0015] [First Embodiment] FIG. 1 is a diagram illustrating an example of a blood vessel classification system 1 according to a first embodiment. In the example illustrated in FIG. 1 , the blood vessel classification system 1 according to the first embodiment assists an operator in performing REBOA on a patient in situations where X-ray fluoroscopy or ultrasound cannot be used, such as in an emergency. The blood vessel classification system 1 includes a three-dimensional structure recognition device 11, a depth image generation unit 12, a learning dataset generation unit 13, a blood vessel classification device 14, and a visualization device 15. The three-dimensional structure recognition device 11 recognizes the three-dimensional structure of the patient's abdominal surface. The three-dimensional structure recognition device 11 recognizes the three-dimensional structure of the patient's abdominal surface and generates three-dimensional data of the patient's abdominal surface, for example, by using technology similar to the "LiDAR scanner" described on the website indicated by the following URL: https: / / bablishe.com / about-lidar-scanner-of-ipad-pro-2020 /

[0016] As the three-dimensional structure recognition device 11, for example, a product described on the website indicated by the following URL can be used: https: / / www.apple.com / jp / ipad-pro / specs / https: / / www.microsoft.com / ja-jp / hololens / hardware

[0017] The former product is used, for example, according to the procedure described on the website indicated at the URL below, to recognize the three-dimensional structure of the external world: https: / / prono82.com / 2020 / 09 / 28 / apple%E7%A4%BElidar%E6%90%AD%E8%BC%89ipad-pro%E3%81%AB%E3%82%88%E3%82%8B3d%E7%82%B9%E7%BE%A4%E3%82%B9%E3%82%AD%E3%83%A3%E3%83%B3%E3%82%A2%E3%83%97%E3%83%AA%E3%81%AE%E3%83%AA%E3%83%AA%E3%83%BC%E3%82%B9 /

[0018] The latter product can be used, for example, according to the procedure described on the website indicated at the following URL, to recognize the three-dimensional structure of the outside world: https: / / docs.microsoft.com / ja-jp / windows / mixed-reality / design / spatial-mapping https: / / zenn.dev / hiromu / articles / 20210421-scene-understanding

[0019] The depth image generation unit 12 generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device 11. A depth image is an image colored according to distance, as described, for example, on the website indicated by the URL below. In other words, the depth image generation unit 12 generates a two-dimensional image of the patient's abdominal surface in which the patient's abdominal surface is colored according to the distance from the three-dimensional structure recognition device 11. The depth image generation unit 12 generates a depth image of the patient's abdominal surface by using, for example, the technology described on the website indicated by the URL below: https: / / www.cit.nihon-u.ac.jp / laboratorydata / kenkyu / kouennkai / reference / No.47 / pdf / 2-53.pdf

[0020] The training dataset generator 13 generates a training dataset. The vascular segment discrimination device 14 uses the deep learning model 14A to discriminate the aortic segment of a patient whose three-dimensional structure on the abdominal surface has been recognized by the three-dimensional structure recognition device 11.

[0021] FIG. 2 is a diagram illustrating an example of processing executed by the training dataset generation unit 13. In the example shown in FIG. 2, the training dataset generation unit 13 generates a training three-dimensional structure of the abdominal surface (the structure shown in the upper center of FIG. 2) from an abdominal CT (Computed Tomography) image (the image shown on the left side of FIG. 2) of a person other than the patient whose aortic segment is to be identified by the vascular segment identification device 14. That is, the training dataset generation unit 13 executes the processing indicated by "Render" in FIG. 2. The training three-dimensional structure of the abdominal surface generated in this processing is similar to the recognition result (three-dimensional structure) obtained if the three-dimensional structure recognition device 11 recognized the three-dimensional structure of the abdominal surface of that person (i.e., the subject of the abdominal CT image). Furthermore, the training dataset generation unit 13 generates a training depth image (the image shown in the upper right side of FIG. 2) from the training three-dimensional structure of the abdominal surface generated by the processing indicated by "Render" in FIG. 2. That is, the learning dataset generation unit 13 executes the process indicated by "Project" in FIG.

[0022] Furthermore, based on the abdominal CT image (the image shown on the left side of FIG. 2 ), the training dataset generation unit 13 generates information (information shown in the lower center of FIG. 2 ) indicating a correspondence between each pixel in the training depth image (the image shown on the upper right side of FIG. 2 ) and any of the first vascular segment corresponding to Zone 1 of the aorta, the second vascular segment corresponding to Zone 2 of the aorta, the third vascular segment corresponding to Zone 3 of the aorta, and other segments (segments shown as "Out of Zones" in FIG. 2 ). That is, the training dataset generation unit 13 executes the process shown by "Label" in FIG. 2 . The training dataset generation unit 13 also generates a training dataset, which is a pair of the training depth image (the image shown on the upper right side of FIG. 2 ) generated by the process shown by "Project" in FIG. 2 and the information (information shown in the lower center of FIG. 2 ) generated by the process shown by "Label" in FIG. 2 , as training data used for training the deep learning model 14A. In the example shown in FIG. 2 , abdominal CT images of a person other than the patient whose aortic division is to be discriminated by the blood vessel division discrimination device 14 are used to generate the training data set. However, in other examples, abdominal MRI (Magnetic Resonance Imaging) images of a person other than the patient whose aortic division is to be discriminated by the blood vessel division discrimination device 14 may be used to generate the training data set. In still other examples, abdominal MRA (Magnetic Resonance Angiography) images of a person other than the patient whose aortic division is to be discriminated by the blood vessel division discrimination device 14 may be used to generate the training data set.

[0023] In the example shown in FIG. 1 , the vascular division discrimination device 14 includes a learning unit 141 and an estimation unit 142. The learning unit 141 trains the deep learning model 14A using a training dataset (e.g., the image shown in the upper right corner of FIG. 2 and the information shown in the lower center of FIG. 2 ) generated by the training dataset generation unit 13. Specifically, in consideration of the fact that a low estimation accuracy of the second vascular division corresponding to Zone 2 of the aorta increases the risk of organ ischemia when REBOA is performed, the learning unit 141 trains the deep learning model 14A so that the estimation accuracy of the second vascular division corresponding to Zone 2 of the aorta using the trained deep learning model 14A is equal to or greater than a predetermined threshold. In the vascular division discrimination system 1 of the first embodiment, a segmentation model is used as the deep learning model 14A. Specifically, in the first example of the vascular division discrimination system 1 of the first embodiment, a semantic segmentation model described, for example, on the website indicated by the URL below is used as the deep learning model 14A. https: / / jp.mathworks.com / content / dam / mathworks / mathworks-dot-com / company / events / webinar-cta / 2459280_Basics_of_semantic_segmentation.pdf https: / / qiita.com / fujiya228 / items / ea30dac6ef827d608a56

[0024] In the second example of the blood vessel category discrimination system 1 of the first embodiment, a segmentation model other than semantic segmentation, such as instance segmentation or panoptic segmentation, which are described on the website indicated by the following URL, may be used as the deep learning model 14A. https: / / www.skillupai.com / blog / tech / segmentation1 /

[0025] In the example shown in FIG. 1 , the estimation unit 142 uses the trained deep learning model 14A to estimate whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit 12 corresponds to a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, or another segment (i.e., a segment that does not correspond to any of the first, second, or third vascular segments). That is, in the example shown in FIG. 1 , after the deep learning model 14A is trained using the training dataset generated by the training dataset generation unit 13, the three-dimensional structure recognition device 11 recognizes the three-dimensional structure of the patient's abdominal surface. Furthermore, the depth image generation unit 12 generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device 11. Furthermore, the estimation unit 142 of the vascular division discrimination device 14 uses the trained deep learning model 14A to estimate whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit 12 corresponds to the first vascular division corresponding to Zone 1 of the aorta, the second vascular division corresponding to Zone 2 of the aorta, the third vascular division corresponding to Zone 3 of the aorta, or another division (the divisions indicated as "Out of Zones" in Figure 2).

[0026] The visualization device 15 generates a virtual image by projecting the estimation result of the patient's aortic division by the blood vessel division discrimination device 14 onto the patient's abdominal surface. The visualization device 15 includes a virtual image generation unit 15A and a virtual image presentation unit 15B. The virtual image generation unit 15A generates a virtual image by projecting the patient's first blood vessel division (Zone 1), second blood vessel division (Zone 2), and third blood vessel division (Zone 3) onto the patient's abdominal surface. The virtual image presentation unit 15B presents the virtual image generated by the virtual image generation unit 15A to a user (e.g., a surgeon) of the blood vessel division discrimination system 1.

[0027] In a first example of the blood vessel division discrimination system 1 of the first embodiment, the visualization device 15 projects the patient's first blood vessel division (Zone 1), second blood vessel division (Zone 2), and third blood vessel division (Zone 3) onto the abdominal surface of the patient by using, for example, techniques described in Non-Patent Documents 2 and 3, and the website indicated by the URL below: https: / / prtimes.jp / main / html / rd / p / 000000073.000004318.html The projection method in the first example of the blood vessel division discrimination system 1 of the first embodiment is, for example, a method using external markers as references, or a method of manually aligning the information to overlay it on the human body.

[0028] In a second example of the blood vessel division discrimination system 1 of the first embodiment, the visualization device 15 uses, for example, a product described on the website indicated by the following URL: https: / / www.apple.com / jp / ipad-pro / specs / In the second example of the blood vessel division discrimination system 1 of the first embodiment, the visualization device 15 directly uses a real three-dimensional object (i.e., the patient's abdominal surface) as a marker by using a technology similar to that described on the website indicated by the following URL: https: / / library.vuforia.com / features / objects / object-reco.html https: / / www.youtube.com / watch?v=jbaUDMvv2Zw

[0029] In a third example of the blood vessel division discrimination system 1 of the first embodiment, a product described on the website indicated by the following URL is used as the visualization device 15: https: / / www.microsoft.com / ja-jp / hololens / hardware In the third example of the blood vessel division discrimination system 1 of the first embodiment, the visualization device 15 uses a technology similar to that described on the website indicated by the following URL, for example, to directly use a real three-dimensional object (i.e., the patient's abdominal surface) as a marker, that is, to directly project the patient's first blood vessel division (Zone 1), second blood vessel division (Zone 2), and third blood vessel division (Zone 3) onto the patient's abdominal surface without any external manipulation. https: / / docs.microsoft.com / ja-jp / azure / object-anchors / overview https: / / car.watch.impress.co.jp / docs / news / 1278295.html https: / / library.vuforia.com / features / objects / object-reco.html https: / / www.youtube.com / watch?v=jbaUDMvv2Zw

[0030] In a third example of the blood vessel division discrimination system 1 of the first embodiment, the visualization device 15 uses a technique similar to that described on the website indicated by the URL below, for example, to generate a virtual image in which the patient's first blood vessel division (Zone 1), second blood vessel division (Zone 2), and third blood vessel division (Zone 3) are projected onto the patient's abdominal surface, and presents the virtual image to a user of the blood vessel division discrimination system 1 (for example, a surgeon). https: / / www.ogis-ri.co.jp / otc / hiroba / technical / point-DX / part6.html https: / / www.tattichan.work / entry / 2019 / 12 / 06 / Reprojection%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6%E6%95%B4%E7%90%86%E3%81%99%E3%82%8B https: / / www.businessinsider.jp / post-185961 https: / / hanada-sekkei.co.jp / armrvr https: / / www.youtube.com / watch?v=QZiQ71EDF-o

[0031] 3 is a flowchart illustrating an example of processing executed in the blood vessel division discrimination system 1 according to the first embodiment. In the example shown in FIG. 3 , in step S11, the training dataset generation unit 13 generates a training dataset used for training the deep learning model 14A. Specifically, in step S11A, the training dataset generation unit 13 generates a training three-dimensional structure of the abdominal surface from any one of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic division is to be discriminated by the blood vessel division discrimination device 14. Next, in step S11B, the training dataset generation unit 13 generates a training depth image from the training three-dimensional structure of the abdominal surface generated in step S11A. Next, in step S11C, the training dataset generation unit 13 generates a training dataset based on either an abdominal CT image, an abdominal MRI image, or an abdominal MRA image, showing the correspondence between each pixel in the training depth image and either the first vascular division corresponding to Zone 1 of the aorta, the second vascular division corresponding to Zone 2 of the aorta, the third vascular division corresponding to Zone 3 of the aorta, or another division (i.e., a division that does not correspond to any of the first vascular division, the second vascular division, or the third vascular division).

[0032] Next, in step S12, the learning unit 141 of the blood vessel division discrimination device 14 trains the deep learning model 14A using the training data set generated in step S11. Next, in step S13, the three-dimensional structure recognition device 11 recognizes the three-dimensional structure of the abdominal surface of a patient whose aortic division is to be discriminated by the blood vessel division discrimination device 14. Next, in step S14, the depth image generation unit 12 generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized in step S13. Next, in step S15, the blood vessel division discrimination device 14 uses the trained deep learning model 14A trained in step S12 to discriminate the aortic division of the patient whose three-dimensional abdominal surface structure was recognized in step S13. Specifically, in step S15, the estimation unit 142 of the vascular division discrimination device 14 uses the trained deep learning model 14A to estimate whether each pixel in the depth image of the patient's abdominal surface generated in step S14 corresponds to a first vascular division corresponding to Zone 1 of the aorta, a second vascular division corresponding to Zone 2 of the aorta, a third vascular division corresponding to Zone 3 of the aorta, or another division (i.e., a division that does not correspond to any of the first, second, or third vascular divisions). Next, in step S16, the virtual image generation unit 15A of the visualization device 15 generates a virtual image in which the patient's first vascular division (Zone 1), second vascular division (Zone 2), and third vascular division (Zone 3) are projected onto the patient's abdominal surface. The virtual image presentation unit 15B of the visualization device 15 presents the virtual image generated by the virtual image generation unit 15A to a user of the vascular division discrimination system 1 (e.g., a surgeon).

[0033] Therefore, according to the blood vessel division discrimination system 1 of the first embodiment, a user of the blood vessel division discrimination system 1 (e.g., a surgeon) can easily identify Zone 1, Zone 2, and Zone 3 of the patient's aorta even in a situation where X-ray fluoroscopy or an ultrasound device cannot be used. As a result, the feasibility of REBOA in a situation where X-ray fluoroscopy or an ultrasound device cannot be used can be improved.

[0034] 1 , the three-dimensional structure recognition device 11 uses a technique similar to that of a LiDAR scanner to recognize the three-dimensional structure of the abdominal surface of a patient whose aortic segment is to be identified by the blood vessel segment identification device 14, and generates three-dimensional data of the patient's abdominal surface. In another example, the three-dimensional structure recognition device 11 may have a function of generating the three-dimensional structure of the abdominal surface of a patient whose aortic segment is to be identified by the blood vessel segment identification device 14 from any one of an abdominal CT image, an abdominal MRI image, and an abdominal MRA image, as in the example shown in FIG.

[0035] 1 , the blood vessel segment determination device 14 outputs, as an estimation result of the patient's aortic segment, information used by the visualization device 15 to generate a virtual image in which the first blood vessel segment (Zone 1), the second blood vessel segment (Zone 2), and the third blood vessel segment (Zone 3) are projected onto the patient's abdominal surface. In another example, the blood vessel segment determination device 14 may output, as an estimation result of the patient's aortic segment, the length from the patient's landmark (i.e., a site suitable for puncture (i.e., a catheter insertion site)) to the patient's first blood vessel segment (Zone 1), the length from the patient's landmark to the patient's second blood vessel segment (Zone 2), and the length from the patient's landmark to the patient's third blood vessel segment (Zone 3), for example, as numerical values.

[0036] In yet another example, a product described on the website indicated by the following URL is used as the visualization device 15: https: / / www.apple.com / jp / ipad-pro / specs / Furthermore, in this example, the technology described on the product's website indicated by the following URL is used to measure the length from the patient's landmark (catheter insertion site) to the patient's first vascular segment (Zone 1), the length from the patient's landmark to the patient's second vascular segment (Zone 2), and the length from the patient's landmark to the patient's third vascular segment (Zone 3). https: / / support.apple.com / ja-jp / guide / ipad / ipad8ac2cfea / ipados Specifically, in this example, a user of blood vessel division identification system 1 (e.g., a surgeon) designates a point corresponding to a patient's landmark (catheter insertion site) and a point corresponding to, for example, the patient's first blood vessel division (Zone 1) on a virtual image presented to the user of blood vessel division identification system 1, and the length between these two points is measured and presented to the user of blood vessel division identification system 1. Therefore, the user of blood vessel division identification system 1 can use the presented result to determine the length of the catheter to be inserted into the patient.

[0037] In another example, the visualization device 15 uses a product described on the website indicated by the following URL: https: / / www.microsoft.com / ja-jp / hololens / hardware Furthermore, in this example, the technology described on the product's website indicated by the following URL is used to measure the length from a patient's landmark (catheter insertion site) to the patient's first vascular segment (Zone 1), the length from the patient's landmark to the patient's second vascular segment (Zone 2), and the length from the patient's landmark to the patient's third vascular segment (Zone 3): https: / / www.windowscentral.com / hololens-gets-virtual-tape-measure In this example, too, a user (e.g., a surgeon) of the blood vessel division identification system 1 specifies a point corresponding to the patient's landmark (catheter insertion site) and a point corresponding to, for example, the patient's first vascular segment (Zone 1) on a virtual image presented to the user of the blood vessel division identification system 1. The length between these two points is then measured and presented to the user of the blood vessel division identification system 1. Therefore, the user of the blood vessel category identification system 1 can use the presented results to determine the length of the catheter to be inserted into the patient.

[0038] In the two examples described above, the balloon catheter inserted into the patient is a balloon catheter with length marks, such as those described on the website indicated by the following URL: https: / / prytimemedical.com / product / er-reboa-plus-catheter /

[0039] In yet another example, the training dataset generation unit 13 may generate a training dataset for learning the running state (tortuosity) of a blood vessel into which a balloon catheter is inserted, based on any one of abdominal CT images (the image shown on the left side of FIG. 2 ), abdominal MRI images, and abdominal MRA images. In this example, the training unit 141 of the blood vessel division discrimination device 14 uses the training dataset to train the deep learning model 14A. The estimation unit 142 of the blood vessel division discrimination device 14 estimates the running state (tortuosity) of a patient's blood vessel into which a balloon catheter is inserted, using the trained deep learning model 14A. Furthermore, when a user of the blood vessel division discrimination system 1 (e.g., a surgeon) specifies a point corresponding to a patient's landmark (catheter insertion site) and a point corresponding to, for example, the patient's first blood vessel division (Zone 1) on a virtual image presented to the user of the blood vessel division discrimination system 1, the length along the blood vessel between the two points is estimated and output based on the linear distance between those two points and the estimation result of the estimation unit 142 of the blood vessel division discrimination device 14. In a variation of this example, the learning dataset generation unit 13 may generate a learning dataset for learning the length along the blood vessel between a point corresponding to a landmark portion (catheter insertion site) of the patient and a point corresponding to, for example, the first blood vessel segment (Zone 1) of the patient, based on any of an abdominal CT image (the image shown on the left side of Figure 2), an abdominal MRI image, and an abdominal MRA image.

[0040] In yet another example, the visualization device 15 may generate a virtual image of a state in which the balloon of a balloon catheter inserted into a patient's blood vessel is placed in any one of the first blood vessel segment (Zone 1), the second blood vessel segment (Zone 2), and the third blood vessel segment (Zone 3) of the patient (i.e., the balloon is inflated), and present this image to a user (e.g., a surgeon) of the blood vessel segment determination system 1. In this example, the user of the blood vessel segment determination system 1 can easily imagine the state in which the balloon is actually placed in any one of the first blood vessel segment (Zone 1), the second blood vessel segment (Zone 2), and the third blood vessel segment (Zone 3) of the patient.

[0041] <Example> In a study to verify the blood vessel division discrimination system 1 of the first embodiment, the present inventor developed a deep learning model 14A that can instantly discriminate the aortic division from the patient's body surface. Specifically, the present inventor combined a segmentation technique based on an advanced deep learning platform with depth information of the patient obtained from the reflection of infrared light irradiated from a camera corresponding to the three-dimensional structure recognition device 11 to estimate the aortic division from three-dimensional information of the patient's body surface.

[0042] The inventor specializes in diagnostic radiology, machine learning, and mixed reality, and has conducted research into the visualization of previously invisible radiation exposure using mixed reality. Medical applications of mixed reality are advancing, and it is beginning to play a role in three-dimensional visualization of organs and telemedicine. The inventor believes that visualizing other unknown information in an intuitive manner could contribute to medical support, and focused on REBOA, a life-saving medical treatment performed even in situations where X-ray fluoroscopy equipment is physically unavailable. This research project was conceived with the belief that highly accurate estimation of the zones to be visualized using deep learning technology would enable visualization with unprecedented accuracy and immersiveness, significantly improving the quality of life-saving medical care.

[0043] The inventors constructed a model that enables zone estimation using deep learning segmentation technology and established a trained model that enables highly accurate estimation by performing the following steps. The training dataset plays an extremely important role in deep learning, and the overall results of the research depend on its completeness. Therefore, the inventors first focused on preparing the training dataset. For training, they used 3D body surface images created from abdominal CT images. The inventors primarily used CT image data owned by Teikyo University Hospital. Furthermore, to avoid bias in the training data, the inventors also utilized an open-access CT image database. The inventors, who are also diagnostic radiologists, segmented the training data and used it as training labels. Furthermore, in case the prediction accuracy was insufficient using body surface images alone, the inventors also collected age and weight information and added it to the training as needed.

[0044] The present inventors used a training dataset to train a semantic segmentation model. During training, the inventors appropriately searched for hyperparameters and modified the network structure to construct an optimal training model. The inventors comprehensively evaluated the trained model using reproducibility, precision, Dice coefficient, etc. In particular, they focused on the reproducibility of Zone 2, which has a high risk of organ ischemia, and aimed to create a model that can identify high-risk areas with high accuracy.

[0045] FIG. 4 is a conceptual diagram illustrating the training of a semantic segmentation model using a training dataset in the inventor's research. FIG. 5 is a conceptual diagram illustrating the validation (test) of a trained semantic segmentation model in the inventor's research. FIG. 6 is a diagram illustrating the results of evaluating the similarity between Zone 1, Zone 2, and Zone 3 estimated using a trained semantic segmentation model in the inventor's research, using the Dice coefficient and the Jaccard coefficient. FIG. 7 is a diagram illustrating the position error of the boundary between Zone 1 and Zone 2, and the position error of the boundary between Zone 2 and Zone 3, estimated using a trained semantic segmentation model in the inventor's research. FIG. 8 is a diagram illustrating four examples in which the position error of the aortic segment estimated using a trained semantic segmentation model in the inventor's research was less than 10 mm. FIG. 9 is a diagram illustrating four examples in which the position error of the aortic segment estimated using a trained semantic segmentation model in the inventor's research was greater than 20 mm.

[0046] Furthermore, the inventor conducted the following to develop an application that can intuitively visualize zones estimated using a trained model using mixed reality, which seamlessly blends reality and virtuality. In his research, the inventor developed an application that runs on an iPad Pro ("iPad" is a registered trademark) (manufactured by Apple Inc.), a tablet device, and estimates and visualizes zones from a trained model. The iPad Pro is equipped with an infrared camera, enabling three-dimensional measurement of real space. Three-dimensional information of a patient's body surface captured by this infrared camera is input into the trained model, and the estimated zone results are projected in real time onto the body surface captured by a conventional camera. As a result, a user of the blood vessel classification system 1 (e.g., a surgeon) can visualize zone classifications simply by holding the iPad Pro over the human body. In this research, the iPad Pro functions as both a three-dimensional structure recognition device 11 and a visualization device 15.

[0047] In addition, the inventors developed an application that runs on the headset device HoloLens2 ("HOLOLENS" is an internationally registered trademark) (manufactured by Microsoft) in their research. It estimates and visualizes zones from a trained model. Like the iPad Pro, the HoloLens2 is equipped with an infrared camera, enabling three-dimensional measurement of real space. Three-dimensional information about the body surface is input into the trained model, and the estimated zones are projected in real time onto the body surface visible through the see-through HoloLens2. This allows a user of the vascular classification system 1 (e.g., a surgeon) to visualize zone classifications simply by wearing the HoloLens2 and peering at the patient. In this research, the HoloLens2 functions as both a three-dimensional structure recognition device 11 and a visualization device 15. Because the HoloLens2 has a different operating system and hardware configuration from the iPad Pro, modifications to the surface of the application are necessary, but the core part of the system, related to zone classification estimation, can be shared (i.e., either the iPad Pro or the HoloLens2 can be used as the visualization device 15).

[0048] FIG. 10 is a diagram for conceptually explaining a technology for projecting the Zone estimation results onto the patient's body surface in real time using HoloLens 2.

[0049] As a result of the above-mentioned research, the inventors have found that it is possible to estimate zone divisions with sufficiently high accuracy from three-dimensional images of a patient's body surface. Specifically, the inventors constructed a dataset from a small portion of the CT images to be used and conducted segmentation tests using deep learning. As a result, the reproducibility of Zone 1 was up to 0.99, the reproducibility of Zone 2 was up to 0.95, and the reproducibility of Zone 3 was up to 0.98. The inventors have been studying implementation methods for visualization using mixed reality based on the knowledge and experience they have gained from their previous research, and are close to realizing technology for projecting zones onto a patient's body surface (i.e., operating the system as blood vessel division discrimination system 1 in an actual device).

[0050] Second Embodiment A second embodiment of the blood vessel division discrimination system, blood vessel division discrimination method, and program of the present invention will now be described. The blood vessel division discrimination system 1 of the second embodiment is configured similarly to the blood vessel division discrimination system 1 of the first embodiment described above, except for the points described below. Therefore, the blood vessel division discrimination system 1 of the second embodiment can achieve the same effects as the blood vessel division discrimination system 1 of the first embodiment described above, except for the points described below.

[0051] As described above, in the blood vessel division discrimination system 1 of the first embodiment, a segmentation model is used as the deep learning model 14A. On the other hand, in the blood vessel division discrimination system 1 of the second embodiment, a pseudo-image generation technology such as a Generative Adversarial Network (GAN) described on the website indicated by the following URL is used as the deep learning model 14A: https: / / ledge.ai / gan / https: / / www.nedo.go.jp / news / press / AA5_101472.html

[0052] Third Embodiment A third embodiment of the blood vessel division discrimination system, blood vessel division discrimination method, and program of the present invention will now be described. The blood vessel division discrimination system 2 of the third embodiment is configured similarly to the blood vessel division discrimination system 1 of the first embodiment described above, except for the points described below. Therefore, the blood vessel division discrimination system 2 of the third embodiment can achieve the same effects as the blood vessel division discrimination system 1 of the first embodiment described above, except for the points described below.

[0053] Fig. 11 is a diagram showing an example of a blood vessel category discrimination system 2 according to the third embodiment. In the example shown in Fig. 11, the blood vessel category discrimination system 2 according to the third embodiment, like the blood vessel category discrimination system 1 according to the first embodiment, supports the surgeon in performing REBOA on a patient in situations where X-ray fluoroscopy or ultrasound equipment cannot be used, such as in an emergency. The blood vessel category discrimination system 2 includes a three-dimensional structure recognition device 21, a learning dataset generation unit 23, a blood vessel category discrimination device 24, and a visualization device 25. The three-dimensional structure recognition device 21 recognizes the three-dimensional structure of the abdominal surface of a patient, like the three-dimensional structure recognition device 11 shown in Fig. 1.

[0054] The training dataset generation unit 23 generates a training dataset. The vascular segment determination device 24 uses the deep learning model 24A to determine the aortic segment of a patient whose three-dimensional structure on the abdominal surface has been recognized by the three-dimensional structure recognition device 21.

[0055] In one example of the blood vessel division discrimination system 2 of the third embodiment, the learning dataset generation unit 23 generates a three-dimensional structure of the abdominal surface for learning from any one of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic division is to be discriminated by the blood vessel division discrimination device 24.

[0056] Furthermore, based on any one of abdominal CT images, abdominal MRI images, and abdominal MRA images, the training dataset generation unit 23 generates information indicating the correspondence between each point on the three-dimensional structure of the training abdominal surface and any one of the first vascular segment corresponding to Zone 1 of the aorta, the second vascular segment corresponding to Zone 2 of the aorta, the third vascular segment corresponding to Zone 3 of the aorta, and other segments. The training dataset generation unit 23 also generates a training dataset, which is a combination of the three-dimensional structure of the training abdominal surface and the above-mentioned information, as training data used for training the deep learning model 24A.

[0057] In the example shown in FIG. 11 , the vascular division discrimination device 24 includes a learning unit 241 and an estimation unit 242. The learning unit 241 trains the deep learning model 24A using the training dataset generated by the training dataset generation unit 23. Specifically, in consideration of the fact that a low estimation accuracy of the second vascular division corresponding to Zone 2 of the aorta increases the risk of organ ischemia when REBOA is performed, the learning unit 241 trains the deep learning model 24A so that the estimation accuracy of the second vascular division corresponding to Zone 2 of the aorta using the trained deep learning model 24A is equal to or greater than a predetermined threshold. In the vascular division discrimination system 2 of the third embodiment, a segmentation model is used as the deep learning model 24A, similar to the vascular division discrimination system 1 of the first embodiment. Specifically, in the first example of the vascular division discrimination system 1 of the third embodiment, the above-described semantic segmentation model is used as the deep learning model 24A.

[0058] In a second example of the blood vessel division discrimination system 2 of the third embodiment, a segmentation model other than semantic segmentation, such as the above-mentioned instance segmentation or panoptic segmentation, may be used as the deep learning model 24A.

[0059] 11 , the estimation unit 242 uses the trained deep learning model 24A to estimate whether each point on the patient's three-dimensional structure recognized by the three-dimensional structure recognition device 21 corresponds to a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, or another segment (i.e., a segment that does not correspond to any of the first, second, or third vascular segment). That is, in the example shown in FIG. 11 , after training of the deep learning model 24A is performed using the training dataset generated by the training dataset generation unit 23, the three-dimensional structure recognition device 21 recognizes the three-dimensional structure of the patient's abdominal surface. In addition, the estimation unit 242 of the vascular classification device 24 uses the trained deep learning model 24A to estimate whether each point on the patient's three-dimensional structure recognized by the three-dimensional structure recognition device 21 corresponds to the first vascular classification corresponding to Zone 1 of the aorta, the second vascular classification corresponding to Zone 2 of the aorta, the third vascular classification corresponding to Zone 3 of the aorta, or another classification.

[0060] The visualization device 25 generates a virtual image by projecting the estimation result of the patient's aortic division by the blood vessel division discrimination device 24 onto the patient's abdominal surface. The visualization device 25 includes a virtual image generation unit 25A and a virtual image presentation unit 25B. The virtual image generation unit 25A generates a virtual image by projecting the patient's first blood vessel division (Zone 1), second blood vessel division (Zone 2), and third blood vessel division (Zone 3) onto the patient's abdominal surface. The virtual image presentation unit 25B presents the virtual image generated by the virtual image generation unit 25A to a user (e.g., a surgeon) of the blood vessel division discrimination system 2.

[0061] In the first example of the blood vessel division discrimination system 2 of the third embodiment, the visualization device 25 projects the patient's first blood vessel division (Zone 1), second blood vessel division (Zone 2), and third blood vessel division (Zone 3) onto the abdominal surface of the patient, similar to the first example of the blood vessel division discrimination system 1 of the first embodiment.

[0062] In the second example of the blood vessel division discrimination system 2 of the third embodiment, similar to the second example of the blood vessel division discrimination system 1 of the first embodiment, the visualization device 25 directly uses an actual three-dimensional object (i.e., the patient's abdominal surface) as a marker, i.e., directly projects the patient's first blood vessel division (Zone 1), second blood vessel division (Zone 2), and third blood vessel division (Zone 3) onto the patient's abdominal surface without any external manipulation.

[0063] In the third example of the blood vessel division discrimination system 2 of the third embodiment, similar to the third example of the blood vessel division discrimination system 1 of the first embodiment, the visualization device 25 directly uses an actual three-dimensional object (i.e., the abdominal surface of the patient) as a marker, i.e., directly projects the first blood vessel division (Zone 1), the second blood vessel division (Zone 2), and the third blood vessel division (Zone 3) of the patient onto the abdominal surface of the patient without any external manipulation.

[0064] In the third example of the blood vessel division discrimination system 2 of the third embodiment, the visualization device 25 generates a virtual image in which the patient's first blood vessel division (Zone 1), second blood vessel division (Zone 2), and third blood vessel division (Zone 3) are projected onto the patient's abdominal surface, as in the third example of the blood vessel division discrimination system 1 of the first embodiment, and presents the virtual image to a user of the blood vessel division discrimination system 2 (e.g., a surgeon).

[0065] 12 is a flowchart illustrating an example of processing executed in the blood vessel division discrimination system 2 of the third embodiment. In the example shown in FIG. 12 , in step S21, the training dataset generation unit 23 generates a training dataset to be used for training the deep learning model 24A. More specifically, in step S21A, the training dataset generation unit 23 generates a three-dimensional structure of the abdominal surface for training from any one of abdominal CT images, abdominal MRI images, and abdominal MRA images of a person different from the patient whose aortic division is to be discriminated by the blood vessel division discrimination device 14. Next, in step S21C, the training dataset generation unit 13 generates a training dataset based on any one of an abdominal CT image, an abdominal MRI image, and an abdominal MRA image, showing the correspondence between each point on the three-dimensional structure of the training abdominal surface and either the first vascular division corresponding to Zone 1 of the aorta, the second vascular division corresponding to Zone 2 of the aorta, the third vascular division corresponding to Zone 3 of the aorta, or another division (i.e., a division that does not correspond to any of the first vascular division, the second vascular division, and the third vascular division).

[0066] Next, in step S22, the learning unit 241 of the blood vessel division discrimination device 24 trains the deep learning model 24A using the training data set generated in step S21. Next, in step S23, the three-dimensional structure recognition device 21 recognizes the three-dimensional structure of the abdominal surface of the patient whose aortic division is to be discriminated by the blood vessel division discrimination device 24. Next, in step S25, the blood vessel division discrimination device 14 uses the trained deep learning model 24A trained in step S22 to discriminate the aortic division of the patient whose three-dimensional abdominal surface structure was recognized in step S23. Specifically, in step S25, the estimation unit 242 of the vascular division discrimination device 24 uses the trained deep learning model 24A to estimate whether each point on the three-dimensional structure of the patient's abdominal surface recognized in step S23 corresponds to a first vascular division corresponding to Zone 1 of the aorta, a second vascular division corresponding to Zone 2 of the aorta, a third vascular division corresponding to Zone 3 of the aorta, or another division (i.e., a division that does not correspond to any of the first, second, or third vascular divisions). Next, in step S26, the virtual image generation unit 25A of the visualization device 25 generates a virtual image in which the patient's first vascular division (Zone 1), second vascular division (Zone 2), and third vascular division (Zone 3) are projected onto the patient's abdominal surface. Furthermore, the virtual image presentation unit 25B of the visualization device 25 presents the virtual image generated by the virtual image generation unit 25A to a user (e.g., a surgeon) of the vascular division discrimination system 2.

[0067] Therefore, according to the blood vessel division discrimination system 2 of the third embodiment, a user of the blood vessel division discrimination system 2 (e.g., a surgeon) can easily identify Zone 1, Zone 2, and Zone 3 of the patient's aorta even in a situation where X-ray fluoroscopy or an ultrasound device cannot be used. As a result, the feasibility of REBOA can be improved in a situation where X-ray fluoroscopy or an ultrasound device cannot be used.

[0068] 11 , the three-dimensional structure recognition device 21 uses a technique similar to that of a LiDAR scanner to recognize the three-dimensional structure of the abdominal surface of a patient whose aortic segment is to be identified by the blood vessel segment identification device 24, and generates three-dimensional data of the patient's abdominal surface. In another example, the three-dimensional structure recognition device 21 may have a function of generating the three-dimensional structure of the abdominal surface of a patient whose aortic segment is to be identified by the blood vessel segment identification device 24 from any one of an abdominal CT image, an abdominal MRI image, and an abdominal MRA image, as in the example shown in FIG.

[0069] 11 , the blood vessel segment determination device 24 outputs, as an estimation result of the patient's aortic segment, information used by the visualization device 25 to generate a virtual image in which the first blood vessel segment (Zone 1), the second blood vessel segment (Zone 2), and the third blood vessel segment (Zone 3) are projected onto the patient's abdominal surface. In another example, the blood vessel segment determination device 24 may output, as an estimation result of the patient's aortic segment, the length from the patient's landmark (i.e., a site suitable for puncture (i.e., a catheter insertion site)) to the patient's first blood vessel segment (Zone 1), the length from the patient's landmark to the patient's second blood vessel segment (Zone 2), and the length from the patient's landmark to the patient's third blood vessel segment (Zone 3), for example, as numerical values.

[0070] [Fourth Embodiment] A fourth embodiment of the blood vessel division discrimination system, blood vessel division discrimination method, and program of the present invention will be described below. The blood vessel division discrimination system 2 of the fourth embodiment is configured similarly to the blood vessel division discrimination system 2 of the third embodiment described above, except for the points described below. Therefore, the blood vessel division discrimination system 2 of the fourth embodiment can achieve the same effects as the blood vessel division discrimination system 2 of the third embodiment described above, except for the points described below.

[0071] As described above, in the blood vessel division discrimination system 2 of the third embodiment, a segmentation model is used as the deep learning model 24A. On the other hand, in the blood vessel division discrimination system 2 of the fourth embodiment, a pseudo image generation technique such as the GAN described above is used as the deep learning model 24A.

[0072] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and appropriate modifications can be made without departing from the spirit of the present invention. The configurations described in the above-described embodiments and examples may be combined.

[0073] Note that all or part of the blood vessel division discrimination systems 1 and 2 in the above-described embodiments may be realized by dedicated hardware or by a memory and a microprocessor. Note that all or part of the blood vessel division discrimination system 1 may be configured by a memory and a CPU (central processing unit), and the functions of each unit of the system may be realized by loading a program into the memory and executing it. Note that a program for realizing all or part of the functions of the blood vessel division discrimination system 1 may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform processing of each unit. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, if a WWW system is used, the term "computer system" also includes the web page provision environment (or display environment). Furthermore, the term "computer-readable recording medium" refers to portable media such as a flexible disk, optical magnetic disk, ROM, CD-ROM, etc., and storage devices such as a hard disk built into a computer system. Furthermore, the term "computer-readable recording medium" includes those that dynamically store a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that store a program for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. The program may also be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already stored in the computer system.

[0074] DESCRIPTION OF SYMBOLS 1...vascular division discrimination system, 11...three-dimensional structure recognition device, 12...depth image generation unit, 13...learning data set generation unit, 14...vascular division discrimination device, 14A...deep learning model, 141...learning unit, 142...estimation unit, 15...visualization device, 15A...virtual image generation unit, 15B...virtual image presentation unit, 2...vascular division discrimination system, 21...three-dimensional structure recognition device, 23...learning data set generation unit, 24...vascular division discrimination device, 24A...deep learning model, 241...learning unit, 242...estimation unit, 25...visualization device, 25A...virtual image generation unit, 25B...virtual image presentation unit

Claims

1. A system comprising: a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface; a depth image generation unit that generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device; a blood vessel division discrimination device that uses a deep learning model to discriminate the aortic division of the patient whose three-dimensional structure of the abdominal surface has been recognized by the three-dimensional structure recognition device; and a training dataset generation unit that generates a training dataset used for training the deep learning model, wherein before training of the deep learning model is performed, the training dataset generation unit: generates a training three-dimensional structure of the abdominal surface from one of abdominal CT (Computed Tomography) images, abdominal MRI (Magnetic Resonance Imaging) images, and abdominal MRA (Magnetic Resonance Angiography) images of a subject other than the patient whose aortic division is to be discriminated by the blood vessel division discrimination device; generates a training depth image from the training three-dimensional structure of the abdominal surface; a training dataset indicating a correspondence between each pixel in the training depth image and a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, or any of other segments, based on the abdominal CT image, the abdominal MRI image, or the abdominal MRA image; after the deep learning model has been trained using the training dataset generated by the training dataset generation unit, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface; the depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device; and the blood vessel segment discrimination device uses the trained deep learning model to estimate whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit corresponds to the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments.

2. The blood vessel division discrimination system according to claim 1, further comprising a visualization device that generates a virtual image in which the estimation result of the aortic division of the patient by said blood vessel division discrimination device is projected onto the abdominal surface of the patient.

3. The vascular classification discrimination device according to claim 1, comprising: a learning unit that trains the deep learning model using the training dataset generated by the training dataset generation unit; and an estimation unit that uses the trained deep learning model to estimate whether each pixel in a depth image of the patient's abdominal surface generated by the depth image generation unit corresponds to the first vascular classification, the second vascular classification, the third vascular classification, or one of the other classifications, and the learning unit trains the deep learning model so that the estimation accuracy of the second vascular classification using the trained deep learning model is equal to or greater than a predetermined threshold.

4. The blood vessel classification discrimination system according to claim 1, wherein the three-dimensional structure recognition device has a function of generating a three-dimensional structure of the patient's abdominal surface from one of an abdominal CT image, an abdominal MRI image, and an abdominal MRA image of the patient.

5. The blood vessel division determination system according to claim 1, wherein the blood vessel division determination device outputs, as an estimation result of the patient's aortic division, the length from the patient's landmark to the patient's first blood vessel division, the length from the patient's landmark to the patient's second blood vessel division, and the length from the patient's landmark to the patient's third blood vessel division.

6. A vascular classification method for a vascular classification system comprising: a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface; a depth image generation unit that generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device; a vascular classification device that uses a deep learning model to classify the aortic classification of the patient whose three-dimensional structure of the abdominal surface has been recognized by the three-dimensional structure recognition device; and a training dataset generation unit that generates a training dataset to be used for training the deep learning model, wherein, before training of the deep learning model, the training dataset generation unit generates a training three-dimensional structure of the abdominal surface from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic classification is to be performed by the vascular classification device, generates a training depth image from the training three-dimensional structure of the abdominal surface, the method comprises: a training dataset generation step of generating a training dataset indicating a correspondence between each pixel in the training depth image and a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, or any of other segments, based on the abdominal CT image, the abdominal MRI image, or the abdominal MRA image; a three-dimensional structure recognition step of recognizing a three-dimensional structure of the patient's abdominal surface by the three-dimensional structure recognition device after the deep learning model has been trained using the training dataset generated in the training dataset generation step; a depth image generation step of generating a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized in the three-dimensional structure recognition step; and a blood vessel segment discrimination step of estimating whether each pixel in the depth image of the patient's abdominal surface generated in the depth image generation step corresponds to the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments, by using the trained deep learning model. Method for determining blood vessel classification.

7. A program for causing a computer constituting a vascular division discrimination device included in a vascular division discrimination system comprising: a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface; a depth image generation unit that generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device; and a training dataset generation unit that generates a training dataset used for training a deep learning model, to execute a learning step of training a deep learning model using the training dataset generated by the training dataset generation unit; and a vascular division discrimination step, wherein the training dataset generation unit generates a training three-dimensional structure of the abdominal surface from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic division is to be discriminated by the vascular division discrimination device, generates a training depth image from the training three-dimensional structure of the abdominal surface, a learning data set indicating a correspondence between each pixel in the learning depth image and a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, or any of other segments, based on the abdominal CT image, the abdominal MRI image, or the abdominal MRA image; after the learning step, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface, and the depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device; and in the vascular segment determination step, by using the trained deep learning model, it is estimated whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit corresponds to the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments.

8. A system comprising: a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface; a blood vessel segment discrimination device that uses a deep learning model to discriminate the aortic segment of a patient whose three-dimensional abdominal surface structure has been recognized by the three-dimensional structure recognition device; and a training dataset generation unit that generates a training dataset to be used for training the deep learning model, wherein before training of the deep learning model is performed, the training dataset generation unit: generates a training three-dimensional structure of the abdominal surface from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic segment is to be discriminated by the blood vessel segment discrimination device; and generates a training dataset indicating a correspondence between each point on the training three-dimensional structure of the abdominal surface and any of a first blood vessel segment corresponding to Zone 1 of the aorta, a second blood vessel segment corresponding to Zone 2 of the aorta, a third blood vessel segment corresponding to Zone 3 of the aorta, and other segments, based on any of the abdominal CT images, MRI images, and MRA images. a blood vessel classification system, wherein after the deep learning model has been trained using the training dataset generated by the training dataset generation unit, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface, and the blood vessel classification device uses the trained deep learning model to estimate whether each point on the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device corresponds to the first blood vessel classification, the second blood vessel classification, the third blood vessel classification, or one of the other classifications.

9. The blood vessel division discrimination system according to claim 8, further comprising a visualization device that generates a virtual image in which the estimation result of the aortic division of the patient by said blood vessel division discrimination device is projected onto the abdominal surface of the patient.

10. The vascular classification system of claim 8, wherein the vascular classification device comprises: a learning unit that uses the training dataset generated by the training dataset generation unit to train the deep learning model; and an estimation unit that uses the trained deep learning model to estimate whether each point on the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device corresponds to the first vascular classification, the second vascular classification, the third vascular classification, or one of the other classifications, and the learning unit trains the deep learning model so that the estimation accuracy of the second vascular classification using the trained deep learning model is equal to or greater than a predetermined threshold.

11. The blood vessel classification discrimination system according to claim 8, wherein the three-dimensional structure recognition device has a function of generating a three-dimensional structure of the patient's abdominal surface from one of an abdominal CT image, an abdominal MRI image, and an abdominal MRA image of the patient.

12. The blood vessel division determination system according to claim 8, wherein the blood vessel division determination device outputs, as an estimation result of the patient's aortic division, the length from the patient's landmark to the patient's first blood vessel division, the length from the patient's landmark to the patient's second blood vessel division, and the length from the patient's landmark to the patient's third blood vessel division.

13. A vascular segment discrimination method for a vascular segment discrimination system comprising: a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface; a vascular segment discrimination device that uses a deep learning model to discriminate the aortic segment of a patient whose three-dimensional abdominal surface structure has been recognized by the three-dimensional structure recognition device; and a training dataset generation unit that generates a training dataset to be used for training the deep learning model, wherein, before training of the deep learning model, the training dataset generation unit: generates a training three-dimensional structure of the abdominal surface from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic segment is to be discriminated by the vascular segment discrimination device; and generates a training dataset indicating a correspondence between each point on the training three-dimensional structure of the abdominal surface and any of a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, and other segments, based on any of the abdominal CT images, MRI images, and MRA images; a three-dimensional structure recognition step in which the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface after the deep learning model has been trained using the training dataset generated in the training dataset generation step; and a blood vessel category determination step in which the blood vessel category determination device uses the trained deep learning model to estimate whether each point on the three-dimensional structure of the patient's abdominal surface recognized in the three-dimensional structure recognition step corresponds to the first blood vessel category, the second blood vessel category, the third blood vessel category, or one of the other categories.

14. A program for causing a computer constituting a vascular division discrimination device included in a vascular division discrimination system, the vascular division discrimination system comprising: a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface; and a training dataset generation unit that generates training datasets used for training a deep learning model, to execute a learning step of training a deep learning model using the training dataset generated by the training dataset generation unit, and a vascular division discrimination step, wherein the training dataset generation unit generates a training three-dimensional structure of the abdominal surface from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a patient different from the patient whose aortic division is to be discriminated by the vascular division discrimination device, and generates the training dataset indicating a correspondence between each point on the training three-dimensional structure of the abdominal surface and any of a first vascular division corresponding to Zone 1 of the aorta, a second vascular division corresponding to Zone 2 of the aorta, a third vascular division corresponding to Zone 3 of the aorta, and other divisions, based on any of the abdominal CT images, MRI images, and MRA images, After the learning step is performed, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface, and in the vascular division determination step, by using the trained deep learning model, it is estimated whether each point on the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device corresponds to the first vascular division, the second vascular division, the third vascular division, or one of the other divisions.

Images