Vascular segmentation discrimination system, vascular segmentation discrimination method, and program

Abstract

To easily identify a Zone 1, a Zone 2, a Zone 3 of an aorta of a patient even under such a situation that radioscopy and an ultrasonic device cannot be used.SOLUTION: A blood vessel section discrimination system recognizes a three-dimensional structure of an abdominal part surface of a patient, generates a depth image of the abdominal part surface of the patient from the three-dimensional structure of the abdominal part surface of the patient, generates a learning data set to be used in learning of a deep learning model, and discriminates an aorta section of the patient by using the learned deep learning model. A learning data set generation unit generates a three-dimensional structure of an abdominal part surface for learning from an abdominal part CT image and the like of a person different from a patient being a discrimination object of an aorta section by a blood vessel section discrimination device, generates a depth image for learning from the three-dimensional structure of the abdominal part surface for learning, and generates a learning data set indicating a correspondence between each pixel in the depth image for learning and any of an aorta Zone 1, an aorta Zone 2, an aorta Zone 3 and others on the basis of the abdominal part CT image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a vascular section discrimination system, a vascular section discrimination method, and a program.

Background Art

[0002] Resuscitative Endovascular Balloon Occlusion of the Aorta (REBOA) is a life-saving emergency technique for controlling bleeding and the amount of bleeding using a balloon catheter for traumatic bleeding with unstable hemodynamics. In REBOA, a balloon is placed in the aorta using a catheter. Different from normal catheter treatments, REBOA is performed even in situations where an X-ray fluoroscope, an ultrasonic device, etc. cannot be used. In recent years, REBOA has also attracted attention as an effective technique not only for traumatic bleeding but also for non-traumatic bleeding. In REBOA, perfusion of non-bleeding parts can be selectively preserved by adjusting Zones 1 to 3 of the aorta where the balloon is placed (Zones 1 to 3 of the aorta are divided according to the branching position of the aorta). Zones 1 to 3 of the aorta are described in, for example, Non-Patent Document 1.

[0003] When an X-ray fluoroscope, an ultrasonic device, etc. can be used and the catheter and Zones 1 to 3 of the aorta can be fluoroscoped by the X-ray fluoroscope, the ultrasonic device, etc., the position of the catheter is adjusted with respect to Zones 1 to 3 of the aorta while confirming the position of the catheter and the position of Zones 1 to 3 of the aorta. For example, in an emergency or the like, REBOA may be performed even in a situation where an X-ray fluoroscope, an ultrasonic device, etc. cannot be used and Zones 1 to 3 of the aorta cannot be fluoroscoped by the X-ray fluoroscope, the ultrasonic device, etc. In such a case, conventionally, the operator has had to perform REBOA without being able to grasp Zones 1 to 3 of the patient's aorta.

Prior Art Documents

[0004] [Non-Patent Document 1] Markus Harboe Olsen et al. “Standardized distances for placement of REBOA in patients with aortic stenosis” Scientific Reports volume 10 Article number: 13410 (2020) [Non-Patent Document 2] Takeshi Takata, Susumu Nakabayashi, Hiroshi Kondo, Masayoshi Yamamoto, Shigeru Furui, Kenshiro Shiraishi, Takenori Kobayashi, Hiroshi Oba, Takahide Okamoto & Jun'ichi Kotoku “Mixed Reality Visualization of Radiation Dose for Health Professionals and Patients in Interventional Radiology” Journal of Medical Systems 45:38 (2021) [Non-Patent Document 3] Emily Rae, Andras Lasso, Matthew S. Holden, Evelyn Morin, Ron Levy & Gabor Fichtinger “Neurosurgical burr hole placement using the Microsoft HoloLens” Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] In view of the above, the present invention aims to provide a vascular segmentation discrimination system, a vascular segmentation discrimination method, and a program that can easily identify Zone 1, Zone 2, and Zone 3 of a patient's aorta even in situations where X-ray fluoroscopy or ultrasound equipment cannot be used, and that can support the realization of REBOA. [Means for solving the problem]

[0006] One aspect of the present invention comprises a three-dimensional structure recognition device for recognizing the three-dimensional structure of a patient's abdominal surface, a depth image generation unit for generating a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device, a vascular segmentation discrimination device for determining the aortic segment of a patient whose three-dimensional structure of the abdominal surface has been recognized by the three-dimensional structure recognition device using a deep learning model, and a learning dataset generation unit for generating a learning dataset used to train the deep learning model, wherein before the training of the deep learning model is performed, the learning dataset generation unit generates abdominal CT (Computed Tomography) images, abdominal MRI (Magnetic Resonance Imaging) images, and abdominal MRA (magnetic resonance) images of a person other than the patient whose aortic segment is to be determined by the vascular segmentation discrimination device. The system generates a 3D structure of the abdominal surface for training from any of the angiography images, generates a training depth image from the 3D structure of the abdominal surface for training, and generates a training dataset showing the correspondence between each pixel in the training depth image and any of the first vascular divisions corresponding to Zone 1 of the aorta, the second vascular divisions corresponding to Zone 2 of the aorta, the third vascular divisions corresponding to Zone 3 of the aorta, and other divisions based on any of the abdominal CT image, abdominal MRI image, and abdominal MRA image, and generates the training dataset generated by the training dataset generation unit. This is a vascular segmentation system in which, after the deep learning model is trained using the method described above, the 3D structure recognition device recognizes the 3D structure of the patient's abdominal surface, the depth image generation unit generates a depth image of the patient's abdominal surface from the 3D structure of the patient's abdominal surface recognized by the 3D structure recognition device, and the vascular segmentation discrimination device estimates, using the trained deep learning model, whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments.

[0007] One aspect of the present invention is a vascular segment discrimination method for a vascular segment discrimination system comprising: a 3D structure recognition device for recognizing the 3D structure of a patient's abdominal surface; a depth image generation unit for generating a depth image of the patient's abdominal surface from the 3D structure of the patient's abdominal surface recognized by the 3D structure recognition device; a vascular segment discrimination device for discriminating the aortic segment of a patient whose 3D structure of the abdominal surface has been recognized by the 3D structure recognition device using a deep learning model; and a learning dataset generation unit for generating a learning dataset used to train the deep learning model, wherein, before the training of the deep learning model is performed, the learning dataset generation unit generates a 3D structure of the abdominal surface for training from any of the abdominal CT images, abdominal MRI images, and abdominal MRA images of a person other than the patient whose aortic segment is to be discriminated by the vascular segment discrimination device; generates a learning depth image from the 3D structure of the abdominal surface for training; and, based on any of the abdominal CT images, abdominal MRI images, and abdominal MRA images, each pixel in the learning depth image and the large The vascular segment discrimination method comprises: a training dataset generation step of generating a training dataset that shows the correspondence between a first vascular segment corresponding to Zone 1 of the arteries, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, and any of the other segments; a three-dimensional structure recognition step in which the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface after the deep learning model has been trained using the training dataset generated in the training dataset generation step; a depth image generation step in which the depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized in the three-dimensional structure recognition step; and a vascular segment discrimination step in which the vascular segment discrimination device estimates whether each pixel in the depth image of the patient's abdominal surface generated in the depth image generation step corresponds to any of the first vascular segment, the second vascular segment, the third vascular segment, and any of the other segments by using the trained deep learning model.

[0008] One aspect of the present invention is a program for a computer constituting a vascular segment discrimination device in a vascular segment discrimination system comprising: a 3D structure recognition device for recognizing the 3D structure of a patient's abdominal surface; a depth image generation unit for generating a depth image of the patient's abdominal surface from the 3D structure of the patient's abdominal surface recognized by the 3D structure recognition device; and a learning dataset generation unit for generating a learning dataset used to train a deep learning model, wherein the learning dataset generation unit generates a 3D structure of the abdominal surface for training from any of the abdominal CT images, abdominal MRI images, and abdominal MRA images of a person other than the patient whose aortic segment discrimination is to be performed by the vascular segment discrimination device, generates a learning depth image from the learning depth image of the abdominal surface, and generates the abdominal The program generates a training dataset showing the correspondence between each pixel in the training depth image and one of the following: a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, and other segments, based on any of the abdominal CT images, abdominal MRI images, and abdominal MRA images. After the training step is performed, the 3D structure recognition device recognizes the 3D structure of the patient's abdominal surface, the depth image generation unit generates a depth image of the patient's abdominal surface from the 3D structure recognized by the 3D structure recognition device, and in the vascular segment discrimination step, the program estimates whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or other segments by using the trained deep learning model.

[0009] One aspect of the present invention comprises a three-dimensional structure recognition device that recognizes the three-dimensional structure of a patient's abdominal surface, a vascular segment discrimination device that uses a deep learning model to determine the aortic segment of a patient whose three-dimensional structure of the abdominal surface has been recognized by the three-dimensional structure recognition device, and a learning dataset generation unit that generates a learning dataset used to train the deep learning model, wherein before the training of the deep learning model is performed, the learning dataset generation unit generates a three-dimensional structure of the abdominal surface for training from any of the abdominal CT images, abdominal MRI images, and abdominal MRA images of a person other than the patient whose aortic segment is to be determined by the vascular segment discrimination device, and based on any of the abdominal CT images, abdominal MRI images, and abdominal MRA images, the learning three-dimensional structure of the abdominal surface This is a vascular segment discrimination system in which a training dataset is generated showing the correspondence between each point on the above and one of the following: a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, and other segments; after the deep learning model is trained using the training dataset generated by the training dataset generation unit, the 3D structure recognition device recognizes the 3D structure of the patient's abdominal surface, and the vascular segment discrimination device estimates whether each point on the 3D structure of the patient's abdominal surface recognized by the 3D structure recognition device corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or other segments by using the trained deep learning model.

[0010] One aspect of the present invention is a vascular segment discrimination method for a vascular segment discrimination system comprising: a 3D structure recognition device that recognizes the 3D structure of a patient's abdominal surface; a vascular segment discrimination device that uses a deep learning model to determine the aortic segment of a patient whose 3D structure of the abdominal surface has been recognized by the 3D structure recognition device; and a learning dataset generation unit that generates a learning dataset used to train the deep learning model, wherein, before the training of the deep learning model is performed, the learning dataset generation unit generates a 3D structure of the abdominal surface for learning from any of the abdominal CT images, abdominal MRI images, and abdominal MRA images of a person other than the patient whose aortic segment is to be determined by the vascular segment discrimination device, and based on any of the abdominal CT images, abdominal MRI images, and abdominal MRA images, each point on the 3D structure of the abdominal surface for learning and the Zo of the aorta The vascular segment discrimination method comprises: a training dataset generation step of generating a training dataset that shows the correspondence between a first vascular segment corresponding to ne1, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, and any of the other segments; a three-dimensional structure recognition step in which the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface after the deep learning model has been trained using the training dataset generated in the training dataset generation step; and a vascular segment discrimination step in which the vascular segment discrimination device estimates whether each point on the three-dimensional structure of the patient's abdominal surface recognized in the three-dimensional structure recognition step corresponds to any of the first vascular segment, the second vascular segment, the third vascular segment, and any of the other segments by using the trained deep learning model.

[0011] One aspect of the present invention is a program for causing a computer constituting a vascular segment discrimination device in a vascular segment discrimination system comprising a 3D structure recognition device for recognizing the 3D structure of a patient's abdominal surface and a learning dataset generation unit for generating a learning dataset used to train a deep learning model, to perform a learning step of training a deep learning model using the learning dataset generated by the learning dataset generation unit, and a vascular segment discrimination step, wherein the learning dataset generation unit generates a 3D structure of the abdominal surface for learning from any of abdominal CT images, abdominal MRI images, and abdominal MRA images of a person other than the patient whose aortic segment is to be discriminated by the vascular segment discrimination device, and the abdominal CT images, abdominal MRI images The program generates a training dataset based on either the abdominal MRA image or the training image, showing the correspondence between each point on the three-dimensional structure of the abdominal surface for training and one of the following: a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, and other segments. After the training step is performed, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface, and in the vascular segment discrimination step, the program estimates whether each point on the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or other segments by using the trained deep learning model. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a vascular segmentation discrimination system, a vascular segmentation discrimination method, and a program that can easily identify Zone 1, Zone 2, and Zone 3 of a patient's aorta even in situations where X-ray fluoroscopy or ultrasound equipment cannot be used, thereby supporting the realization of REBOA. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows an example of the blood vessel classification system 1 of the first embodiment. [Figure 2]This diagram illustrates an example of the processing performed by the training dataset generation unit 13. [Figure 3] This is a flowchart illustrating an example of the process performed in the blood vessel classification system 1 of the first embodiment. [Figure 4] This diagram conceptually illustrates the training of a semantic segmentation model using a training dataset in the inventor's research. [Figure 5] This diagram conceptually illustrates the validation (Test) of a trained semantic segmentation model in the inventor's research. [Figure 6] This figure shows the results of evaluating the similarity between Zone 1, Zone 2, and Zone 3, estimated using a pre-trained semantic segmentation model in the inventor's research, using the Dice coefficient and the Jaccard coefficient. [Figure 7] This figure shows the position error of the boundary line between Zone 1 and Zone 2, and the position error of the boundary line between Zone 2 and Zone 3, estimated using a pre-trained semantic segmentation model in the inventor's research. [Figure 8] This figure shows four examples in the inventor's research where the positional error of the aortic segment estimated using a trained semantic segmentation model was less than 10 mm. [Figure 9] This figure shows four examples in the inventor's research where the positional error of the aortic segment estimated using a trained semantic segmentation model was greater than 20 mm. [Figure 10] This diagram conceptually illustrates a technique that uses HoloLens 2 to project zone estimation results onto a patient's body surface in real time. [Figure 11] This figure shows an example of the blood vessel classification system 2 of the third embodiment. [Figure 12] This is a flowchart illustrating an example of the process performed in the blood vessel classification system 2 of the third embodiment.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, with reference to the drawings, embodiments of the blood vessel classification discrimination system, blood vessel classification discrimination method, and program of the present invention will be described.

[0015] [First Embodiment] FIG. 1 is a diagram showing an example of the blood vessel classification discrimination system 1 of the first embodiment. In the example shown in FIG. 1, the blood vessel classification discrimination system 1 of the first embodiment supports the implementation of REBOA for a patient by an operator in a situation where X-ray fluoroscopy or an ultrasonic device cannot be used, such as in an emergency. The blood vessel classification discrimination system 1 includes a three-dimensional structure recognition device 11, a depth image generation unit 12, a learning data set generation unit 13, a blood vessel classification discrimination device 14, and a visualization device 15. The three-dimensional structure recognition device 11 recognizes the three-dimensional structure of the patient's abdominal surface. The three-dimensional structure recognition device 11 recognizes the three-dimensional structure of the patient's abdominal surface and generates three-dimensional data of the patient's abdominal surface by using, for example, the same technology as the "LiDAR scanner" described on the website shown by the following URL. https: / / bablishe.com / about-lidar-scanner-of-ipad-pro-2020 /

[0016] As the three-dimensional structure recognition device 11, for example, products described on the websites shown by the following URLs can be used. https: / / www.apple.com / jp / ipad-pro / specs / https: / / www.microsoft.com / ja-jp / hololens / hardware

[0017] The former product is used in the procedure described on the website shown by the following URL, for example, to recognize the three-dimensional structure of the external world. https: / / prono82.com / 2020 / 09 / 28 / apple%E7%A4%BElidar%E6%90%AD%E8%BC%89ipad-pro%E3%81%AB%E3%82%88%E3%82%8B3d%E7%82%B9%E7%BE%A4%E3%82%B9%E3%82%AD%E3%83%A3%E3%83%B3%E3%82%A2%E3%83%97%E3%83%AA%E3%81%AE%E3%83%AA%E3%83%AA%E3%83%BC%E3%82%B9 /

[0018] Furthermore, the latter product is used according to the procedure described on the website, for example, at the URL below, to recognize the three-dimensional structure of the external environment. https: / / docs.microsoft.com / ja-jp / windows / mixed-reality / design / spatial-mapping https: / / zenn.dev / hiromu / articles / 20210421-scene-understanding

[0019] The depth image generation unit 12 generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device 11. A depth image is an image that is color-coded according to distance, as shown in the website at the URL below, for example. In other words, the depth image generation unit 12 generates a two-dimensional image of the patient's abdominal surface, color-coded according to the distance from the three-dimensional structure recognition device 11. The depth image generation unit 12 generates the depth image of the patient's abdominal surface by using the technology shown in the website at the URL below, for example. https: / / www.cit.nihon-u.ac.jp / laboratorydata / kenkyu / kouennkai / reference / No.47 / pdf / 2-53.pdf

[0020] The training dataset generation unit 13 generates a training dataset. The vascular segmentation discrimination device 14 uses a deep learning model 14A to determine the aortic segment of a patient whose three-dimensional structure on the abdominal surface has been recognized by the three-dimensional structure recognition device 11.

[0021] Figure 2 is a diagram illustrating an example of the processing performed by the training dataset generation unit 13. In the example shown in Figure 2, the training dataset generation unit 13 generates a 3D structure of the abdominal surface for training (shown in the upper center of Figure 2) from an abdominal CT (Computed Tomography) image (shown on the left side of Figure 2) of a person different from the patient whose aortic segment is being determined by the vascular segment discrimination device 14. In other words, the training dataset generation unit 13 performs the process indicated by "Render" in Figure 2. The 3D structure of the abdominal surface for training generated in this process will be similar to the recognition result (3D structure) obtained if the 3D structure recognition device 11 were to recognize the 3D structure of the abdominal surface of that person (i.e., the subject of the abdominal CT image). Furthermore, the training dataset generation unit 13 generates a training depth image (the image shown in the upper right of Figure 2) from the 3D structure of the abdominal surface for training, which is generated by the process shown as "Render" in Figure 2. In other words, the training dataset generation unit 13 executes the process shown as "Project" in Figure 2.

[0022] Furthermore, the training dataset generation unit 13 generates information (shown in the lower center of Figure 2) indicating the correspondence between each pixel in the training depth image (shown in the upper right of Figure 2) and one of the following: the first vascular segment corresponding to Zone 1 of the aorta, the second vascular segment corresponding to Zone 2 of the aorta, the third vascular segment corresponding to Zone 3 of the aorta, and other segments (shown as "Out of Zones" in Figure 2). In other words, the training dataset generation unit 13 performs the process shown as "Label" in Figure 2. Furthermore, the training dataset generation unit 13 generates a training dataset, which is a pair of training depth images (shown in the upper right of Figure 2) generated by the process shown as "Project" in Figure 2, and information (shown in the lower center of Figure 2) generated by the process shown as "Label" in Figure 2, as training data used for training the deep learning model 14A. In the example shown in Figure 2, abdominal CT images of a person other than the patient targeted for aortic segmentation by the vascular segmentation discrimination device 14 are used to generate the training dataset. However, in other examples, abdominal MRI (Magnetic Resonance Imaging) images of a person other than the patient targeted for aortic segmentation by the vascular segmentation discrimination device 14 may be used to generate the training dataset, and in yet another example, abdominal MRA (magnetic resonance angiography) images of a person other than the patient targeted for aortic segmentation by the vascular segmentation discrimination device 14 may be used to generate the training dataset.

[0023] In the example shown in Figure 1, the blood vessel classification device 14 comprises a learning unit 141 and an estimation unit 142. The learning unit 141 trains a deep learning model 14A using a training dataset (for example, the image shown in the upper right of Figure 2 and the information shown in the lower center of Figure 2) generated by the training dataset generation unit 13. In detail, considering that the risk of organ ischemia increases if the estimation accuracy of the second vascular segment corresponding to Zone 2 of the aorta is low when REBOA is performed, the learning unit 141 trains the deep learning model 14A so that the estimation accuracy of the second vascular segment corresponding to Zone 2 of the aorta using the trained deep learning model 14A is above a predetermined threshold. In the blood vessel classification system 1 of the first embodiment, a segmentation model is used as the deep learning model 14A. In detail, in the first example of the blood vessel classification system 1 of the first embodiment, a semantic segmentation model described on the website indicated by the URL below is used as the deep learning model 14A. https: / / jp.mathworks.com / content / dam / mathworks / mathworks-dot-com / company / events / webinar-cta / 2459280_Basics_of_semantic_segmentation.pdf https: / / qiita.com / fujiya228 / items / ea30dac6ef827d608a56

[0024] In a second example of the blood vessel classification system 1 of the first embodiment, a segmentation model other than semantic segmentation, such as instance segmentation or panoptic segmentation, as described on the website indicated at the URL below, may be used as the deep learning model 14A. https: / / www.skillupai.com / blog / tech / segmentation1 /

[0025] In the example shown in Figure 1, the estimation unit 142 uses a trained deep learning model 14A to estimate whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit 12 belongs to one of the following categories: the first vascular segment corresponding to Zone 1 of the aorta, the second vascular segment corresponding to Zone 2 of the aorta, the third vascular segment corresponding to Zone 3 of the aorta, or other segments (i.e., segments that do not fall under any of the first, second, or third vascular segments). In other words, in the example shown in Figure 1, after the deep learning model 14A is trained using the training dataset generated by the training dataset generation unit 13, the 3D structure recognition device 11 recognizes the 3D structure of the patient's abdominal surface. The depth image generation unit 12 then generates a depth image of the patient's abdominal surface from the 3D structure recognized by the 3D structure recognition device 11. Furthermore, the estimation unit 142 of the vascular segment discrimination device 14 uses the trained deep learning model 14A to estimate whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit 12 belongs to one of the following segments: the first vascular segment corresponding to Zone 1 of the aorta, the second vascular segment corresponding to Zone 2 of the aorta, the third vascular segment corresponding to Zone 3 of the aorta, or other segments (shown as "Out of Zones" in Figure 2).

[0026] The visualization device 15 generates a virtual image projected onto the patient's abdominal surface the estimation result of the patient's aortic segmentation by the vascular segmentation discrimination device 14. The visualization device 15 comprises a virtual image generation unit 15A and a virtual image presentation unit 15B. The virtual image generation unit 15A generates a virtual image projecting the patient's first vascular zone (Zone 1), second vascular zone (Zone 2), and third vascular zone (Zone 3) onto the patient's abdominal surface. The virtual image presentation unit 15B presents the virtual image generated by the virtual image generation unit 15A to the user of the vascular zone discrimination system 1 (e.g., the surgeon).

[0027] In the first example of the vascular segment discrimination system 1 of the first embodiment, the visualization device 15 projects the patient's first vascular segment (Zone 1), second vascular segment (Zone 2), and third vascular segment (Zone 3) onto the patient's abdominal surface by using, for example, Non-Patent Document 2, Non-Patent Document 3, and the technology described on the website indicated by the URL below. https: / / prtimes.jp / main / html / rd / p / 000000073.000004318.html The projection method of the first example of the blood vessel classification system 1 of the first embodiment is, for example, a method using a reference external marker, or a method of overlaying information onto the human body by manually aligning it.

[0028] In a second example of the blood vessel classification system 1 of the first embodiment, the visualization device 15 is, for example, a product listed on the website indicated by the URL below. https: / / www.apple.com / jp / ipad-pro / specs / In a second example of the vascular segment discrimination system 1 of the first embodiment, the visualization device 15 uses technology similar to that described on the website indicated by the URL below to directly use a real three-dimensional object (i.e., the patient's abdominal surface) as a marker, that is, to project the patient's first vascular segment (Zone 1), second vascular segment (Zone 2), and third vascular segment (Zone 3) directly onto the patient's abdominal surface without any external intervention. https: / / library.vuforia.com / features / objects / object-reco.html https: / / www.youtube.com / watch?v=jbaUDMvv2Zw

[0029] In the third example of the blood vessel classification system 1 of the first embodiment, the visualization device 15 is, for example, a product listed on the website indicated by the URL below. https: / / www.microsoft.com / ja-jp / hololens / hardware In the third example of the vascular segment discrimination system 1 of the first embodiment, the visualization device 15 uses technology similar to that described on the website indicated by the URL below to directly use a real three-dimensional object (i.e., the patient's abdominal surface) as a marker, that is, to project the patient's first vascular segment (Zone 1), second vascular segment (Zone 2), and third vascular segment (Zone 3) directly onto the patient's abdominal surface without any external intervention. https: / / docs.microsoft.com / ja-jp / azure / object-anchors / overview https: / / car.watch.impress.co.jp / docs / news / 1278295.html https: / / library.vuforia.com / features / objects / object-reco.html https: / / www.youtube.com / watch?v=jbaUDMvv2Zw

[0030] In the third example of the vascular segment discrimination system 1 of the first embodiment, the visualization device 15 generates a virtual image projecting the patient's first vascular segment (Zone 1), second vascular segment (Zone 2), and third vascular segment (Zone 3) onto the patient's abdominal surface by using technology similar to that described on the website indicated by the URL below, and presents the virtual image to the user of the vascular segment discrimination system 1 (e.g., the surgeon). https: / / www.ogis-ri.co.jp / otc / hiroba / technical / point-DX / part6.html https: / / www.tattichan.work / entry / 2019 / 12 / 06 / Reprojection%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6%E6%95%B4%E7%90%86%E3%81%99%E3%82%8B https: / / www.businessinsider.jp / post-185961 https: / / hanada-sekkei.co.jp / armrvr https: / / www.youtube.com / watch?v=QZiQ71EDF-o

[0031] Figure 3 is a flowchart illustrating an example of the process performed in the blood vessel classification system 1 of the first embodiment. In the example shown in Figure 3, in step S11, the training dataset generation unit 13 generates a training dataset to be used for training the deep learning model 14A. In detail, in step S11A, the training dataset generation unit 13 generates a 3D structure of the abdominal surface for training from either an abdominal CT image, abdominal MRI image, or abdominal MRA image of a person other than the patient whose aortic segment is being determined by the vascular segment discrimination device 14. Next, in step S11B, the training dataset generation unit 13 generates a training depth image from the 3D structure of the abdominal surface for training that was generated in step S11A. Next, in step S11C, the training dataset generation unit 13 generates a training dataset based on either an abdominal CT image, an abdominal MRI image, or an abdominal MRA image, showing the correspondence between each pixel in the training depth image and one of the following: a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, and other segments (i.e., segments that do not fall under any of the first, second, or third vascular segments).

[0032] Next, in step S12, the learning unit 141 of the blood vessel classification device 14 trains the deep learning model 14A using the training dataset generated in step S11. Next, in step S13, the 3D structure recognition device 11 recognizes the 3D structure of the abdominal surface of the patient whose aortic segment is to be determined by the vascular segment discrimination device 14. Next, in step S14, the depth image generation unit 12 generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized in step S13. Next, in step S15, the vascular segmentation device 14 uses the trained deep learning model 14A, which was trained in step S12, to determine the aortic segment of the patient whose three-dimensional structure of the abdominal surface was recognized in step S13. In detail, in step S15, the estimation unit 142 of the vascular segment discrimination device 14 uses a trained deep learning model 14A to estimate whether each pixel in the depth image of the patient's abdominal surface generated in step S14 belongs to one of the following categories: the first vascular segment corresponding to Zone 1 of the aorta, the second vascular segment corresponding to Zone 2 of the aorta, the third vascular segment corresponding to Zone 3 of the aorta, or other segments (i.e., segments that do not fall under any of the first, second, or third vascular segments). Next, in step S16, the virtual image generation unit 15A of the visualization device 15 generates a virtual image projecting the patient's first vascular zone (Zone 1), second vascular zone (Zone 2), and third vascular zone (Zone 3) onto the patient's abdominal surface. The virtual image presentation unit 15B of the visualization device 15 then presents the virtual image generated by the virtual image generation unit 15A to the user of the vascular zone discrimination system 1 (e.g., the surgeon).

[0033] Therefore, according to the vascular segmentation discrimination system 1 of the first embodiment, a user of the vascular segmentation discrimination system 1 (e.g., a surgeon) can easily identify Zone 1, Zone 2, and Zone 3 of the patient's aorta even in situations where X-ray fluoroscopy or ultrasound equipment cannot be used. As a result, the feasibility of REBOA in situations where X-ray fluoroscopy or ultrasound equipment cannot be used can be improved.

[0034] As described above, in the example shown in Figure 1, the 3D structure recognition device 11 recognizes the 3D structure of the abdominal surface of the patient whose aortic segment is to be determined by the vascular segment discrimination device 14, using the same technology as a LiDAR scanner, and generates 3D data of the patient's abdominal surface. In other examples, the 3D structure recognition device 11 may have the function of generating a 3D structure of the abdominal surface of a patient from any of the abdominal CT images, abdominal MRI images, or abdominal MRA images of a patient whose aortic segment is to be determined by the vascular segment determination device 14, as shown in the example in Figure 2.

[0035] As described above, in the example shown in Figure 1, the vascular segmentation discrimination device 14 outputs information used by the visualization device 15 to generate a virtual image in which the first vascular segment (Zone 1), the second vascular segment (Zone 2), and the third vascular segment (Zone 3) are projected onto the patient's abdominal surface as the estimated result of the patient's aortic segmentation. In other examples, the vascular segmentation discrimination device 14 may output, for example numerically, the length from the patient's landmark (i.e., a site suitable for puncture (i.e., the catheter insertion site)) to the patient's first vascular segment (Zone 1), the length from the patient's landmark to the patient's second vascular segment (Zone 2), and the length from the patient's landmark to the patient's third vascular segment (Zone 3) as the estimated results of the patient's aortic segmentation.

[0036] Furthermore, in other examples, the visualization device 15 may be a product listed on the website indicated by the URL below. https: / / www.apple.com / jp / ipad-pro / specs / Furthermore, in this example, the length from the patient's landmark (catheter insertion site) to the patient's first vascular zone (Zone 1), the length from the patient's landmark to the patient's second vascular zone (Zone 2), and the length from the patient's landmark to the patient's third vascular zone (Zone 3) are measured by using the technology described on the product's website at the URL below. https: / / support.apple.com / ja-jp / guide / ipad / ipad8ac2cfea / ipados Specifically, in this example, the user of the vascular segmentation discrimination system 1 (e.g., the surgeon) specifies a point corresponding to the patient's landmark (catheter insertion site) and a point corresponding to the patient's first vascular segment (Zone 1) on the virtual image presented to the user of the vascular segmentation discrimination system 1. The length between these two points is then measured and presented to the user of the vascular segmentation discrimination system 1. Therefore, the user of the vascular segmentation discrimination system 1 can determine the length of the catheter to be inserted into the patient by using the presented results.

[0037] In other examples, the visualization device 15 may be a product listed on the website indicated by the URL below. https: / / www.microsoft.com / ja-jp / hololens / hardware Furthermore, in this example, the length from the patient's landmark (catheter insertion site) to the patient's first vascular zone (Zone 1), the length from the patient's landmark to the patient's second vascular zone (Zone 2), and the length from the patient's landmark to the patient's third vascular zone (Zone 3) are measured by using the technology described on the product's website at the URL below. https: / / www.windowscentral.com / hololens-gets-virtual-tape-measure In this example as well, the user of the vascular segmentation discrimination system 1 (e.g., the operator) specifies a point corresponding to the patient's landmark (catheter insertion site) and a point corresponding to the patient's first vascular segment (Zone 1) on the virtual image presented to the user of the vascular segmentation discrimination system 1. The length between these two points is then measured and presented to the user of the vascular segmentation discrimination system 1. Therefore, the user of the vascular segmentation discrimination system 1 can determine the length of the catheter to be inserted into the patient by using the presented results.

[0038] In the two examples mentioned above, the balloon catheter inserted into the patient is one that has length marks, such as those shown on the website at the URL below. https: / / prytimemedical.com / product / er-reboa-plus-catheter /

[0039] In yet another example, the training dataset generation unit 13 may generate a training dataset for learning the course (tortuosity) of the blood vessel into which the balloon catheter is inserted, based on either an abdominal CT image (the image shown on the left in Figure 2), an abdominal MRI image, or an abdominal MRA image. In this example, the learning unit 141 of the vascular segmentation discrimination device 14 trains a deep learning model 14A using its training dataset. The estimation unit 142 of the vascular segmentation discrimination device 14 estimates the course (tortuosity) of the patient's blood vessels into which the balloon catheter will be inserted, using the trained deep learning model 14A. Furthermore, when a user of the vascular segmentation discrimination system 1 (e.g., an operator) specifies a point corresponding to the patient's landmark (catheter insertion site) and a point corresponding to, for example, the patient's first vascular segment (Zone 1) on the virtual image presented to the user of the vascular segmentation discrimination system 1, the length along the blood vessel between those two points is estimated and output based on the straight-line distance between those two points and the estimation result of the estimation unit 142 of the vascular segmentation discrimination device 14. In a modified version of this example, the training dataset generation unit 13 may generate a training dataset based on an abdominal CT image (the image shown on the left in Figure 2), an abdominal MRI image, or an abdominal MRA image to learn the length along the blood vessels between a point corresponding to a patient's landmark (catheter insertion site) and a point corresponding to, for example, the patient's first vascular segment (Zone 1).

[0040] In another example, the visualization device 15 may generate a virtual image of the balloon of a balloon catheter inserted into the patient's blood vessel, in a state where it is positioned in one of the patient's first vascular zone (Zone 1), second vascular zone (Zone 2), or third vascular zone (Zone 3) (i.e., the balloon is inflated), and present it to the user of the vascular zone discrimination system 1 (e.g., the operator). In this example, the user of the vascular zone discrimination system 1 can easily visualize the state in which the balloon is actually positioned in one of the patient's first vascular zone (Zone 1), second vascular zone (Zone 2), or third vascular zone (Zone 3).

[0041] <Examples> In a study verifying the vascular segmentation system 1 of the first embodiment, the inventors developed a deep learning model 14A that can instantly identify the aortic segment from the patient's body surface. Specifically, the inventors combined segmentation technology based on advanced deep learning with patient depth information obtained from the reflection of infrared light emitted from a camera equivalent to a 3D structure recognition device 11 to estimate the aortic segment from the 3D information of the patient's body surface.

[0042] The inventor's expertise lies in diagnostic radiology technology, machine learning, and mixed reality. Previously, research has focused on visualizing invisible radiation exposure using mixed reality. The medical applications of mixed reality are advancing, with applications in three-dimensional organ visualization and telemedicine beginning to emerge. The inventor believes that visualizing other unknown information in an intuitive way can support medical care, and therefore focused on REBOA, a life-saving medical procedure performed even when X-ray fluoroscopy equipment is physically unusable. By using deep learning technology to accurately estimate the zone divisions to be visualized, this research aims to achieve a level of accuracy and immersion far superior to conventional methods, significantly improving the quality of life-saving medical care.

[0043] The inventors constructed a model that enables zone estimation using deep learning-based segmentation techniques, and in order to establish a trained model that enables highly accurate estimation, they performed the following: The training dataset plays a crucial role in deep learning, and its quality significantly impacts the overall research outcome. Therefore, the inventors first focused on preparing the training dataset. Three-dimensional images of the body surface, created from abdominal CT images, were used for training. The inventors primarily used data held by Teikyo University Hospital as the CT images. Furthermore, to avoid bias in the training data, the inventors also utilized open-access CT image databases. The inventors, who are also radiological technologists, performed segmentation of the training data and used it as training labels. In addition, the inventors collected age and weight information in case the prediction accuracy was insufficient with only body surface images, and added it to the training as needed.

[0044] The inventors trained a semantic segmentation model using a training dataset. During training, the inventors appropriately searched for hyperparameters and modified the network structure to construct an optimal trained model. The inventors comprehensively evaluated the trained model using reproducibility, precision, and Dice coefficients. In particular, they focused on the reproducibility of Zone 2, which has a high risk of organ ischemia, and aimed to create a model that can accurately identify high-risk regions.

[0045] Figure 4 is a conceptual diagram illustrating the training of a semantic segmentation model using a training dataset in the inventor's research. Figure 5 is a conceptual diagram illustrating the validation (Test) of a trained semantic segmentation model in the inventor's research. Figure 6 is a diagram showing the results of evaluating the similarity of Zone 1, Zone 2, and Zone 3 estimated using a trained semantic segmentation model in the inventor's research, using the Dice coefficient and the Jaccard coefficient. Figure 7 is a diagram showing the position error of the boundary line between Zone 1 and Zone 2, the position error of the boundary line between Zone 2 and Zone 3, etc., estimated using a trained semantic segmentation model in the inventor's research. Figure 8 is a diagram showing four examples in the inventor's research where the position error of the aortic segment estimated using a trained semantic segmentation model was less than 10 mm. Figure 9 is a diagram showing four examples in the inventor's research where the position error of the aortic segment estimated using a trained semantic segmentation model was greater than 20 mm.

[0046] Furthermore, the inventors have taken the following steps to develop an application that allows for the intuitive visualization of Zones estimated using a trained model, utilizing mixed reality that seamlessly blends reality and virtual reality. In this research, the inventors developed an application that runs on an iPad Pro ("iPad" is a registered trademark) (manufactured by Apple Inc.), a tablet device, and estimates and visualizes zones from a trained model. The iPad Pro is equipped with an infrared camera, enabling 3D measurement of the real world. Three-dimensional information of the patient's body surface captured by this infrared camera is input into the trained model, and the zone estimation results are projected in real time onto the body surface captured by a regular camera. As a result, users of the vascular zone classification system 1 (e.g., surgeons) can visualize zone classifications simply by holding the iPad Pro over the human body. In this research, the iPad Pro functions as both a 3D structure recognition device 11 and a visualization device 15.

[0047] Furthermore, the inventors developed an application in their research that operates on a headset-type terminal, HoloLens 2 ("HOLOLENS" is an internationally registered trademark) (manufactured by Microsoft), and estimates and visualizes zones from a trained model. Like the iPad Pro, HoloLens 2 is equipped with an infrared camera, enabling tertiary measurement of the real world. Three-dimensional information of the body surface is used as input to the trained model, and the zone estimation results are projected in real time onto the body surface visible through the see-through HoloLens 2. This allows users of the vascular segmentation discrimination system 1 (e.g., surgeons) to visualize zone divisions simply by wearing HoloLens 2 and looking at the patient. In this research, HoloLens 2 functions as both a 3D structure recognition device 11 and a visualization device 15. Because the HoloLens 2 has a different operating system and hardware configuration than the iPad Pro, modifications to the surface-level parts of the application are necessary. However, the core part of the system related to zone classification estimation can be shared (that is, it is possible to use either the iPad Pro or the HoloLens 2 as the visualization device 15).

[0048] Figure 10 is a diagram conceptually illustrating a technique that uses HoloLens 2 to project zone estimation results onto a patient's body surface in real time.

[0049] As a result of the research described above, the inventors have found that it is possible to estimate zone divisions from 3D images of a patient's body surface with sufficiently high accuracy. Specifically, the inventors constructed a dataset from a small portion of the CT images to be used and performed deep learning-based segmentation tests. The results showed that the reproducibility of Zone 1 was up to 0.99, Zone 2 was up to 0.95, and Zone 3 was up to 0.98. Based on the knowledge and experience gained from previous research, the inventors have been investigating implementation methods for visualization using mixed reality, and are on track to realize a technology that projects Zones onto the patient's body surface (i.e., to operate it as a vascular segmentation discrimination system 1 in a real device).

[0050] [Second Embodiment] The following describes a second embodiment of the blood vessel classification system, blood vessel classification method, and program of the present invention. The vascular classification system 1 of the second embodiment is configured in the same way as the vascular classification system 1 of the first embodiment described above, except for the points described later. Therefore, the vascular classification system 1 of the second embodiment can achieve the same effects as the vascular classification system 1 of the first embodiment described above, except for the points described later.

[0051] As described above, in the blood vessel classification system 1 of the first embodiment, a segmentation model is used as the deep learning model 14A. On the other hand, in the blood vessel classification system 1 of the second embodiment, a pseudo-image generation technique such as GAN (Generative Adversarial Network), as shown on the website indicated at the URL below, is used as the deep learning model 14A. https: / / ledge.ai / gan / https: / / www.nedo.go.jp / news / press / AA5_101472.html

[0052] [Third Embodiment] The following describes a third embodiment of the blood vessel classification system, blood vessel classification method, and program of the present invention. The vascular classification system 2 of the third embodiment is configured in the same way as the vascular classification system 1 of the first embodiment described above, except for the points described later. Therefore, the vascular classification system 2 of the third embodiment can achieve the same effects as the vascular classification system 1 of the first embodiment described above, except for the points described later.

[0053] Figure 11 shows an example of the blood vessel classification system 2 of the third embodiment. In the example shown in Figure 11, the vascular segmentation discrimination system 2 of the third embodiment, like the vascular segmentation discrimination system 1 of the first embodiment, supports the realization of REBOA by the operator on the patient in situations where X-ray fluoroscopy or ultrasound equipment cannot be used, such as in emergencies. The vascular segmentation discrimination system 2 comprises a 3D structure recognition device 21, a learning data set generation unit 23, a vascular segmentation discrimination device 24, and a visualization device 25. The 3D structure recognition device 21 recognizes the 3D structure of the patient's abdominal surface, similar to the 3D structure recognition device 11 shown in Figure 1.

[0054] The training dataset generation unit 23 generates a training dataset. The vascular segmentation discrimination device 24 uses a deep learning model 24A to determine the aortic segment of a patient whose three-dimensional structure on the abdominal surface has been recognized by the three-dimensional structure recognition device 21.

[0055] In an example of the vascular segmentation discrimination system 2 of the third embodiment, the learning dataset generation unit 23 generates a three-dimensional structure of the abdominal surface for learning from either an abdominal CT image, abdominal MRI image, or abdominal MRA image of a person other than the patient whose aortic segmentation is to be discriminated by the vascular segmentation discrimination device 24.

[0056] Furthermore, the training dataset generation unit 23 generates information based on either abdominal CT images, abdominal MRI images, or abdominal MRA images, showing the correspondence between each point on the three-dimensional structure of the abdominal surface for training and one of the following: the first vascular segment corresponding to Zone 1 of the aorta, the second vascular segment corresponding to Zone 2 of the aorta, the third vascular segment corresponding to Zone 3 of the aorta, and other segments. Furthermore, the training dataset generation unit 23 generates a training dataset, which is a combination of the three-dimensional structure of the abdominal surface for training and the information described above, as training data to be used for training the deep learning model 24A.

[0057] In the example shown in Figure 11, the blood vessel classification device 24 comprises a learning unit 241 and an estimation unit 242. The learning unit 241 trains the deep learning model 24A using the training dataset generated by the training dataset generation unit 23. In detail, considering that the risk of organ ischemia increases if the estimation accuracy of the second vascular segment corresponding to Zone 2 of the aorta is low when REBOA is performed, the learning unit 241 trains the deep learning model 24A so that the estimation accuracy of the second vascular segment corresponding to Zone 2 of the aorta using the trained deep learning model 24A is above a predetermined threshold. In the third embodiment of the vascular segmentation discrimination system 2, a segmentation model is used as the deep learning model 24A, similar to the vascular segmentation discrimination system 1 in the first embodiment. In detail, in the first example of the third embodiment of the blood vessel classification system 1, the semantic segmentation model described above is used as the deep learning model 24A.

[0058] In the second example of the blood vessel classification system 2 of the third embodiment, a segmentation model other than semantic segmentation, such as instance segmentation or panoptic segmentation as described above, may be used as the deep learning model 24A.

[0059] In the example shown in Figure 11, the estimation unit 242 uses a trained deep learning model 24A to estimate whether each point on the patient's 3D structure recognized by the 3D structure recognition device 21 belongs to one of the following categories: the first vascular section corresponding to Zone 1 of the aorta, the second vascular section corresponding to Zone 2 of the aorta, the third vascular section corresponding to Zone 3 of the aorta, or other categories (i.e., categories that do not fall under any of the first, second, or third vascular sections). In other words, in the example shown in Figure 11, after the deep learning model 24A is trained using the training dataset generated by the training dataset generation unit 23, the 3D structure recognition device 21 recognizes the 3D structure of the patient's abdominal surface. Furthermore, the estimation unit 242 of the vascular segment discrimination device 24 uses the trained deep learning model 24A to estimate whether each point on the patient's 3D structure recognized by the 3D structure recognition device 21 corresponds to one of the following segments: the first vascular segment corresponding to Zone 1 of the aorta, the second vascular segment corresponding to Zone 2 of the aorta, the third vascular segment corresponding to Zone 3 of the aorta, or another segment.

[0060] The visualization device 25 generates a virtual image projected onto the patient's abdominal surface the estimation result of the patient's aortic segmentation by the vascular segmentation discrimination device 24. The visualization device 25 comprises a virtual image generation unit 25A and a virtual image presentation unit 25B. The virtual image generation unit 25A generates a virtual image projecting the patient's first vascular zone (Zone 1), second vascular zone (Zone 2), and third vascular zone (Zone 3) onto the patient's abdominal surface. The virtual image presentation unit 25B presents the virtual image generated by the virtual image generation unit 25A to the user of the vascular zone discrimination system 2 (e.g., the surgeon).

[0061] In the first example of the vascular segment discrimination system 2 of the third embodiment, the visualization device 25 projects the patient's first vascular segment (Zone 1), second vascular segment (Zone 2), and third vascular segment (Zone 3) onto the patient's abdominal surface, similar to the first example of the vascular segment discrimination system 1 of the first embodiment.

[0062] In the second example of the third embodiment of the vascular segment discrimination system 2, the visualization device 25, similar to the second example of the first embodiment of the vascular segment discrimination system 1, directly uses a real three-dimensional object (i.e., the patient's abdominal surface) as a marker, that is, without any external intervention, it projects the patient's first vascular segment (Zone 1), second vascular segment (Zone 2), and third vascular segment (Zone 3) directly onto the patient's abdominal surface.

[0063] In the third example of the vascular segment discrimination system 2 of the third embodiment, the visualization device 25, similar to the third example of the vascular segment discrimination system 1 of the first embodiment, directly uses a real three-dimensional object (i.e., the patient's abdominal surface) as a marker, that is, without any external intervention, it projects the patient's first vascular segment (Zone 1), second vascular segment (Zone 2), and third vascular segment (Zone 3) directly onto the patient's abdominal surface.

[0064] In the third example of the vascular segment discrimination system 2 of the third embodiment, the visualization device 25 generates a virtual image projecting the patient's first vascular segment (Zone 1), second vascular segment (Zone 2), and third vascular segment (Zone 3) onto the patient's abdominal surface, similar to the third example of the vascular segment discrimination system 1 of the first embodiment, and presents the virtual image to the user of the vascular segment discrimination system 2 (e.g., the surgeon).

[0065] Figure 12 is a flowchart illustrating an example of the process performed in the blood vessel classification system 2 of the third embodiment. In the example shown in Figure 12, in step S21, the training dataset generation unit 23 generates a training dataset to be used for training the deep learning model 24A. In detail, in step S21A, the training dataset generation unit 23 generates a 3D structure of the abdominal surface for training from either an abdominal CT image, abdominal MRI image, or abdominal MRA image of a person other than the patient whose aortic segment is being determined by the vascular segment discrimination device 14. Next, in step S21C, the training dataset generation unit 13 generates a training dataset based on either an abdominal CT image, an abdominal MRI image, or an abdominal MRA image, showing the correspondence between each point on the three-dimensional structure of the abdominal surface for training and one of the following: the first vascular division corresponding to Zone 1 of the aorta, the second vascular division corresponding to Zone 2 of the aorta, the third vascular division corresponding to Zone 3 of the aorta, and other divisions (i.e., divisions that do not fall under any of the first, second, or third vascular divisions).

[0066] Next, in step S22, the learning unit 241 of the blood vessel classification device 24 trains the deep learning model 24A using the training dataset generated in step S21. Next, in step S23, the 3D structure recognition device 21 recognizes the 3D structure of the abdominal surface of the patient whose aortic segment is to be determined by the vascular segment discrimination device 24. Next, in step S25, the vascular segmentation device 14 uses the trained deep learning model 24A, which was trained in step S22, to determine the aortic segment of the patient whose three-dimensional structure of the abdominal surface was recognized in step S23. In detail, in step S25, the estimation unit 242 of the vascular segment discrimination device 24 uses a trained deep learning model 24A to estimate whether each point on the three-dimensional structure of the patient's abdominal surface, recognized in step S23, belongs to one of the following categories: the first vascular segment corresponding to Zone 1 of the aorta, the second vascular segment corresponding to Zone 2 of the aorta, the third vascular segment corresponding to Zone 3 of the aorta, or other segments (i.e., segments that do not fall under any of the first, second, or third vascular segments). Next, in step S26, the virtual image generation unit 25A of the visualization device 25 generates a virtual image projecting the patient's first vascular zone (Zone 1), second vascular zone (Zone 2), and third vascular zone (Zone 3) onto the patient's abdominal surface. The virtual image presentation unit 25B of the visualization device 25 then presents the virtual image generated by the virtual image generation unit 25A to the user of the vascular zone discrimination system 2 (e.g., the surgeon).

[0067] Therefore, according to the vascular segmentation discrimination system 2 of the third embodiment, users of the vascular segmentation discrimination system 2 (e.g., surgeons) can easily identify Zone 1, Zone 2, and Zone 3 of the patient's aorta even in situations where X-ray fluoroscopy or ultrasound equipment cannot be used. As a result, the feasibility of REBOA in situations where X-ray fluoroscopy or ultrasound equipment cannot be used can be improved.

[0068] As described above, in the example shown in Figure 11, the 3D structure recognition device 21 recognizes the 3D structure of the abdominal surface of the patient whose aortic segment is to be determined by the vascular segment discrimination device 24, using the same technology as a LiDAR scanner, and generates 3D data of the patient's abdominal surface. In other examples, the 3D structure recognition device 21 may have the function of generating a 3D structure of the abdominal surface of a patient from any of the abdominal CT images, abdominal MRI images, or abdominal MRA images of a patient whose aortic segment is to be determined by the vascular segment discrimination device 24, as shown in the example in Figure 2.

[0069] As described above, in the example shown in Figure 11, the vascular segmentation discrimination device 24 outputs information used by the visualization device 25 to generate a virtual image in which the first vascular segment (Zone 1), the second vascular segment (Zone 2), and the third vascular segment (Zone 3) are projected onto the patient's abdominal surface as the estimated result of the patient's aortic segmentation. In other examples, the vascular segmentation discrimination device 24 may output, for example numerically, the length from the patient's landmark (i.e., a site suitable for puncture (i.e., the catheter insertion site)) to the patient's first vascular segment (Zone 1), the length from the patient's landmark to the patient's second vascular segment (Zone 2), and the length from the patient's landmark to the patient's third vascular segment (Zone 3) as the estimated results of the patient's aortic segmentation.

[0070] [Fourth Embodiment] A fourth embodiment of the blood vessel classification system, blood vessel classification method, and program of the present invention will be described below. The vascular classification system 2 of the fourth embodiment is configured in the same way as the vascular classification system 2 of the third embodiment described above, except for the points described later. Therefore, the vascular classification system 2 of the fourth embodiment can achieve the same effects as the vascular classification system 2 of the third embodiment described above, except for the points described later.

[0071] As described above, in the third embodiment of the blood vessel classification system 2, a segmentation model is used as the deep learning model 24A. On the other hand, in the blood vessel classification system 2 of the fourth embodiment, pseudo-image generation techniques such as the above-mentioned GAN are used as the deep learning model 24A.

[0072] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and modifications can be made as appropriate without departing from the spirit of the present invention. The configurations described in each of the embodiments and examples above may be combined.

[0073] Furthermore, all or part of the blood vessel classification systems 1 and 2 in the above embodiments may be implemented by dedicated hardware, or by memory and a microprocessor. Furthermore, the blood vessel classification system 1 may consist of all or part of a memory and a CPU (central processing unit), and its functions may be realized by loading and executing programs into the memory to realize the functions of each part of the system. Alternatively, a program to implement all or part of the functions of the blood vessel classification system 1 may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed to perform the processing of each part. The term "computer system" here includes hardware such as the operating system and peripheral devices. Furthermore, if a WWW system is used, the "computer system" also includes the homepage provisioning environment (or display environment). Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording media" also includes those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. In addition, the above-mentioned programs may be for the purpose of realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with programs already recorded in the computer system. [Explanation of Symbols]

[0074] 1...Vascular blood vessel classification system, 11...3D structure recognition device, 12...Depth image generation unit, 13...Training dataset generation unit, 14...Vascular blood vessel classification device, 14A...Deep learning model, 141...Learning unit, 142...Estimation unit, 15...Visualization device, 15A...Virtual image generation unit, 15B...Virtual image presentation unit, 2...Vascular blood vessel classification system, 21...3D structure recognition device, 23...Training dataset generation unit, 24...Vascular blood vessel classification device, 24A...Deep learning model, 241...Learning unit, 242...Estimation unit, 25...Visualization device, 25A...Virtual image generation unit, 25B...Virtual image presentation unit

Claims

1. A three-dimensional structure recognition device that recognizes the three-dimensional structure of the abdominal surface of a patient, A depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device, A vascular segmentation discrimination device that uses a deep learning model to determine the aortic segment of a patient whose three-dimensional structure on the abdominal surface has been recognized by the three-dimensional structure recognition device, The system includes a training dataset generation unit that generates a training dataset used for training the deep learning model, Before the training of the deep learning model is performed, the training dataset generation unit performs the following: A three-dimensional structure of the abdominal surface for learning purposes is generated from either an abdominal CT (Computed Tomography) image, an abdominal MRI (Magnetic Resonance Imaging) image, or an abdominal MRA (magnetic resonance angiography) image of a person other than the patient whose aortic segment is being determined by the aforementioned vascular segment determination device. A learning depth image is generated from the three-dimensional structure of the abdominal surface used for learning. Based on the abdominal CT image, the abdominal MRI image, and the abdominal MRA image, a training dataset is generated that shows the correspondence between each pixel in the training depth image and one of the following: a first vascular segment corresponding to Zone 1 of the aorta, a second vascular segment corresponding to Zone 2 of the aorta, a third vascular segment corresponding to Zone 3 of the aorta, and other segments. After the deep learning model is trained using the training dataset generated by the training dataset generation unit, The three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface, The depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device. The vascular segment discrimination device estimates whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit belongs to the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments, by using the trained deep learning model. Vascular classification system.

2. The system includes a visualization device that generates a virtual image projecting the estimated aortic segment of the patient, obtained by the aforementioned vascular segment discrimination device, onto the patient's abdominal surface. The vascular classification system according to claim 1.

3. The aforementioned blood vessel classification device is A learning unit that trains the deep learning model using the learning dataset generated by the learning dataset generation unit, The system includes an estimation unit that uses the trained deep learning model to estimate whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments, The learning unit trains the deep learning model so that the estimation accuracy of the second blood vessel segment using the trained deep learning model is equal to or greater than a predetermined threshold. The vascular classification system according to claim 1.

4. The three-dimensional structure recognition device has the function of generating a three-dimensional structure of the patient's abdominal surface from any of the patient's abdominal CT images, abdominal MRI images, or abdominal MRA images. The vascular classification system according to claim 1.

5. The aforementioned vascular segmentation device estimates the patient's aortic segmentation as follows: The length from the patient's landmark to the patient's first vascular segment, the length from the patient's landmark to the patient's second vascular segment, and the length from the patient's landmark to the patient's third vascular segment are excluded. The vascular classification system according to claim 1.

6. A three-dimensional structure recognition device that recognizes the three-dimensional structure of the abdominal surface of a patient, A depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device, A vascular segmentation discrimination device that uses a deep learning model to determine the aortic segment of a patient whose three-dimensional structure on the abdominal surface has been recognized by the three-dimensional structure recognition device, A method for determining the blood vessel classification of a blood vessel classification system, comprising a training dataset generation unit that generates a training dataset used for training the deep learning model, Before the training of the deep learning model is performed, the training dataset generation unit generates the training dataset. A three-dimensional structure of the abdominal surface for learning purposes is generated from either an abdominal CT image, abdominal MRI image, or abdominal MRA image of a person other than the patient whose aortic segment is to be determined by the aforementioned vascular segment discrimination device. A learning depth image is generated from the three-dimensional structure of the abdominal surface used for learning. A training dataset generation step, based on any of the abdominal CT image, abdominal MRI image, and abdominal MRA image, generates a training dataset that shows the correspondence between each pixel in the training depth image and any of the first vascular divisions corresponding to Zone 1 of the aorta, the second vascular divisions corresponding to Zone 2 of the aorta, the third vascular divisions corresponding to Zone 3 of the aorta, and other divisions. After the training model is trained using the training dataset generated in the training dataset generation step, the 3D structure recognition device performs a 3D structure recognition step in which it recognizes the 3D structure of the patient's abdominal surface. The depth image generation unit performs a depth image generation step in which it generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized in the three-dimensional structure recognition step, The vascular segmentation discrimination device includes a vascular segmentation discrimination step which estimates whether each pixel in the depth image of the patient's abdominal surface generated in the depth image generation step corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or the other segment, by using the trained deep learning model. Method for determining blood vessel classification.

7. A three-dimensional structure recognition device that recognizes the three-dimensional structure of the abdominal surface of a patient, A depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device, A computer constituting a vascular segment discrimination device, which is part of a vascular segment discrimination system that includes a training dataset generation unit that generates training datasets used to train a deep learning model, A learning step in which a deep learning model is trained using the learning dataset generated by the learning dataset generation unit, A program for performing the vascular segmentation step, The aforementioned training dataset generation unit is: A three-dimensional structure of the abdominal surface for learning purposes is generated from either an abdominal CT image, abdominal MRI image, or abdominal MRA image of a person other than the patient whose aortic segment is to be determined by the aforementioned vascular segment discrimination device. A learning depth image is generated from the three-dimensional structure of the abdominal surface used for learning. Based on the abdominal CT image, the abdominal MRI image, and the abdominal MRA image, a training dataset is generated that shows the correspondence between each pixel in the training depth image and one of the first vascular divisions corresponding to Zone 1 of the aorta, the second vascular division corresponding to Zone 2 of the aorta, the third vascular division corresponding to Zone 3 of the aorta, and other divisions. After the execution of the learning step, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface, and the depth image generation unit generates a depth image of the patient's abdominal surface from the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device. In the vascular segment discrimination step, by using the trained deep learning model, it is estimated whether each pixel in the depth image of the patient's abdominal surface generated by the depth image generation unit corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments. program.

8. A three-dimensional structure recognition device that recognizes the three-dimensional structure of the abdominal surface of a patient, A vascular segmentation discrimination device that uses a deep learning model to determine the aortic segment of a patient whose three-dimensional structure on the abdominal surface has been recognized by the three-dimensional structure recognition device, The system includes a training dataset generation unit that generates a training dataset used for training the deep learning model, Before the training of the deep learning model is performed, the training dataset generation unit performs the following: A three-dimensional structure of the abdominal surface for learning purposes is generated from either an abdominal CT image, abdominal MRI image, or abdominal MRA image of a person other than the patient whose aortic segment is to be determined by the aforementioned vascular segment discrimination device. Based on the abdominal CT image, the abdominal MRI image, and the abdominal MRA image, a training dataset is generated that shows the correspondence between each point on the three-dimensional structure of the abdominal surface for learning and one of the first vascular divisions corresponding to Zone 1 of the aorta, the second vascular division corresponding to Zone 2 of the aorta, the third vascular division corresponding to Zone 3 of the aorta, and other divisions. After the deep learning model is trained using the training dataset generated by the training dataset generation unit, The three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface, The vascular segment discrimination device uses the trained deep learning model to estimate whether each point on the three-dimensional structure of the patient's abdominal surface, recognized by the three-dimensional structure recognition device, corresponds to the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments. Vascular classification system.

9. The system includes a visualization device that generates a virtual image projecting the estimated aortic segment of the patient, obtained by the aforementioned vascular segment discrimination device, onto the patient's abdominal surface. The vascular segmentation system according to claim 8.

10. The aforementioned blood vessel classification device is A learning unit that trains the deep learning model using the learning dataset generated by the learning dataset generation unit, The system includes an estimation unit that uses the trained deep learning model to estimate whether each point on the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments, The learning unit trains the deep learning model so that the estimation accuracy of the second blood vessel segment using the trained deep learning model is equal to or greater than a predetermined threshold. The vascular segmentation system according to claim 8.

11. The three-dimensional structure recognition device has the function of generating a three-dimensional structure of the patient's abdominal surface from any of the patient's abdominal CT images, abdominal MRI images, or abdominal MRA images. The vascular segmentation system according to claim 8.

12. The aforementioned vascular segmentation device estimates the patient's aortic segmentation as follows: The length from the patient's landmark to the patient's first vascular segment, the length from the patient's landmark to the patient's second vascular segment, and the length from the patient's landmark to the patient's third vascular segment are excluded. The vascular segmentation system according to claim 8.

13. A three-dimensional structure recognition device that recognizes the three-dimensional structure of the abdominal surface of a patient, A vascular segmentation discrimination device that uses a deep learning model to determine the aortic segment of a patient whose three-dimensional structure on the abdominal surface has been recognized by the three-dimensional structure recognition device, A method for determining the blood vessel classification of a blood vessel classification system, comprising a training dataset generation unit that generates a training dataset used for training the deep learning model, Before the training of the deep learning model is performed, the training dataset generation unit generates the training dataset. A three-dimensional structure of the abdominal surface for learning purposes is generated from either an abdominal CT image, abdominal MRI image, or abdominal MRA image of a person other than the patient whose aortic segment is to be determined by the aforementioned vascular segment discrimination device. A training dataset generation step, based on any of the abdominal CT image, abdominal MRI image, and abdominal MRA image, generates a training dataset that shows the correspondence between each point on the three-dimensional structure of the abdominal surface for learning and any of the first vascular divisions corresponding to Zone 1 of the aorta, the second vascular divisions corresponding to Zone 2 of the aorta, the third vascular divisions corresponding to Zone 3 of the aorta, and other divisions. After the training model is trained using the training dataset generated in the training dataset generation step, the 3D structure recognition device performs a 3D structure recognition step in which it recognizes the 3D structure of the patient's abdominal surface. The vascular segmentation discrimination device includes a vascular segmentation discrimination step in which, by using the trained deep learning model, it estimates whether each point on the three-dimensional structure of the patient's abdominal surface recognized in the three-dimensional structure recognition step corresponds to one of the first vascular segment, the second vascular segment, the third vascular segment, or the other segment. Method for determining blood vessel classification.

14. A three-dimensional structure recognition device that recognizes the three-dimensional structure of the abdominal surface of a patient, A computer constituting a vascular segment discrimination device, which is part of a vascular segment discrimination system that includes a training dataset generation unit that generates training datasets used to train a deep learning model, A learning step in which a deep learning model is trained using the learning dataset generated by the learning dataset generation unit, A program for performing the vascular segmentation step, The aforementioned training dataset generation unit is: A three-dimensional structure of the abdominal surface for learning purposes is generated from either an abdominal CT image, abdominal MRI image, or abdominal MRA image of a person other than the patient whose aortic segment is to be determined by the aforementioned vascular segment discrimination device. Based on the abdominal CT image, the abdominal MRI image, and the abdominal MRA image, a training dataset is generated that shows the correspondence between each point on the three-dimensional structure of the abdominal surface for learning and one of the first vascular divisions corresponding to Zone 1 of the aorta, the second vascular division corresponding to Zone 2 of the aorta, the third vascular division corresponding to Zone 3 of the aorta, and other divisions. After the execution of the learning step, the three-dimensional structure recognition device recognizes the three-dimensional structure of the patient's abdominal surface. In the vascular segment discrimination step, by using the trained deep learning model, it is estimated whether each point on the three-dimensional structure of the patient's abdominal surface recognized by the three-dimensional structure recognition device corresponds to the first vascular segment, the second vascular segment, the third vascular segment, or any of the other segments. program.

Images