Image segmentation and classification apparatus, method and recording medium for tubular structures

Abstract

The present application provides a tubular structure image segmentation and classification device and method, which utilizes a neural network to segment and classify a tubular structure image, and the tubular structure image segmentation and classification device comprises: an input unit configured to input a tubular structure image; an image pair generation unit configured to generate an image pair composed of a local image containing local features of the tubular structure and a global image containing global features of the tubular structure according to the input image; and a segmentation and classification unit configured to segment and classify the image pair by the neural network.

Description

Technical Field

[0001] This invention relates to an apparatus, method, and recording medium for image segmentation and classification of tubular structures. More particularly, it relates to an apparatus, method, and recording medium for segmenting and classifying images of tubular structures using neural networks. Background Technology

[0002] Human organs and tissues include tubular structures such as blood vessels and trachea. In this specification, these tubular organs and tissues are collectively referred to as tubular structures. Extracting and separating tubular structures such as blood vessels through segmentation and classification is a crucial step in disease diagnosis, assessment, and treatment decision-making.

[0003] In existing technologies, traditional methods such as Hessian matrix, graph cut, and vascular structure matching are used for the extraction and separation of tubular structures. Additionally, a method combining deep learning and traditional approaches has been proposed to perform tubular structure extraction and separation through independent steps.

[0004] Several deep learning-based methods for blood vessel segmentation have been proposed in the prior art. For example, Patent Document 1 (CN111091573A) discloses a deep learning-based method and system for segmenting pulmonary blood vessels in CT images. It includes a blood vessel segmentation step and a blood vessel arteriovenous differentiation step, and by combining the arteriovenous probability map obtained from the blood vessel arteriovenous differentiation with the preliminary blood vessel segmentation results, it can segment small intrapulmonary blood vessels and large extrapulmonary blood vessels, and distinguish between arteries and veins.

[0005] Existing technologies also include multi-stage extraction techniques for other structures in the human body. For example, Patent Document 2 (Japanese Patent Application Laid-Open No. 2020-068797) discloses a technique for rapidly extracting images of target cross-sections from image data obtained from a medical imaging device with minimal computation. Patent Document 2 discloses a method of transforming the image size of multiple cross-sectional images of different sizes to match the input image size of a learning model, and then performing multi-stage extraction of structures from the cross-sectional images using the learning model. Specifically, Patent Document 2 uses a fetus in the human abdomen as an example. In the case where the target cross-section contains a first structure and a second structure located inside the first structure, the learning data includes an image of the target cross-section containing both the first and second structures. In this case, the cross-sectional image of the region of interest used for learning can be an image magnified by cutting out a portion of the target cross-sectional image that includes the first structure. By applying the learning model to the cross-sectional image obtained from the image data, when the first structure is detected, an image can be generated by extracting the region including the first structure from the cross-sectional image. The learning model is then applied to the cut-out image to detect the second structure. Summary of the Invention

[0006] Problems in the prior art

[0007] Existing techniques for extracting and separating tubular structures such as blood vessels cannot achieve optimal extraction performance, especially in methods that combine deep learning with traditional post-processing methods. Because existing techniques perform deep learning processing and post-processing separately, they require adjusting parameters based on many assumptions, which may sometimes lead to failures in the separation and extraction of tubular structures such as blood vessels.

[0008] Means for solving technical problems

[0009] This invention addresses the problems of the prior art. It proposes an image segmentation and classification method and apparatus for tubular structures, combining local and global features of images containing tubular structures to simultaneously segment and classify them, achieving precise extraction and separation of tubular structures without the need for traditional post-processing.

[0010] This invention first generates image pairs reflecting local and global features in preprocessing, and then performs tubular structure segmentation and classification based on a deep learning method that combines local and global features. Thus, tubular structures can be extracted and separated from medical images without post-processing.

[0011] Specifically, according to one aspect of the present invention, an image segmentation and classification apparatus for tubular structures is provided, which uses a neural network to segment and classify images of tubular structures. The apparatus comprises: an input unit for inputting an image of the tubular structure; an image pair generation unit for generating an image pair based on the input image, consisting of a local image containing local features of the tubular structure and a corresponding global image containing global features of the tubular structure; and a segmentation and classification unit for segmenting and classifying the image pair using the neural network.

[0012] This invention performs segmentation and classification of tubular structures based on a deep learning method that combines local and global features. It requires no parameter tuning or post-processing and can reliably be applied to multiple scripts using deep learning. According to this invention, tubular structures of similar length, such as blood vessels, can be segmented and classified with high accuracy, for example, arteries and veins. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating an example of the structure of the tubular structure segmentation and classification device according to the first embodiment.

[0014] Figure 2 This is a comparative diagram used to illustrate the features of the processing performed by the tubular structure segmentation and classification device of the first embodiment.

[0015] Figure 3 This is a flowchart illustrating an example of blood vessel segmentation and classification processing performed by the tubular structure segmentation and classification device of the first embodiment.

[0016] Figure 4 This is a schematic diagram illustrating the network structure of the tubular structure segmentation and classification device according to the first embodiment and the relationship between the steps of each process performed.

[0017] Figure 5 This is a schematic diagram illustrating the process of step S200 performed by the tubular structure segmentation and classification device according to the first embodiment.

[0018] Figure 6 This is a schematic diagram illustrating the process of step S301 performed by the tubular structure segmentation and classification device of the first embodiment.

[0019] Figure 7 This is a schematic diagram illustrating the process of step S302 performed by the tubular structure segmentation and classification device of the first embodiment.

[0020] Figure 8 This is a schematic diagram illustrating the process of step S303 performed by the tubular structure segmentation and classification device of the first embodiment.

[0021] Figure 9 This is a schematic diagram illustrating an example of blood vessel classification processing performed by the tubular structure segmentation and classification device according to the second embodiment. Detailed Implementation

[0022] Hereinafter, embodiments of the tubular structure segmentation and sorting apparatus, method, and storage medium of this application will be described in detail with reference to the accompanying drawings. However, the tubular structure segmentation and sorting apparatus, method, and storage medium of this application are not limited to the embodiments shown below. Furthermore, in the following description, the same components will be given common reference numerals and repeated descriptions will be omitted.

[0023] (First Embodiment)

[0024] First, the tubular structure segmentation and classification device of the first embodiment will be described. The tubular structure segmentation and classification device of this application can exist in the form of a medical image diagnostic device such as an ultrasound diagnostic device or an MRI imaging device, or it can exist independently in the form of a workstation or the like.

[0025] Figure 1 This is a block diagram illustrating an example of the structure of the tubular structure segmentation and classification device 1 according to the first embodiment. The tubular structure segmentation and classification device 1 of this embodiment utilizes a neural network to segment and classify images of tubular structures, such as... Figure 1 As shown, the tubular structure segmentation and classification device 1 mainly includes an input unit 10, an image pair generation unit 20, and a segmentation and classification unit 30. The input unit 10 is used to input an image of the tubular structure. The image pair generation unit 20 is used to generate an image pair based on the input image, consisting of a local image containing local features of the tubular structure and a corresponding global image containing global features of the tubular structure. The segmentation and classification unit 30 uses a neural network to segment and classify the image pair, generating a segmented image of the tubular structure.

[0026] The tubular structure segmentation and classification device 1 can be, for example, equipped in a tubular structure segmentation and classification device such as an ultrasonic diagnostic device. In this case, the tubular structure segmentation and classification device 1 also includes a control unit (omitted from the illustrations), an ultrasonic probe, a display, an input / output interface, and a device body. The input unit 10, the image pair generation unit 20, and the segmentation and classification unit 30 are mounted in the control unit and are communicatively connected to these ultrasonic probes, the display, the input / output interface, and the device body. Since the structure and function of the control unit, the ultrasonic probe, the display, the input / output interface, and the device body are well known to those skilled in the art, detailed descriptions are omitted.

[0027] The following is a detailed description of the processing performed by the tubular structure segmentation and classification device 1 of this embodiment.

[0028] Figure 2This is a comparative diagram used to illustrate the features of the processing performed by the tubular structure segmentation and classification device of the first embodiment.

[0029] Figure 2 The features of the present invention are illustrated by comparing it with existing technology processes. The left side is a flowchart of the tubular structure segmentation and classification process in the prior art, and the right side is a flowchart of the process performed by the tubular structure segmentation and classification device of the first embodiment. Figure 2 As shown, in the processing performed by the tubular structure segmentation and classification apparatus according to the first embodiment of the present invention, steps S200 and S300 are characteristic steps of the present invention. In step S200, the image pair generation unit 20 generates an image pair based on the input image, consisting of a local image containing local features of the tubular structure and a global image corresponding to the local image containing global features of the tubular structure. Next, in step S300, the segmentation and classification unit 30 performs deep learning-based segmentation and classification on the image pair generated by the image pair generation unit 20 to obtain the image segmentation and classification result of the tubular structure.

[0030] This invention first generates an image pair consisting of a local image containing local features of the tubular structure and a global image containing global features of the tubular structure corresponding to the local image (step S200). Then, based on the image pair, it performs segmentation and classification of the tubular structure by utilizing both local and global features through deep learning (step S300). Therefore, unlike the prior art which takes image blocks (step S200') as the processing object, performs segmentation and classification based on image blocks first (S301'), and then corrects the segmentation and classification results (S302'), this invention does not require post-processing or parameter adjustment as in the prior art, and can obtain fine segmentation and classification results with reliable deep learning methods.

[0031] The following uses Figure 3 and Figure 4 Taking blood vessels as an example, the overall processing of the tubular structure segmentation and classification device in this embodiment will be explained.

[0032] Figure 3 This is a flowchart illustrating an example of blood vessel segmentation and classification processing performed by the tubular structure segmentation and classification device of the first embodiment.

[0033] First, in step S100, medical image data containing blood vessels, which is the object of processing, is input to the tubular structure segmentation and classification device 1 through the input unit 10.

[0034] Next, in step S200, as preprocessing, the image pair generation unit 20 generates an image pair consisting of a local image containing local features of the tubular structure and a global image corresponding to the local image and containing global features of the tubular structure. Details of the processing in step S200 will be discussed below. Figure 5 Please provide an explanation.

[0035] Next, as part of step S300, the segmentation and classification unit 30 performs blood vessel segmentation based on deep learning utilizing local features (step S301), coarse artery / vein classification based on deep learning utilizing global features (step S302), and fine artery / vein classification based on deep learning utilizing both local and global features (step S303). Details regarding the processing of step S300 will be provided below. Figure 6 Please provide an explanation.

[0036] Finally, in step S400, the arterial / venous segmentation and classification results of the blood vessels are output.

[0037] The tubular structure segmentation and classification device of the first embodiment performs deep learning-based segmentation and classification processing of blood vessels. Before detailing each step of the segmentation and classification processing, the network structure of the tubular structure segmentation and classification device of the first embodiment will be described first. Figure 4 This is a schematic diagram showing the network structure of the tubular structure segmentation and classification device according to the first embodiment and the relationship between the various steps performed.

[0038] like Figure 4 As shown, the network structure of the tubular structure segmentation and classification device of the first embodiment may include the following three deep neural networks: a segmentation neural network (neural network A) for performing blood vessel segmentation using local features in step S301; a classification neural network 1 (neural network B) for performing coarse artery / vein classification using global features in step S302; and a classification neural network 2 (neural network C) for performing fine artery / vein classification using both local and global features in step S303.

[0039] After inputting the medical image in step S100, step S200 obtains an image pair consisting of a local image SP containing local features of the tubular structure and a global image LP corresponding to the local image and containing global features of the tubular structure, wherein there is a spatial transformation relationship between the local image SP and the global image LP.

[0040] Step S300 includes sub-steps S301 to S303.

[0041] First, in sub-step S301, the local image SP and global image LP contained in the image pair are segmented into tubular structures using a segmentation neural network (neural network A), respectively, to obtain the mask image V-SP of the local image SP and the mask image V-LP of the global image LP. These two segmentation processes can be independent of each other. Figure 4 The solid and dashed arrows in the diagram represent the data flows for local and global image segmentation, respectively.

[0042] Subsequently, in sub-step S302, the global image LP reflecting global features and the mask image V-LP of the global image LP are coarsely classified, for example, into veins / arteries, by the classification neural network 1 (neural network B) to obtain the classified global classification image A / V-LP.

[0043] Subsequently, in sub-step S303, the mask image V-SP of the local image SP and the preparatory local classification image A / V-SP1 obtained by performing an inverse spatial transformation on the global classification image A / V-LP are used as inputs. The classification neural network 2 (neural network C) is used to perform fine classification, such as vein / artery classification, to obtain the finely classified local classification image A / V-SP.

[0044] Finally, in step S400, the segmentation and classification results are output.

[0045] Therefore, the deep learning-based tubular structure segmentation and classification processing of this invention combines both local and global features, requiring no parameter tuning or post-processing, and can be reliably applied to multiple scripts using deep learning methods. According to this invention, tubular structures of length such as blood vessels can be segmented and classified with high accuracy, such as arteries and veins.

[0046] The following is combined Figures 5-9 The process performed by the tubular structure segmentation and classification device of the first embodiment is described in detail.

[0047] Figure 5 This is a schematic diagram illustrating the process of step S200 performed by the tubular structure segmentation and classification device according to the first embodiment.

[0048] First, the image pair generation unit 20 can perform image preprocessing on the medical image data input in step S100 in step S201. Image preprocessing is used to eliminate irrelevant information in the image and restore useful real information, thereby improving the reliability of feature extraction, image segmentation, classification, etc. The image preprocessing in step S201 may include traditional grayscale conversion, geometric transformation, image enhancement, etc., and can be implemented by various methods in the prior art, which will not be described in detail here. In addition, preprocessing is not essential to this invention, and the preprocessing step can be omitted depending on the situation.

[0049] Next, in steps S202 and S203, the image pair generation unit 20 extracts small image patches SP containing local features of blood vessels from the preprocessed image as local images, and extracts large image patches LP1 containing at least the small image patches and global features of the blood vessels as preliminary global images. Specifically, the local features of blood vessels reflect local morphological features, such as the shape of the vessel's ends, branches, etc. Specifically, the global features of blood vessels reflect global morphological features, such as the shape and branches of the vessel's root and its ends, etc.

[0050] like Figure 5 As shown, the small image patch extracted in step S202 includes the end of the target blood vessel but not its root. The large image patch extracted in step S203, as shown in the yellow box in LP1, includes the small image patch SP and also includes the root of the target blood vessel. There are no particular limitations on the method for extracting image patches from the input image data; it can be traditional sampling, image cropping, etc., which will not be described in detail here.

[0051] In addition, in step S203, the image pair generation unit 20 also resizes the extracted large image block, i.e., the preparatory global image LP1, and adjusts it to an image LP with the same file size as the local image, which is then used as the global image. The above-mentioned resizing of the large image block can be achieved by downsampling or by traditional geometric transformation methods such as adjusting resolution or scaling, without any particular limitations.

[0052] Furthermore, in step S204, the image pair generation unit 20 also uses the global image LP obtained in step S203 and the local image SP obtained in step S202 to obtain a transformation matrix θ representing the spatial transformation relationship between the global image LP and the local image SP, for subsequent processing. Since the global image LP contains the local image SP, the global image LP and the local image SP can be considered as images that can be obtained from each other through conventional spatial transformations such as scaling and cropping (where obtaining the global image LP from the local image SP also requires the original image). Spatial transformation of the image can be achieved, for example, through matrix operations. The transformation matrix θ used to obtain the global image LP from the local image SP and the original image through matrix operations is defined as the transformation matrix θ of this invention. The transformation matrix θ can be based on the global image LP and the local image SP, transforming the local image SP into the global image LP through a virtual spatial transformation. Figure 5 The matrix algorithm, etc. (the original image illustration is omitted here) is obtained using various methods in the existing technology, and will not be described in detail here.

[0053] Through the processing of steps S202 to S204 above, the tubular structure segmentation and classification device of the first embodiment obtains an image pair consisting of a local image SP containing local features of blood vessels and a global image LP containing global features of blood vessels corresponding to the local image, and obtains a transformation matrix θ representing the spatial transformation relationship between the global image LP and the local image SP.

[0054] The above-described process of obtaining an image pair consisting of a local image SP containing local features of the tubular structure and a corresponding global image LP containing global features of the tubular structure through step S200 is merely an example. The method for obtaining the image pair is not limited to this. For instance, the above example illustrates obtaining the local image SP and the global image LP by extracting small and large image blocks respectively in steps S202 and S203. However, the acquisition of the local image SP and the global image LP can certainly be achieved in other ways. For example, step S202 can be omitted, and the required small image block can be obtained as the local image SP from the large image block LP1 extracted in step S203 through spatial transformations such as cropping. Furthermore, the size adjustment in step S203 is not mandatory; this will be explained in detail later. According to the present invention, as long as an image pair consisting of a local image SP containing local features of the tubular structure and a corresponding global image LP containing global features of the tubular structure can be obtained, it is sufficient.

[0055] Figure 6 This is a schematic diagram illustrating the process of step S301 performed by the tubular structure segmentation and classification device of the first embodiment.

[0056] Step S301 uses a segmentation network to segment mask images of blood vessels from the image pair. Specifically, the segmentation classification unit 30 uses a neural network A, such as U-Net, and takes the local image SP and global image LP from the image pair obtained in step S200 as input to segment a local mask image V-SP containing local features of blood vessels and a global mask image V-LP containing global features of blood vessels, respectively. Details of deep neural networks and methods for segmenting images using deep neural networks to obtain mask images are well known in the art and will not be described in detail here. Furthermore, as described above... Figure 5 As explained, since the global image LP is obtained by resizing the prepared global image LP1 through, for example, downsampling, it is a lower precision image compared to the local image SP. Therefore, when segmenting the global image LP using the neural network A, unlike the normal segmentation of the local image SP, during the training phase, the global image LP can be allowed to enter the network A in only one direction, and the backpropagation of the loss is not allowed, so as to avoid reducing the performance of the network A during repeated optimization. Figure 6 The dashed and solid arrows in the diagram represent the segmentation of the global image LP and the local image SP, respectively.

[0057] Figure 7 This is a schematic diagram illustrating the process of step S302 performed by the tubular structure segmentation and classification device of the first embodiment.

[0058] Step S302 uses classification network 1 (network B) to perform coarse classification of arteries / veins on the global image, including sub-steps S3021 to S3023.

[0059] Specifically, in sub-step S3021, the segmentation and classification unit 30 first inputs the global image LP obtained in step S200 and the global mask image V-LP obtained in step S301. Then, in sub-step S3022, the segmentation and classification unit 30 uses, for example, a neural network B such as U-Net to obtain a global classification image A / V-LP representing the blood vessel / vein classification result of the global image through a classification algorithm based on the global image LP and the global mask image V-LP. The global image contains global features; therefore, compared to classification using only local features, classification based on the global image considers the global morphology including the root, resulting in fewer classification errors and more accurate classification results, such as for arteries and veins. Details regarding deep neural networks and methods for classifying images using deep neural networks are well known in the art and will not be elaborated upon here.

[0060] Subsequently, in sub-step S3023, the segmentation and classification unit 30 uses the transformation matrix θ obtained in step S204 to perform an inverse spatial transformation on the global classification image A / V-LP obtained in sub-step S3022, which is the opposite of the virtual spatial transformation performed when obtaining the transformation matrix θ, to obtain a preliminary local classification image A / V-SP1 representing the coarse classification result of blood vessels / veins in the local image.

[0061] The sub-step S3023 can be performed as follows: First, based on the previously acquired preliminary global image LP1, the global classification image A / V-LP is resized by, for example, upsampling, so that its file size is the same as that of the preliminary global image LP1. Then, an inverse spatial transformation is performed on the resized global classification image according to the transformation matrix θ, thereby obtaining the preliminary local classification image A / V-SP1. This inverse spatial transformation is the opposite of the virtual spatial transformation performed when obtaining the transformation matrix θ in step S204. Specifically, the result of performing the inverse spatial transformation is as follows: Figure 7 As shown, this is equivalent to cropping an image corresponding to the size of SP from the resized global classification image, for example, by cropping.

[0062] In sub-step S3023, upsampling is performed before the inverse spatial transformation. This upsampling is the inverse process corresponding to the downsampling in step S203. Neither of these processes is mandatory; for example, if the downsampling in step S203 is omitted, the upsampling in sub-step S3023 can also be omitted. By using the downsampling in step S203 and the corresponding upsampling in sub-step S3023, the processing object image input to the segmentation network and the processing object image input to the classification network 2 can be made to have the same size, which helps to speed up processing, improve processing accuracy, and reduce the amount of GPU memory used.

[0063] Figure 7 In the classification images A / V-LP and A / V-SP1, the classification results for arteries / veins are represented by color intensity; for example, dark areas represent veins and light areas represent arteries. Figure 7 This display method also applies to other accompanying figures. Using color depth to represent artery / vein classification results, and using darker areas for veins and lighter areas for arteries, is just one example; the display methods for veins and arteries are not limited to this.

[0064] Figure 8 This is a schematic diagram illustrating the process of step S303 performed by the tubular structure segmentation and classification device of the first embodiment.

[0065] Step S303 utilizes classification network 2 (network C) to perform fine classification of arteries / veins on the local image. Specifically, in order to obtain fine artery / vein classification results, the segmentation and classification unit 30 takes the local image SP obtained in step S200, the local mask image V-SP obtained in step S301, and the preliminary local classification image A / V-SP1 obtained in step S302 as input, and performs artery / vein classification processing using a neural network C such as U-Net, for example.

[0066] As described above, the local mask image V-SP obtained in step S301 is a mask image obtained directly from the local image through segmentation based on a deep learning network, reflecting the correct segmentation result containing the local features of the target blood vessel. On the other hand, the preliminary local classification image A / V-SP1 obtained in step S302 is obtained from the classification result based on a deep learning network that utilizes global features such as blood vessel root features. Compared with classification that only utilizes local features, classification that utilizes global features such as blood vessel roots takes into account the global morphology including the root, and therefore can obtain more accurate classification results, such as for arteries and veins. In step S303, the local mask image V-SP, which reflects the target blood vessel segmentation result with local features, and the preliminary local classification image A / V-SP1, which reflects the target blood vessel classification result with global features, are used together as input. By combining local and global features, classification errors, such as those in the blood vessel edge portion, in the preliminary local classification image A / V-SP1 can be corrected based on the local mask image V-SP, resulting in more accurate arterial / vein segmentation and classification results for the target blood vessel.

[0067] According to the tubular structure segmentation and classification apparatus of the first embodiment, since it combines local and global features, it can reliably apply deep learning methods to multiple scripts without parameter adjustment or post-processing in, for example, deep learning-based artery / vein segmentation and classification of blood vessels. According to the present invention, tubular structures of equal length, such as blood vessels, can be segmented and classified with high accuracy, such as arteries and veins. Moreover, the present invention, by combining local and global features, achieves higher segmentation and classification accuracy compared to methods that use only a single image patch size for both local and global image patches.

[0068] Furthermore, according to the tubular structure segmentation and classification apparatus of the first embodiment, since the size of the input image is small during frame learning, the amount of GPU memory used can be reduced, for example.

[0069] The first embodiment has been described above, but the present invention may be implemented in various different forms other than the first embodiment described above.

[0070] (Second Implementation)

[0071] In the first embodiment described above, an example was given of a complete process including vessel segmentation, coarse artery / vein classification, and fine artery / vein classification, starting from image preprocessing. However, the implementation is not limited to this. For example, the fine classification processing based on step S303 of the first embodiment can be modularized separately, that is, the target vessel segmentation result and the coarse artery / vein classification result obtained by other methods can be directly input into a deep learning-based classification neural network to obtain a fine artery / vein classification result. The following will use... Figure 9 The second embodiment of the present invention, which involves finely classifying and modularizing individual components, is illustrated by an example.

[0072] Figure 9 This is a schematic diagram illustrating an example of blood vessel classification processing performed by the tubular structure segmentation and classification device according to the second embodiment. In the description of the second embodiment, the differences from the first embodiment described above are mainly explained. Furthermore, in the description of the second embodiment, structures identical to those in the first embodiment are labeled with the same reference numerals and their descriptions are omitted.

[0073] According to the tubular structure segmentation and classification device of the second embodiment, the original image of the target blood vessel obtained in advance by other methods, the mask image of the target blood vessel obtained by segmenting the original image, and the coarse classification result image of the target blood vessel artery / vein obtained by classifying the original image are directly input into the classification neural network to obtain a fine blood vessel artery / vein classification result.

[0074] from Figure 9 As can be seen from the diagram, the coarse classification results of the vein terminal portion marked by the circle clearly contain errors. The classification neural network can correct these classification errors based on the segmentation results of the target blood vessel. Therefore, the method of the second embodiment obtains a clearer classification result that is closer to the true value (GT).

[0075] Compared with the first embodiment, the second embodiment uses the fine classification module of blood vessels as a post-processing module and uses it separately. Therefore, in addition to the effects of the first embodiment, it can also achieve the beneficial technical effect of independently applying the fine classification module of arteries / veins to an existing system to obtain reliable classification results.

[0076] The above embodiments use the classification of blood vessels and arteries / veins as examples, but the present invention can also be applied to other tubular structures besides blood vessels, as well as classifications other than arteries / veins. For example, when applying the present invention to liver blood vessels, hepatic veins and portal veins can be considered as different categories of liver blood vessels, and accurate classification results of hepatic veins and portal veins can be obtained by applying the present invention.

[0077] The above embodiments use a U-Net as an example to illustrate neural networks A, B, and C. However, neural networks A, B, and C are not limited to U-Nets; they can be other convolutional neural networks or other deep learning networks. Furthermore, neural networks A, B, and C can be the same type of neural network or different types of neural networks.

[0078] The image processing, segmentation, classification, training and inference of deep learning neural networks described in the above embodiments can all be implemented using various methods in the prior art, and detailed descriptions are omitted here.

[0079] This invention can be implemented not only as the aforementioned tubular structure segmentation and classification device, but also as a tubular structure segmentation and classification method, program, or medium storing the tubular structure segmentation and classification program.

[0080] The tubular structure segmentation and classification device of this application can be installed in a medical image diagnostic device, or it can be used independently for processing. In such cases, the tubular structure segmentation and classification device has a processing circuit that performs the same processes as described above, and a memory that stores programs corresponding to each function and various information. Furthermore, the processing circuit obtains two-dimensional or three-dimensional medical image data from a medical image diagnostic device or image storage device, such as an ultrasound diagnostic device, via a network, and performs the aforementioned processing using the obtained medical image data. Here, the processing circuit is a processor that implements the functions corresponding to each program by reading and executing programs from memory.

[0081] The term "processor" used in the above description refers to circuits such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), Application Specific Integrated Circuits (ASICs), and programmable logic devices (e.g., simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field-programmable gate arrays (FPGAs)). The processor performs its function by reading and executing a program stored in memory. Alternatively, instead of storing the program in memory, the program can be directly loaded into the processor's circuitry. In this case, the processor performs its function by reading and executing the program loaded into the circuitry. Furthermore, the processors in this embodiment are not limited to a single circuit configuration; multiple independent circuits can be combined to form a single processor to perform its function.

[0082] Furthermore, the constituent elements of the devices illustrated in the above embodiments are functional concepts and do not necessarily need to be physically configured as shown in the illustrations. That is, the specific form of the distribution / combination of the devices is not limited to the form shown in the illustrations, and they can be configured, in any unit, functionally or physically, in a distributed / combined manner depending on various loads or usage conditions. Moreover, the processing functions performed by each device can be implemented, in whole or in part, by a CPU and the program parsed and executed by that CPU, or implemented as hardware as interconnect logic.

[0083] Furthermore, the processing method described in the above embodiments can be implemented by executing a pre-prepared processing program by a computer such as a personal computer or workstation. This processing program can be distributed via a network such as the Internet. Alternatively, the processing program can be recorded on a non-transitory recording medium that can be read by a computer, such as a hard disk, floppy disk (FD), CD-ROM, MO, DVD, USB memory, or SD card memory, and executed by the computer after reading it from the non-transitory recording medium.

[0084] Several embodiments of the present invention have been described, but these embodiments are merely illustrative and not intended to limit the scope of the invention. These embodiments can be implemented in a wide variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the invention, as well as within the scope of the present invention and its equivalents.

Claims

1. An image segmentation and classification device for tubular structures, which utilizes a neural network to segment and classify images of tubular structures, characterized in that, have: Input unit: Input image of the tubular structure; The image pair generation unit generates, based on the input image, an image pair consisting of a local image containing local features of the tubular structure and a corresponding global image containing global features of the tubular structure; and Segment the classification units and perform the following processing: The local image is segmented using a first deep learning network to obtain a local mask image of the tubular structure; The global image is segmented using the first deep learning network to obtain the global mask image of the tubular structure; Based on the global image and the global mask image, a global classification image of the tubular structure is generated by classifying the global image through a second deep learning network. Extract the portion corresponding to the local image from the global classification image and use it as the local classification image; as well as Using the local image, the local mask image, and the local classification image, a classification image of the tubular structure is generated through a third deep learning network.

2. The image segmentation and classification device with a tubular structure as described in claim 1, characterized in that, The image is processed by the generation unit as follows: From the preprocessed image, an image containing local structures other than the root of the tubular structure is extracted as the local image; From the preprocessed image, extract an image that contains at least the local image and the root of the tubular structure as a preliminary global image; as well as The prepared global image is downsampled to generate an image with the same file size as the local image, which is then used as the global image.

3. The image segmentation and classification device with a tubular structure as described in claim 2, characterized in that, When extracting the local classification image, the segmentation and classification unit first performs upsampling on the global classification image corresponding to the downsampling to obtain a preliminary global classification image with the same file size as the preliminary global image. Then, it uses a transformation matrix representing the relationship between the global image and the local image to extract the local classification image from the preliminary global classification image.

4. The image segmentation and classification device with a tubular structure as described in any one of claims 1 to 3, characterized in that, The tubular structure is a blood vessel, and the classification is divided into venous vessels and arterial vessels.

5. An image classification device for tubular structures, which uses a neural network to classify images of tubular structures, characterized in that, have: The input unit takes into account an original image containing a tubular structure, a mask image obtained by segmenting the original image using a first neural network, and a coarse classification image obtained by coarse classification of the original image using a second neural network. as well as The classification unit uses the input original image, the mask image, and the coarse classification image to generate a classification image that finely classifies the types of tubular structures through a third neural network.

6. A method for image segmentation and classification of tubular structures, comprising using a neural network to segment and classify images of tubular structures, characterized in that, Includes the following steps: Input steps: Input an image of the tubular structure; The image pair generation step involves generating an image pair based on the input image, consisting of a local image containing local features of the tubular structure and a corresponding global image containing global features of the tubular structure; and The segmentation and classification steps involve segmenting the local image using a first deep learning network to obtain a local mask image of the tubular structure; and segmenting the global image using the first deep learning network to obtain a global mask image of the tubular structure. Based on the global image and the global mask image, a global classification image of the tubular structure is generated by classifying the global image through a second deep learning network; the portion corresponding to the local image is extracted from the global classification image as a local classification image; and a classification image of the tubular structure is generated by a third deep learning network using the local image, the local mask image, and the local classification image.

7. A computer-readable storage medium storing an image segmentation and classification program for tubular structures, the program utilizing a neural network to segment and classify images of tubular structures, characterized in that, The program causes the computer to perform the following steps: Input steps: Input an image of the tubular structure; The image pair generation step involves generating an image pair based on the input image, consisting of a local image containing local features of the tubular structure and a corresponding global image containing global features of the tubular structure; and The segmentation and classification steps involve segmenting the local image using a first deep learning network to obtain a local mask image of the tubular structure; and segmenting the global image using the first deep learning network to obtain a global mask image of the tubular structure. Based on the global image and the global mask image, a global classification image of the tubular structure is generated by classifying the global image through a second deep learning network; the portion corresponding to the local image is extracted from the global classification image as a local classification image; and a classification image of the tubular structure is generated by a third deep learning network using the local image, the local mask image, and the local classification image.

Images