Automatic segmentation method and system for performing the automatic segmentation method
By combining multiple convolutional neural networks, segmenting 3D medical images for different reconstruction axes or planes, the problems of waste of resources and insufficient efficiency in the prior art are solved, and a fast and accurate 3D medical image segmentation effect is achieved.
Patent Information
- Application Number
- CN201980013054.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-01-10
- Filing Date
- 2019-01-10
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2039-01-10
AI Technical Summary
The neural networks used in the prior art have problems of waste of resources and insufficient efficiency and accuracy in 3D medical image segmentation, especially the voxels of 3D images based on standard non-specific architectures are difficult to effectively process voxels of 3D images.
Using combined N different convolutional neural networks (CNNs), each CNN is assigned to a different reconstruction axis or plane, adapting to image volume through structured geometry and architecture, performing multiple parallel or sequential analysis and segmentation, combined with known Algorithms such as U-Net, ResNet or DenseNet realize automatic segmentation of voxels.
This achieves complex 3D image segmentation under limited resources, providing fast and accurate results, and enhancing the reliability of results through cross-checking.
Smart Images
Figure CN111868781B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and more specifically, to image processing and analysis, especially medical image segmentation, and to automatic segmentation of 3D medical images by one or more neural networks through structured convolution according to the geometry or structure of the 3D medical images. Background Art
[0002] A three-dimensional image formed from a medical imaging device, such as a scanner, MRI, ultrasound, CT or SPEC type image, consists of a set of voxels, which are the basic unit of a 3D image. A voxel is a 3D extension of a pixel, which is the basic unit of a 2D image. Each voxel is associated with a gray level or density that can be considered as the result of a 2D function F(x, y) or a 3D function F(x, y, z), where x, y and z represent spatial coordinates (see Figure 1 ).
[0003] In a 3D image, voxels can be viewed in 2D along different axes or planes. The 3 main axes or planes in medical images are the axial, sagittal, and frontal axes or planes (see Figure 2 ). However, an unlimited number of axes or planes can be created using different angles.
[0004] Typically, a 2D or 3D medical image contains a set of anatomical and pathological structures (organs, bones, tissues, ...) or artificial elements (stents, implants, instruments, ...) that the clinician must describe in order to assess the situation and determine and plan his treatment strategy. In this regard, organs and pathologies must be identified in the image, which means labeling (e.g. coloring) each pixel of a 2D image or each voxel of a 3D image. This process is called segmentation.
[0005] For example, Figure 3 Represents the various stages of 3D medical image segmentation in transverse view.
[0006] There are many known methods for segmentation, in particular automatic methods using algorithms, in particular AI algorithms.
[0007] In this context, numerous variants of neural networks are used in the prior art, all based on standard non-specific architectures, resulting in global and inappropriate waste of resources and lack of efficiency and accuracy. Summary of the invention
[0008] The main object of the present invention is to propose a new method and a new system aimed at overcoming the above limitations.
[0009] The invention therefore relates to a method for automatic segmentation of features visible in 3D medical images of objects composed of voxels, such as anatomical and pathological structures or instruments.
[0010] The method is characterized by comprising:
[0011] providing a global software device or arrangement that combines N different convolutional neural networks CNN, N ≥ 2, and having a structured geometry or architecture that is compatible and comparable to the structured geometry or architecture of the image volume, and
[0012] The voxels of the volume forming the 3D image are analyzed according to N different reconstruction axes or planes, each CNN being assigned to the analysis of the voxels belonging to one axis or plane. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The invention will be better understood with the aid of the following description, which relates to some preferred embodiments given as non-limiting examples and illustrated with reference to the accompanying drawings, in which:
[0014] Figure 1 It is shown that in a 3D image, a voxel is a 3D extension of a pixel, which is the basic unit of a 2D image. Each voxel is associated with a grayscale or density that can be considered as the result of a 2D function F(x, y) or a 3D function F(x, y, z), where x, y, and z represent spatial coordinates.
[0015] Figure 2 A 3D image is shown in which the voxels can be viewed in 2D according to different axes or planes. The 3 main axes or planes in medical images are the axial, sagittal and frontal axes or planes.
[0016] Figure 3 Various stages of 3D medical image segmentation in transverse view are shown.
[0017] - Figure 4 is a schematic and symbolic representation of a unique global CNN algorithm integrating different analysis axes or planes of a 3D image, according to a first embodiment of the present invention, a specific CNN being assigned to each axis or plane;
[0018] - Figure 5 is a schematic and symbolic representation of yet another embodiment of the present invention, showing a serial or sequential arrangement of CNNs;
[0019] - Figure 6 is a schematic and symbolic representation of another embodiment of the present invention, showing a parallel arrangement of CNNs; and
[0020] - Figure 7 and8 is a schematic and symbolic representation of other embodiments of the present invention including a parallel arrangement of CNNs. DETAILED DESCRIPTION
[0021] In the attached figure Figure 4-8 Shown in is a method for automatic segmentation of features visible in 3D medical images of objects consisting of voxels, such as anatomical and pathological structures or instruments.
[0022] According to the invention, the method comprises providing a global software device or arrangement, which combines N different Convolutional Neural Networks (CNNs), N≥2, and has a structured geometry or architecture adapted and comparable to the structured geometry or architecture of the image volume, and analyzes voxels of the volume forming a 3D image according to N different reconstruction axes or planes, each CNN being assigned to the analysis of voxels belonging to one axis or plane.
[0023] Therefore, the present invention provides a structured organization and collaborative working of multiple CNNs taking into account the geometry, structure and content of medical images.
[0024] This particular composite computing system combining N different CNNs (N≥2, preferably N≥3) according to N different reconstruction axes or planes of the 3D image volume, possibly integrated in a single framework, allows extending the use of known CNNs for analyzing and segmenting 2D images to 3D images.
[0025] Typically, a known CNN algorithm that can be used within the methods and systems of the present invention is “U-Net” (see, for example: “U-Net: Convolutional Networks for Biomedical Image Segmentation”; O. Ronneberger et al.; MICCAI 2015, Part III, LNCS 3951, pp 234-241, Springer IPS).
[0026] “U-Net” can be combined with other known architectures, such as “ResNet” or “DenseNet”.
[0027] Advantageously, as in Figure 4-8As shown in , the method of the present invention may include, for each of the N reconstructed planes of the 3D image, analyzing and segmenting a 2D image formed by voxels of a given plane with the help of a dedicated CNNᵢ among the N CNNs provided, wherein the CNN is constructed similarly to the 3D image volume, and combining the intermediate or final results of the N analyses performed by the different CNNs.
[0028] By segmenting the 3D image volume for analysis and by merging (combining) the results of these partial analyses into a single 3D image segmentation, the invention allows to implement a complex segmentation process with limited resources and quickly provide accurate and in some way cross-checked results.
[0029] The results of N analyses and segmentations can be combined or merged by the following operations:
[0030] - For each voxel, combine the intermediate activations of the N networks (see Figure 4 - so-called “feature combination”). The resulting merged information is then received as input data by a global CNN which provides the final segmentation of the image;
[0031] - Combine the exit information of N different CNNs (see Figure 7 and 8 - late fusion or merging), for example by (weighted) summation, multiplication of classifiers or other adapted prediction integration operations known to those skilled in the art.
[0032] according to Figure 4 In a first embodiment of the invention shown in , the method may comprise providing a single neural network integrating in its structure N different CNNs, the N different CNNs advantageously working in parallel, the N different CNNs automatically segmenting anatomical and pathological structures or instruments visible in a 3D medical image, each CNN analyzing all voxels of the volume forming the 3D image according to a different reconstruction plane or axis, and combining, by convolution, the results of different 2D analyses and segmentations in the final structure of the neural network having a structured geometry.
[0033] In this case, the very internal structure of the single CNN thus formed integrates 3D axial image analysis, as well as the combination of information generated by different 2D analyses and segmentations.
[0034] Of course, such multiple parallel 2D processing together with the combination of the results of these processing can also be managed by a parallel arrangement of N independent CNNs that are not integrated into a single framework ( Figure 6 ).
[0035] according to Figure 5In a second alternative embodiment of the present invention shown in , the method may further include performing N sequential operating steps or image processing steps, wherein each step is performed by a CNN for automatically segmenting anatomical and pathological structures or instruments visible in the 3D medical image, each CNNᵢ of the CNNs being constructed according to a specific reconstruction plane for each of the N different CNNs by using the image segmentation performed by the previous network CNN in the sequence i-1 The result provided (when it exists) analyzes all voxels of the volume forming the 3D image.
[0036] In this case, the CNN 2D sequential processing scheme is integrated into the global architecture of the algorithmic framework, or into the structured geometry of the CNN arrangement, and the final combination of information of the sequence is therefore inherently integrated in the structure of the framework or arrangement.
[0037] about Figure 6-8 In the preferred embodiment of the present invention shown in , the segmentation method mainly includes a combination of two consecutive operation steps, the first step includes performing N segmentations, each segmentation is performed according to one of N different reconstruction axes or planes, and the second step includes combining the results of these N segmentations into a single segmentation of the anatomical and pathological structure or instrument visible in the 3D medical image.
[0038] Advantageously, the first operating step is performed by N different CNNs working in parallel or sequentially, wherein each of these CNNs automatically segments anatomical and pathological structures or instruments visible in the 3D medical image independently of the other CNNs, each CNN analyzing all voxels of the volume forming the 3D image according to a different reconstruction plane for each of the N different CNNs.
[0039] according to Figure 7 In a first alternative implementation of the invention shown in , a second operating step dedicated to combining the results of the N segmentations of the first step into a single segmentation of anatomical and pathological structures or instruments visible in the 3D medical image is performed by assigning a label to each voxel of the image volume, said label corresponding to the combination of the N labels assigned to the same voxel during the N segmentations of the first step.
[0040] according to Figure 8 In a second alternative implementation of the invention shown in , a second operation step dedicated to combining the results of the N segmentations of the first step into a single segmentation of anatomical and pathological structures or instruments visible in the 3D medical image is performed by assigning a label to each voxel of the image volume, said label corresponding to the combination of the N labels assigned to the same voxel and to one or more neighboring voxels of this voxel during the N segmentations of the first step.
[0041] In a 3D image volume, the neighboring voxels of a given voxel can involve three different groups of voxels, namely (e.g. Figure 8 ):
[0042] - a set of 6 voxels of the voxels involved in the face-to-face contact;
[0043] - A set of 18 voxels of voxels involved in face-to-face or edge-to-edge contact;
[0044] - Includes the above set of 18 voxels plus 8 additional voxels that are in single-point contact with the corner tips of the voxel in question.
[0045] like Figure 4-8 As shown in the diagram, the N reconstruction planes (used when implementing the method of the present invention) preferably include a sagittal plane 2 and a frontal or coronal plane 3, and at least one other plane that is perpendicular to the transverse or axial plane 1, contains the intersection line of the sagittal plane and the coronal plane and is angularly shifted toward these planes around the intersection line.
[0046] In addition to or as an alternative to the above, the N reconstruction planes may also include:
[0047] - A plane parallel to the sagittal plane 2 or the coronal plane 3, and / or
[0048] - A plurality of mutually parallel axial planes 1.
[0049] like Figure 4-8 As symbolically shown in , the present invention also includes a system for performing the automatic segmentation method as described above.
[0050] The system is characterized in that it comprises at least one computer device hosting N different convolutional neural networks (CNNᵢ) in a structured arrangement and allowing said N different convolutional neural networks (CNNᵢ) to work together in a serial or parallel organized manner, wherein N ≥ 2, each CNNᵢ being adapted and configured to automatically and independently of the other CNNs perform a segmentation of anatomical and pathological structures or instruments visible in the 3D medical image to be processed by analyzing the voxels of the volume forming the 3D image according to N different reconstruction axes or planes, each CNN being assigned to the analysis of voxels belonging to one axis or plane.
[0051] Preferably, the system further comprises means for combining and possibly displaying the results of the analysis and segmentation performed by the N different CNNs.
[0052] According to the first alternative, N different CNNs can be arranged into a serial architecture as follows ( Figure 5): Each of the N different CNNs CNNᵢ analyzes all voxels of the volume forming the 3D image according to a specific reconstruction plane for each of the N different CNNs and uses the voxels generated by the previous network CNN i-1 (when it exists) the result provided.
[0053] According to the second alternative, N different CNNs can be arranged into a parallel architecture, and the results of the N different CNNs can be combined in the final stage within a single algorithm framework ( Figure 4 , 6 and 8).
[0054] Of course, the invention is not limited to at least one embodiment described and represented in the accompanying drawings. Various modifications are still possible, especially from the perspective of the composition of the various elements, or by substitution of technical equivalents, without exceeding the scope of protection of the invention.
Claims
1. An automatic segmentation method for automatically segmenting features visible in a 3D medical image of an object consisting of voxels, the automatic segmentation method being implemented by using a global software device or arrangement combining N different convolutional neural networks CNN, N ≥ 3, and having a structured geometry corresponding to the structured geometry of the image volume, The method is characterized in that comprising analyzing voxels of said volume forming a 3D image according to N different reconstruction planes, each CNN being assigned to the analysis of voxels belonging to one plane, and It mainly consists of a combination of two consecutive operating steps: A first step consists in performing N segmentations, each segmentation being performed by analyzing and segmenting a 2D medical image according to one of N different reconstruction planes, by means of a corresponding dedicated CNNᵢ among the N CNNs provided, said 2D medical image being formed by all voxels belonging to each of the given planes and forming pixels of said 2D medical image, and The second step consists in combining the final results of these N mutually independent 2D segmentations into a single 3D segmentation of the anatomical and pathological structures or instruments visible in the 3D medical image.
2. The automatic segmentation method according to claim 1, characterized in that It comprises providing a single neural network, which integrates N different CNNs in its structure, and the N different CNNs work in parallel. The N different CNNs automatically segment anatomical and pathological structures or instruments visible in 3D medical images, each CNN analyzes all voxels of the volume forming the 3D image according to different reconstruction planes, and in the final structure of the neural network with a structured geometry, the results of different 2D analysis and segmentation are combined by convolution.
3. The automatic segmentation method according to claim 1, characterized in that The method comprises performing N sequential operation steps or image processing steps, wherein each step is performed by a CNN for automatically segmenting anatomical and pathological structures or instruments visible in a 3D medical image, each CNNᵢ of the CNNs following a specific reconstruction plane for each of the N different CNNs and if the previous network CNN exists in the sequence i-1 By using the previous network CNN in the sequence i-1 The results provided are for analyzing all voxels of the volume forming the 3D image.
4. The automatic segmentation method according to claim 1, characterized in that The first operation step is performed by N different CNNs working in parallel or sequentially, wherein each of these CNNs automatically segments anatomical and pathological structures or instruments visible in the 3D medical image independently of the other CNNs, each CNN analyzing all voxels of the volume forming the 3D image according to a different reconstruction plane for each of the N different CNNs.
5. The automatic segmentation method according to claim 1 or 4, characterized in that A second operating step dedicated to combining the results of the N segmentations of the first step into a single segmentation of anatomical and pathological structures or instruments visible in the 3D medical image is carried out by assigning to each voxel of the image volume a label corresponding to the combination of the N labels assigned to the same voxel during the N segmentations of the first step.
6. The automatic segmentation method according to claim 1 or 4, characterized in that A second operating step dedicated to combining the results of the N segmentations of the first step into a single segmentation of anatomical and pathological structures or instruments visible in the 3D medical image is carried out by assigning to each voxel of the image volume a label corresponding to the combination of the N labels assigned to the same voxel and to one or more neighboring voxels of this voxel during the N segmentations of the first step.
7. The automatic segmentation method according to any one of claims 1 to 6, characterized in that The N planes include a sagittal plane (2) and a frontal or coronal plane (3), and at least one other plane perpendicular to the axial plane (1), containing the intersection line of the sagittal plane and the coronal plane and angularly shifted toward these planes around the intersection line.
8. The automatic segmentation method according to any one of claims 1 to 6, characterized in that The N reconstruction planes used include planes parallel to the sagittal plane (2) or the coronal plane (3).
9. The automatic segmentation method according to any one of claims 1 to 6, characterized in that The N reconstruction planes used include a plurality of mutually parallel axial planes (1).
10. A system for performing an automatic segmentation method, the automatic segmentation method as claimed in any one of claims 1 to 9, characterized in that The system comprises at least one computer device hosting N different convolutional neural networks (CNNᵢ) in a structured arrangement and allowing the N different convolutional neural networks (CNNᵢ) to work together in a serial or parallel organized manner, where N ≥ 3, wherein each CNNᵢ is adapted and configured to automatically and independently of the other CNNs perform a segmentation of anatomical and pathological structures or instruments visible in the 3D medical image to be processed by analyzing the voxels of the volume forming the 3D image according to N different reconstruction planes, i.e. by performing N segmentations, each segmentation being performed by analyzing and segmenting a 2D medical image according to one of the N different reconstruction planes with the help of a corresponding dedicated CNNᵢ among the provided N CNNs, the 2D medical image being formed by all voxels belonging to each of a given plane and forming pixels of the 2D medical image, each CNNᵢ being therefore assigned to the analysis of the voxels belonging to one plane, and wherein the system further comprises means for combining and displaying the results of the analysis and 2D segmentation performed by the N different CNNᵢ.
11. The system according to claim 10, characterized in that The N different CNNs are arranged in a serial architecture as follows: each CNNᵢ of the N different CNNs analyzes all voxels of the volume forming the 3D image according to a specific reconstruction plane for each of the N different CNNs, and if there is a previous network CNN i-1 Use the CNN network created by the previous i-1 The results provided.
12. The system according to claim 10, characterized in that The N different CNNs are arranged into a parallel architecture, and within a single algorithm framework, the results of the N different CNNs are combined in the final stage.