Systems and methods for image segmentation and classification using deep convolutional neural networks

Abstract

The present invention is entitled System and Method for Image Segmentation and Classification Using Deep Down-Convolutional Neural Networks. The present invention provides methods and systems for image segmentation and / or classification using convolutional neural networks (CNNs). In one embodiment, a method includes: receiving an image having a first size, downsampling the image to produce a downsampled image of a predetermined size, wherein the predetermined size is less than the first size, feeding the downsampled image to a CNN, wherein a first convolutional layer of the CNN includes a plurality of first convolutional filters, each of the plurality of first convolutional filters having a receptive field size greater than a threshold receptive field size, identifying one or more anatomical structures of the downsampled image using the plurality of first convolutional filters; and mapping the one or more anatomical structures into a segmentation map or image classification using one or more subsequent layers of the CNN. In this way, the number of encoding layers of a trained CNN can be significantly reduced.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to image processing using convolutional neural networks, and more particularly to systems and methods for segmenting and / or classifying medical images using convolutional neural networks with reduced depth. Background Art

[0002] Medical imaging systems are often used to obtain internal physiological information of a subject (e.g., a patient). For example, medical imaging systems can be used to obtain images of a patient's skeletal structure, brain, heart, lungs, and various other features. Medical imaging systems can include magnetic resonance imaging (MRI) systems, computed tomography (CT) systems, x-ray systems, ultrasound systems, and various other imaging modalities.

[0003] Analysis and processing of medical images increasingly include the use of machine learning models to segment and / or classify anatomical regions of interest. One such method for segmenting and / or classifying medical images includes using multiple convolutional layers of a convolutional neural network (CNN) to identify features present in the medical image, and mapping the identified features into a segmentation map or image classification. For example, an MRI image of an organ of interest may be acquired, and the region of the image including the organ of interest may be automatically marked / segmented in a segmentation map generated by a trained CNN. As another example, an image of a patient's abdomen may be classified as an abdominal image by using one or more convolutional layers to identify one or more features of the image, and passing the identified features to a classification network that is configured to output the most likely image classification of the medical image from a finite list of predetermined image classification labels.

[0004] One disadvantage associated with conventional CNNs is that a large number of convolutional layers are required to identify the anatomical region of interest and / or classify medical images. For example, one limitation of deep CNNs is the vanishing gradient phenomenon encountered when attempting to train conventional CNNs, where the gradient of the cost / loss function for learning the convolution filter weights decreases with each layer of the CNN, which can lead to slow and computationally intensive training of "deep" networks. A related limitation of conventional CNNs is the large parameter space to be optimized during training, because the number of convolutional filter weights to be optimized increases with each additional convolutional layer, and the probability of convergence to a local optimum increases with the number of parameters to be optimized. Conventional CNNs, which may include hundreds of thousands to millions of parameters, can consume a lot of computing resources during training and during specific implementations. This can lead to longer training times and slower medical image analysis. Furthermore, conventional CNNs may perform particularly poorly when attempting to segment a region of interest that occupies a relatively large portion of a medical image (e.g., greater than 20% of the area of ​​the medical image), or when attempting to determine an image classification (which may rely on information from the spatial distance portion of the medical image), because conventional convolutional filters include receptive fields that occupy a small portion of the image, and thus such segmentation / classification relies on the CNN to "learn" the correct set of relatively small features into the desired larger combined features. Summary of the invention

[0005] The inventors of this article have identified systems and methods for image segmentation and classification using CNNs with reduced encoder depth, which can produce high-resolution accurate segmentation maps and high-precision image classifications without consuming the computing resources or time of conventional CNNs. In one embodiment, a segmentation map or image classification can be generated by a method comprising the following steps: receiving an image having a first size, downsampling the image to produce a downsampled image of a predetermined size, wherein the predetermined size is less than the first size, feeding the downsampled image to a convolutional neural network (CNN), wherein a first convolutional layer of the CNN includes a plurality of first convolutional filters, each of the plurality of first convolutional filters having a receptive field size greater than a threshold receptive field size, using the plurality of first convolutional filters to identify one or more anatomical structures of the downsampled image, and using one or more subsequent layers of the CNN to map the one or more anatomical structures into a segmentation map or image classification. By providing a first convolutional layer of a CNN having a plurality of filters with a receptive field greater than a threshold size, larger / more complex features can be recognized by the first convolutional layer without relying on a deep encoder. Furthermore, by downsampling the image before segmentation / classification, larger convolutional filters and more convolutional filters can be used in the first convolutional layer without substantially increasing the number of parameters of the first convolutional layer compared to conventional CNNs.

[0006] In this way, it is possible to reduce the number of convolutional layers / parameters in a CNN while maintaining segmentation / classification accuracy, since a CNN comprising a reduced number of convolutional layers / parameters can be trained and implemented faster than a conventional CNN, and furthermore, the probability of the CNN learning a set of locally optimal (and not globally optimal) parameters may be reduced.

[0007] The above advantages and other advantages and features of the present specification will be apparent from the following detailed description when taken alone or in conjunction with the accompanying drawings. It should be understood that the above summary is provided to introduce a selection of concepts further described in the detailed description in a simplified form. This is not meant to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. In addition, the claimed subject matter is not limited to embodiments that address any disadvantages mentioned above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Various aspects of the present disclosure may be better understood by reading the following detailed description and referring to the accompanying drawings, in which:

[0009] Figure 1 A block diagram illustrating an exemplary embodiment of an image processing system is shown;

[0010] Figure 2 An embodiment of an image segmentation system including a deep reduced CNN is shown;

[0011] Figure 3 A flowchart illustrating an exemplary method for segmenting a medical image using a deep reduced CNN is shown;

[0012] Figure 4 A flowchart illustrating an exemplary method of using a deep reduced CNN to determine image classification;

[0013] Figure 5 A flowchart of a first exemplary method for refining region of interest boundaries in a segmentation map produced by a depth reduction CNN is shown;

[0014] Figure 6 A flowchart of a second exemplary method for refining region of interest boundaries in a segmentation map produced by a depth reduction CNN is shown;

[0015] Figure 7 A flowchart showing an exemplary method for training a depth-reduced CNN is shown;

[0016] Figure 8 An exemplary embodiment of a first method for refining a region of interest boundary in a segmentation map is shown; and

[0017] Fig. 9 An exemplary embodiment of a second method for refining the boundaries of a region of interest in a segmentation map is shown;

[0018] The accompanying drawings illustrate specific aspects of the described systems and methods for determining segmentation maps and / or image classification of medical images using a deep reduced CNN. Together with the following description, the accompanying drawings illustrate and explain the structures, methods, and principles described herein. In the accompanying drawings, the sizes of components may be exaggerated or otherwise modified for clarity. Well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the described components, systems, and methods. DETAILED DESCRIPTION

[0019] The following description relates to systems and methods for segmenting and / or classifying medical images using a deep reduced CNN. A conventional CNN architecture includes a plurality of convolutional layers (the plurality of convolutional layers are also referred to as encoders or the encoding portion of a CNN) configured to detect features present in an input image, and a plurality of subsequent layers configured to map the identified features into one or more outputs, such as a segmentation map or image classification. Each convolutional layer includes one or more convolutional filters, and each convolutional filter "takes over" / receives input from each sub-region of an input image or a previous feature map, thereby identifying pixel intensity patterns and / or feature patterns that match the learned weights of the convolutional filter. The size of the sub-region of the input image or previous feature map from which the convolutional filter receives input is called the kernel size or receptive field size of the convolutional filter. Convolutional filters with smaller receptive field sizes are limited to identifying relatively small features (e.g., lines, edges, corners), while convolutional filters with larger receptive fields (or convolutional filters located at deeper layers of the encoding portion of a CNN) are able to identify larger features / complex features (e.g., eyes, noses, faces, etc.).

[0020] Conventional CNNs for medical image segmentation and / or classification include a relatively deep encoding portion, typically including five or more convolutional layers, wherein each of the convolutional layers includes a convolutional filter with a relatively small receptive field size (e.g., 3×3 pixels / feature channel, which corresponds to approximately 0.0137% of the area of ​​a conventional 256×256 input image). In conventional methods, the input image is 256×256, which is a standard size in the field of image processing, but in some applications, larger images may be used. Images smaller than 256×256 are generally not used for neural network-based image processing because the information content of the image may decrease as the resolution decreases. In conventional CNNs, relatively shallow convolutional layers (e.g., the first convolutional layer) extract atomic / elemental features such as lines and edges, while deeper convolutional layers extract composite features representing combinations of features identified / extracted by previous layers, such as a first convolutional layer that identifies corners and lines in an image, and a second convolutional layer that identifies squares and triangles in an image based on a combination / pattern of previously identified corners and lines. Conventional CNNs use "deep" networks (e.g., networks that include 5 or more convolutional layers) where the receptive field size of the convolutional filters in the first convolutional layer is relatively small, such as 3×3. Conventional CNNs show poor performance in segmenting regions of interest (ROIs) that occupy a relatively large portion of the image (e.g., greater than 25%) and in image classification tasks that involve classifying the entire image based on the overall content of the image. In other words, conventional CNNs utilize convolutional filters with receptive fields that are significantly smaller than the image to be classified or the ROI to be segmented, and therefore rely on the CNN to learn how to synthesize the relatively small spatial features extracted by the first convolutional layer into larger features (such as ROIs) to be labeled / segmented, or image classification based on the content of the entire image. Combining small spatial features extracted by earlier convolutional layers of the network with later convolutional layers of the network is very sensitive to the choice of network architecture and the training process, and is also prone to causing network parameters to converge to local minima during training, because using larger and deeper networks means searching a much larger number of network parameters during training and subsequently higher dimensionality loss cases. Another disadvantage of using deep CNNs with a large number of parameters is the time and computational resources used during training and implementation.

[0021] The inventors of this article have identified systems and methods that can at least partially solve the above-mentioned problems. In one embodiment, a method for segmenting and / or classifying an image includes: receiving an image having a first size, downsampling the image to produce a downsampled image of a predetermined size, wherein the predetermined size is less than the first size, feeding the downsampled image to a trained CNN, wherein a first convolutional layer of the trained CNN includes a plurality of first convolutional filters, each of the plurality of convolutional filters having a receptive field size greater than a threshold size, using the plurality of first convolutional filters to identify one or more features of the downsampled image, and using one or more subsequent layers of the trained CNN to map the one or more features into a segmentation map or image classification. By setting a receptive field size threshold based on the ROI to be segmented and / or the image size, richer feature spatial relationships can be identified earlier in the CNN, thereby enabling more effective identification of ROIs and / or large image features related to overall image classification. In one embodiment, the first convolutional layer of the CNN may include a plurality of convolutional filters having a receptive field size of 6% to 100% (and any amount in between) of the size of the downsampled input image. Furthermore, downsampling the images before feeding them to a trained CNN enables the use of convolutional filters with larger receptive field sizes relative to the input image size, and / or the use of a larger number of convolutional filters, without an accompanying increase in computational complexity, training time, implementation time, etc.

[0022] In one embodiment, if Figure 1 The image processing system 100 shown may store one or more trained depth reduction CNNs in the convolutional neural network module 108. The trained CNNs stored in the convolutional neural network module 108 may be based on the following examples: Figure 7 The image processing system 100 may receive and process images acquired via various imaging modalities (such as MRI, X-ray, ultrasound, CT, etc.), and may determine a segmentation map of one or more ROIs present in the image, and / or determine a standard view classification of one or more images. Specifically, the image processing system 100 may implement the following: Figure 3 The method 300 shown in FIG. Figure 2 The image segmentation system 200 shown in the figure can generate a segmentation map of one or more ROIs in the image. The image processing system 100 can also be used as Figure 4One or more operations of method 400 shown determine the image classification of the image. The segmentation map generated according to one or more operations of method 300 can be further processed according to one or more operations of method 500 and / or method 600 to refine the ROI boundaries of one or more ROIs identified therein. The ROI boundary refinement methods of method 500 and method 600 are shown in Figures 800 and 900, respectively.

[0023] Now turn to Figure 1 , which shows an image processing system 100 according to an exemplary embodiment. In some embodiments, the image processing system 100 is incorporated into an imaging system such as a medical imaging system. In some embodiments, at least a portion of the image processing system 100 is disposed at a device (e.g., an edge device, a server, etc.) that is communicatively coupled to the medical imaging system via a wired connection and / or a wireless connection. In some embodiments, at least a portion of the image processing system 100 is disposed at a device (e.g., a workstation) located away from the medical imaging system, the device being configured to receive images from the medical imaging system or from a storage device configured to store images acquired by the medical imaging system. The image processing system 100 may include an image processing device 102, a user input device 130, and a display device 120. In some embodiments, the image processing device 102 may be communicatively coupled to a picture archiving and communication system (PACS), and may receive images from the PACS and / or send images to the PACS.

[0024] Image processing device 102 includes processor 104, and this processor is configured to execute the machine-readable instruction stored in non-transitory memory 106.Processor 104 can be single-core or multi-core, and the program executed thereon can be configured to be used for parallel or distributed processing.In some embodiments, processor 104 can optionally include separate components distributed in two or more devices, and it can be remotely located and / or configured for coordinated processing.In some embodiments, one or more aspects of processor 104 can be virtualized and executed by a networked computing device capable of remote access configured with a cloud computing configuration.

[0025] The non-transitory memory 106 may store a convolutional neural network module 108, a training module 112, and image data 114. The convolutional neural network module 108 may include one or more trained or untrained convolutional neural networks, including a plurality of weights and biases, activation functions, pooling functions, and instructions for implementing one or more convolutional neural networks to segment ROIs and / or determine image classifications of various images, including 2D and 3D medical images. In some embodiments, the convolutional neural network module 108 may include a depth-reduced CNN having less than 5 convolutional layers, and the one or more depth-reduced CNNs may be used to determine a segmentation map and / or image classification of an input medical image by performing one or more operations of methods 300 and / or 400.

[0026] The convolutional neural network module 108 may include various metadata related to trained and / or untrained CNNs. In some embodiments, the CNN metadata may include an indication of the training data used to train the CNN, the training method used to train the CNN, and the accuracy / validation score of the trained CNN. In some embodiments, the convolutional neural network module 108 may include metadata indicating the following: the type of ROI for which the CNN was trained to generate a segmentation map, the size of the input image that the trained CNN is configured to process, and the type of anatomical structure and / or the type of imaging modality to which the trained CNN can be applied. In some embodiments, the convolutional neural network module 108 is not provided at the image processing device 102, but is provided at a remote device that is communicatively coupled to the image processing device 102 via a wired or wireless connection.

[0027] The non-transitory memory 106 also includes a training module 112 that includes machine executable instructions for training one or more CNNs in the deep neural network stored in the convolutional neural network module 108. In some embodiments, the training module 112 may include instructions for training the deep neural network according to the following example: Figure 7 Instructions for training a reduced depth CNN by performing one or more operations of method 700 shown in FIG. 1 and discussed in more detail below. In one embodiment, training module 112 may include a gradient descent algorithm, a loss / cost function, and machine executable rules for generating and / or selecting training data for training a reduced depth CNN. In some embodiments, training module 112 is not provided at image processing device 102, but is remotely provided and communicatively coupled to image processing device 102.

[0028] The non-transitory memory 106 may also include an image data module 114, which includes images / imaging data acquired by one or more imaging devices, including but not limited to ultrasound images, MRI images, PET images, X-ray images, CT images. The images stored in the image data module 114 may include medical images from various imaging modalities or from various medical imaging device configurations / models, and may include images of various views of anatomical regions of one or more patients. In some embodiments, the medical images stored in the image data module 114 may include information identifying the imaging modality and / or imaging device (e.g., the model and manufacturer of the imaging device) that acquired the medical image. In some embodiments, the images stored in the image data module 114 may include metadata indicating one or more acquisition parameters used to acquire the image. In one example, the metadata of the image may be stored in the DICOM header of the image. In some embodiments, the image data module 114 may include an x-ray image acquired by an x-ray device, an MR image captured by an MRI system, a CT image captured by a CT imaging system, a PET image captured by a PET system, and / or one or more additional types of medical images.

[0029] In some embodiments, non-transitory memory 106 may include components disposed on two or more devices that may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of non-transitory memory 106 may include a remotely accessible networked storage device configured in a cloud computing configuration.

[0030] The image processing system 100 may also include a user input device 130. The user input device 130 may include one or more of a touch screen, a keyboard, a mouse, a trackpad, a motion sensing camera, or other devices configured to enable a user to interact with and manipulate data within the image processing system 100. In some embodiments, the user input device 130 enables a user to select one or more types of ROIs to be segmented in a medical image.

[0031] The display device 120 may include one or more display devices utilizing almost any type of technology. In some embodiments, the display device 120 may include a computer monitor, a touch screen, a projector, or other display devices known in the art. The display device 120 may be configured to receive data from the image processing device 102 and display a segmentation map of the medical image, which segmentation map displays the location of one or more regions of interest. In some embodiments, the image processing device 102 may determine a standard view classification of the medical image, may select a graphical user interface (GUI) based on the standard view classification of the image, and may display the medical image and the GUI via the display device 120. The display device 120 may be combined with the processor 104, the non-transitory memory 106, and / or the user input device 130 in a shared housing, or may be a peripheral display device, and may include a monitor, a touch screen, a projector, or other display devices known in the art, which may enable a user to view images, and / or interact with various data stored in the non-transitory memory 106.

[0032] It should be understood that Figure 1 The image processing system 100 shown is for illustration and not limitation. Another suitable image processing system may include more, fewer, or different components.

[0033] Go to Figure 2 , a block diagram of an image segmentation system 200 for generating segmentation maps from medical images using a depth reduction CNN is shown. The image segmentation system 200 can be implemented by an image processing system (such as the image processing system 100) or other appropriately configured computing systems. The image segmentation system 200 is configured to receive one or more images (such as image 202) including a region of interest, and use a depth reduction CNN 221 to map the one or more received images into one or more corresponding segmentation maps, such as an upsampled segmentation map 214. In some embodiments, the segmentation system 200 can be configured to segment one or more anatomical regions of interest, such as kidneys, blood vessels, tumors, etc., or to segment non-anatomical regions of interest, such as traffic signs, text, vehicles, etc.

[0034] The image 202 may include a two-dimensional (2D) or three-dimensional (3D) array of pixel intensity values ​​in one or more color channels, or a time series of a 2D or 3D array of pixel intensity values. In some embodiments, the image 202 includes a grayscale / grayscale image. In some embodiments, the image 202 includes a color image with two or more color channels. The image 202 may include a medical image or a non-medical image, and may include an anatomical region or a non-anatomical region of interest. In some embodiments, the image 202 may not include the region of interest. The segmentation system 200 may receive the image 202 from a medical imaging device or other imaging device via a wired or wireless connection, and may also receive metadata associated with the image 202, which indicates one or more acquisition parameters of the image 202, including an indication of the imaging modality used to acquire the image 202, an indication of the imaging device settings used to acquire the image 202, an indication of the field of view (FOV) data, etc.

[0035] The image 202 may have a first size / resolution, i.e., may include a matrix / array of pixel intensity values ​​having a first number of data points. The first size / resolution may be a function of the imaging modality and imaging settings used to acquire the image 202 and / or the file format used to store the image 202. As indicated by downsampling 216, the image 202 is downsampled from the first size to a downsampled image 204 having a smaller second size / resolution. The second size / resolution may be predetermined based on the desired number of input nodes / neurons to be used in the depth reduction CNN 221. In some embodiments, the number of pixel intensity values ​​of the image 202 may be reduced to a predetermined number of pixel intensity values ​​by downsampling 216, wherein the predetermined number of pixel intensity values ​​corresponds to the number of input nodes / neurons in the depth reduction CNN. The desired number of input nodes / neurons may then be selected based on the desired receptive field size of the convolutional filters in the first convolutional layer 218 and / or the desired total number of convolutional filters in the first convolutional layer 218, such that as the receptive field size of the convolutional filters in the first convolutional layer 218 increases or as the total number of convolutional filters in the first convolutional layer 218 increases, the number of input nodes / neurons (and thus the second size) decreases, thereby keeping the total number of network parameters below a threshold number of parameters. By keeping the total number of network parameters below the threshold number of network parameters, the number of network parameters is associated with computational complexity and implementation time, thereby controlling the computational complexity and / or implementation time within a predetermined range.

[0036] Downsampling 216 may include one or more data pooling or downsampling operations, including one or more of max pooling, decimation, average pooling, and compression. In some embodiments, downsampling 216 may include dynamically determining a downsampling ratio, wherein the downsampling ratio is determined based on a ratio of a first size of image 202 to a predetermined second size of downsampled image 204, wherein as the ratio of the first size to the second size increases, the downsampling ratio also increases. Downsampling 216 generates downsampled image 204 from image 202.

[0037] The downsampled image 204 includes a downsampled / compressed image, wherein the size / resolution of the downsampled image 204 is equal to a predetermined size, also referred to herein as a second size. As described above, the selection of the predetermined size may reduce the desired number of parameters / weights of the CNN 221 based on depth. By reducing the size of the image 202 to produce the downsampled image 204, the receptive field of one or more convolution filters of the first convolution layer 218, such as the first convolution filter 217, may occupy a threshold area greater than one or more regions of interest without causing a significant reduction in computational efficiency or a significant increase in implementation time. The ROI present in the image 202 may occupy a smaller number of pixels in the image 202 in the downsampled image 204. For example, the image of an organ within the image 202 may occupy an area of ​​400 pixels, and when the image 202 is downsampled using a downsampling ratio of 2, the downsampled image of the organ within the downsampled image 204 may occupy 100 pixels, and the convolution filter weights are reduced by 75%. In this manner, a convolution filter having a receptive field size of 100 pixels may cover a larger portion of the organ (as imaged in the downsampled image 204) than a convolution filter having a receptive field size of 400 pixels. Thus, by downsampling the image 202, a convolution filter that covers a larger portion of the ROI may be employed in the first convolution layer 218 of the depth reduction CNN 221, such that the convolution filter is able to cover a threshold area larger than one or more regions of interest without proportionally increasing the number of convolution filter weights.

[0038] The downsampled image 204 may be fed to an input layer of the depth reduction CNN 221 and propagated / mapped to a segmentation map of one or more ROIs within the downsampled image 204, such as the segmentation map 212. The depth reduction CNN 221 is shown to include a first convolutional layer 218 having a plurality of first convolutional filters (including a first convolutional filter 217), a second convolutional layer 220 having a plurality of second convolutional filters, a third convolutional layer 222 having a plurality of third convolutional filters, and a classification layer 224 configured to receive features extracted by the convolutional layer and generate the segmentation map 212 therefrom. The depth reduction CNN may also include additional non-convolutional layers, such as an input layer, an output layer, a fully connected layer, etc. However, the depth reduction CNN 221 is not limited to the first convolutional layer 218. Figure 2The number and arrangement of convolutional layers are emphasized.

[0039] The first convolutional layer 218 maps image data (e.g., pixel intensity data) from the downsampled image 204 into a plurality of first feature maps 206, which include one or more extracted / identified features generated by the plurality of first convolutional filters of the first convolutional layer 218. Each of the plurality of first convolutional filters has a receptive field size greater than a predetermined receptive field size threshold. The receptive field size threshold may be predetermined based on a predetermined second size of the downsampled image 204, and further based on an expected relative coverage / shape of one or more regions of interest to be segmented. In some embodiments, the receptive field size threshold may be set to 20% or more of the expected size of the anatomical structure (e.g., blood vessels, femurs, organs, etc.). In one embodiment, if the region of interest is expected to occupy 25% of the image area / volume, the threshold receptive field size of the convolutional filters within the first convolutional layer 218 may be set to 0.25×A×B, where A is the predetermined second size of the downsampled image 204, and B is the expected relative coverage of the ROI. In some embodiments, the expected relative coverage of the ROI may be 20% or more. In a specific example, if the downsampled image 204 includes a 100×100 matrix of pixel intensity values, the region of interest to be segmented is expected to occupy 30% of the image 202, and the receptive field is expected to cover 80% of the region of interest, then the receptive field size threshold may be set to (100×100)×0.30×0.8=2,400 pixels.

[0040] The shapes of the multiple first convolution filters can also be set based on the expected shape of the ROI to be segmented. In one example, if the ROI to be segmented has a generally rectangular shape, the shapes of the receptive fields of the multiple first convolution filters can be set to a rectangle (in the case of a 2D image) or a cuboid (in the case of a 3D image). If the shape of the ROI to be segmented includes one or more axes of symmetry, or includes one or more repeating subunits, the shapes and sizes of the receptive fields of the multiple first convolution filters can be set based thereon, thereby taking advantage of the symmetry or modular nature of the ROI. In one embodiment, if the ROI to be segmented includes a blood vessel whose length is much greater than the width / diameter, the convolution filter in the first convolution layer 218 of the depth reduction CNN221 may have a receptive field with a width of several pixels (e.g., 1 to 5 pixels), and the length is set based on an estimate of the width of the blood vessel to be segmented, so that the receptive field can cover at least a majority of the width of the blood vessel without covering a majority of the length of the blood vessel. As in Figure 2 As can be seen in FIG. 2 , the first convolution filter 217 occupies most of the region of interest to be segmented and has a square shape since the ROI to be segmented has a substantially rectangular shape.

[0041] The number of the plurality of first convolution filters may be set based on the expected shape / size variation of the ROI to be segmented. In general, as the receptive field size of the plurality of first convolution filters increases, the number of the plurality of first convolution filters may also increase to cope with the increased range of possible shapes / sizes of features that may be identified / extracted therefrom. An advantage of employing convolution filters in the first convolution layer 218 having a relatively large receptive field size is that the depth of the CNN 221 may be reduced, because small features are no longer aggregated through multiple convolution layers to form larger features, which are then used to segment the ROI. Conversely, using convolution filters having a receptive field size that covers a large portion of the ROI in at least one dimension, features that include a significant portion (e.g., greater than 25%) of the ROI are identified in the first convolution layer 218. As the receptive field size threshold increases, the number of the plurality of first convolution filters may also increase to cope with the diversity of shapes / sizes of features that may be identified via the plurality of first convolution filters. In other words, as the depth decreases, the CNN 221 is configured to identify features in the first convolution layer that are larger than features conventionally detected in the first convolution layer of the CNN, and the range of variation of the features may also be greater than the range of variation of features detected in the first layer of the conventional CNN, so a greater number of convolution filters may be used in the first convolution layer 218 than in the subsequent layers 220, 222, and 224. Similarly, the receptive field size of the plurality of first convolution filters in the first convolution layer 218 may be greater than the receptive field size of the convolution filters in the subsequent convolution filters 220 and 222.

[0042] The plurality of first feature maps 206 include a plurality of output values ​​from the first convolutional layer 218, wherein each output value corresponds to a degree of match between one or more convolutional filters in the first convolutional layer 218 and the pixel intensity data of the downsampled image 204. Each different filter in the first convolutional layer 218 may generate a different feature map in the plurality of first feature maps 206. The first convolutional layer 218 identifies / extracts features from the downsampled image 204, and for each convolutional filter in the plurality of first convolutional filters, generates a corresponding feature map in the plurality of first feature maps 206. In other words, the number of feature maps in the plurality of first feature maps 206 is equal to the number of the plurality of first convolutional filters in the first convolutional layer 218. As the number of convolutional filters in the plurality of first convolutional filters is greater than the number of convolutional filters in the subsequent convolutional layers of the depth reduction CNN 221, the number of feature maps in the plurality of first feature maps is also greater than the number of feature maps in the plurality of second feature maps 208 or the plurality of third feature maps 210.

[0043] The second convolution layer 220 receives the plurality of first feature maps 206 as input and identifies / extracts feature patterns therein using a plurality of second convolution filters to generate a plurality of second feature maps 208. The second convolution layer 220 may include one or more convolution filters, wherein the receptive field size of the one or more convolution filters of the second convolution layer 220 may be smaller than a threshold receptive field size.

[0044] The plurality of second feature maps 208 may include a plurality of output values ​​generated by applying one or more convolution filters of the second convolution layer 220 to the plurality of first feature maps 206 .

[0045] The third convolution layer 222 receives the plurality of second feature maps 208 as input and uses a plurality of third convolution filters to identify / extract feature patterns therein to generate a plurality of third feature maps 210. The third convolution layer 222 may include one or more convolution filters, wherein the receptive field size of the one or more convolution filters of the third convolution layer 222 may be smaller than the threshold receptive field size.

[0046] The plurality of third feature maps 210 may include a plurality of output values ​​generated by applying one or more convolution filters of the third convolution layer 222 to the plurality of second feature maps 208. The plurality of third feature maps 210 are passed to the classification layer 224.

[0047] The classification layer 224 receives the plurality of third feature maps 210 as input and maps the features represented therein into a classification label for each of the plurality of pixels of the downsampled image 204. Based on the learned parameters of the depth reduction CNN 221, the classification label may include a label indicating which of a limited and predetermined set of categories a given pixel most likely belongs to. In one embodiment, the classification layer 224 classifies each pixel of the downsampled image 204 as belonging to an ROI or belonging to a non-ROI. In some embodiments, the depth reduction CNN 221 may generate a segmentation map including more than one type of ROI, and the classification label output by the classification layer 224 may include an indication of which type of ROI the pixel belongs to, or whether the pixel does not belong to an ROI. The classification layer 224 may include a softmax or other similar function known in the field of machine learning, which may receive one or more feature channels corresponding to a single location or sub-region of the downsampled image 204 as input, and it may output a single most likely classification label for the location or sub-region.

[0048] The output of the classification layer 224 includes a matrix or array of pixel classifications and is referred to as a segmentation map 212. Figure 22, the segmentation map 212 visually indicates the region of the downsampled image 204 corresponding to the ROI. The ROI indicated by the segmentation map 212 is the kidney, however it should be understood that a segmentation system such as the segmentation system 200 may be used to segment various other anatomical regions of interest or non-anatomical regions of interest. The size / resolution of the segmentation map 212 is substantially similar to the second size of the downsampled image 204, and therefore has a size / resolution that is substantially smaller than the first size / resolution of the image 202.

[0049] As shown by upsampling 226, the segmentation map 212 is upsampled to generate an upsampled segmentation map 214, wherein the size / resolution of the upsampled segmentation map 214 is equal to the first size of the image 202. The upsampling may include one or more known image upscaling methods, such as maximum upsampling, minimum upsampling, average upsampling, bilinear interpolation, etc. In one embodiment, the upsampling 226 may include applying one or more upconvolution filters to the segmentation map 212.

[0050] Since the segmentation map 212 is generated from the downsampled data of the downsampled image 204, the upsampled segmentation map 214 may include a pixelated / coarse ROI boundary, which is visible in the upsampled segmentation map 214. The inventors herein have identified systems and methods for refining the coarse / pixelated ROI boundary, which will be referred to below. Figure 5 and Figure 6 Discuss in more detail.

[0051] Thus, the exemplary segmentation system 200 illustrates one embodiment of a system that can receive an image and generate a segmentation map using a depth-reduced CNN 221 (with associated reduced training complexity / time and implementation complexity / time), wherein the depth of the depth-reduced CNN 221 can be significantly reduced (e.g., less than 6 convolutional layers) compared to a conventional CNN while maintaining the accuracy of the generated segmentation map. In contrast to conventional approaches, the image segmentation system 200 is able to directly learn (and thus subsequently identify) the structure of large features that include a large portion of the range of the region of interest in at least a first dimension (e.g., length, width, height, depth) in one convolutional filter in a first convolutional layer, thereby increasing the likelihood of accurately identifying the ROI and significantly reducing the dimensionality of the network parameter optimization space, thereby increasing training speed, inference, and reducing the probability of overfitting and increasing the probability of converging to a parameter set that globally minimizes the cost.

[0052] It should be understood that the architecture and configuration of the reduced depth CNN 221 are for illustration and not limitation. Without departing from the scope of the present disclosure, other appropriate CNN architectures may be used herein to determine segmentation maps and / or image classifications. Specifically, without departing from the scope of the present disclosure, additional layers may be used, including fully connected / dense layers, regularization layers, etc. In addition, after one or more convolutional layers and / or other layers, various types of activation functions known in the field of machine learning may be used. These described embodiments are merely examples of systems and methods for determining segmentation maps using a depth-reduced CNN, and the skilled person will understand that the specific details described in the embodiments may be modified in practice without departing from the essence of the present disclosure.

[0053] Go to Figure 3 , a flow chart of an exemplary method 300 for generating a segmentation map of a ROI using a depth reduction CNN is shown. In one embodiment, the image processing system 100 implementing the segmentation system 200 may perform one or more operations of the method 300 to generate a segmentation map of the ROI.

[0054] Method 300 may begin at operation 302, which includes an image processing system receiving an image having a first size. In some embodiments, the image may include a 2D image or a 3D image, or a time sequence of 2D images or 3D images. In some embodiments, the image includes a medical image acquired via a medical imaging device, and may include an anatomical region of interest to be segmented, such as an organ, a tumor, an implant, or other region of interest. The image may include metadata indicating the type of anatomical region of interest captured by the medical image, what imaging modality was used to acquire the image, the size of the image, and one or more acquisition settings / parameters used during acquisition of the image.

[0055] At operation 304, the image processing system downsamples the image received at operation 302 to produce a downsampled image with a predetermined second size, wherein the second size is smaller than the first size. In some embodiments, the second size is less than 50% of the first size. The downsampling ratio can be dynamically determined based on the ratio between the first size and the predetermined size. Downsampling can include pooling pixel intensity data of the image received at operation 302, compressing pixel intensity data, and / or extracting pixel intensity data, thereby producing a downsampled image of a predetermined size. In some embodiments, the predetermined size can be dynamically selected from a list of predetermined sizes based on the ROI to be segmented and / or the indicated ROI included in the image. In some embodiments, the second size can be selected so that the anatomical structure included therein maintains sufficient resolution to be recognized by the observer.

[0056] At operation 306, the image processing system feeds the downsampled image generated at operation 304 to a trained depth reduction CNN, wherein a first convolutional layer of the trained depth reduction CNN includes a plurality of first convolutional filters having a receptive field size greater than a threshold receptive field size. In embodiments where the input image received at operation 302 includes a 2D image containing 2D imaging data, the receptive field size threshold is a receptive field area threshold. In some embodiments, the receptive field area threshold includes 5% to 100% of an area of ​​a predetermined size of the downsampled image, or any fraction therebetween. In embodiments where the input image received at operation 302 includes a 3D image containing 3D imaging data, the receptive field size threshold is a receptive field volume threshold. In some embodiments, the receptive field volume threshold includes 5% to 100% of a volume of a predetermined size of the downsampled image, or any fraction therebetween.

[0057] In some embodiments, the image processing system may select a trained depth reduction CNN from a plurality of trained depth reduction CNNs based on one or more ROIs for which the trained depth reduction CNN is trained to generate a segmentation map. The image processing system may determine which type of ROI the trained depth reduction CNN is configured to segment based on metadata associated with the trained depth reduction CNN. In some embodiments, a threshold receptive field size of the trained depth reduction CNN is selected based on the desired ROI to be segmented, and also based on the shape / aspect ratio of the desired ROI. In some embodiments, a threshold receptive field size of a plurality of first convolutional filters in a first convolutional layer of the trained depth reduction CNN may be selected based on an expected / estimated coverage of the ROI in the downsampled image (before training). In some embodiments, a first number of the plurality of first convolutional filters is greater than the number of convolutional filters in any of the one or more subsequent layers of the trained depth reduction CNN. In some embodiments, the first number of the plurality of first convolutional filters is in the range of 100 to 3000 (inclusive), or any integer therebetween. In some embodiments, none of the subsequent layers of the trained depth reduction CNN include convolution filters having a receptive field size greater than a threshold receptive field size. In some embodiments, the trained CNN includes less than 6 convolutional layers. In some embodiments, the receptive field size threshold may be 50% to 100% of the size of the downsampled image. In some embodiments, at least one dimension of the receptive field size threshold may be selected based on the anatomical region of interest to be segmented. In one embodiment, the threshold size includes a threshold length, and wherein the threshold length is greater than 50% of the length of the anatomical region of interest in at least a first dimension / direction.

[0058] At operation 308, the image processing system uses a plurality of first convolution filters of the trained depth reduction CNN to identify one or more features in the downsampled image. Since the receptive field size of each of the plurality of first convolution filters in the first convolution layer accounts for a significant portion of the expected ROI, the features extracted / identified at operation 308 may include a significant portion of the ROI present in the downsampled image. In some embodiments, the entire ROI may be identified by the filter in the first convolution layer of the trained depth reduction CNN. In short, the filter identifies / extracts a pattern by calculating the dot product between the filter weights of the convolution filter and the pixel intensity values ​​of the downsampled image on the receptive field of the filter, and the greater the magnitude of the dot product, the higher the match between the filter and the pixel intensity pattern in the downsampled image. The dot product can be fed to an activation function and then output to a feature map, which records the matching degree and spatial information of the area in the downsampled image where a match is found.

[0059] At operation 310, the image processing system maps the one or more identification features identified at operation 308 into a segmentation map of one or more regions of interest using one or more subsequent layers of the trained CNN. In some embodiments, the segmentation map has a second predetermined size that is equal to the size of the downsampled image generated at operation 304. The segmentation map may include a plurality of pixel classifications corresponding to a plurality of pixels in the downsampled image, wherein each pixel classification provides a designation of which of a finite and predetermined set of categories a pixel of the downsampled image most likely belongs to. In some embodiments, one or more subsequent layers of the trained depth reduction CNN may include one or more additional convolutional layers configured to receive the feature map generated by the first convolutional neural network. In some embodiments, the receptive field size of the convolutional filters in each of the subsequent convolutional layers is less than a threshold receptive field size, and the number of convolutional filters in each of the subsequent layers is less than the number of convolutional filters in the first convolutional layer.

[0060] At operation 312, the image processing system upsamples / upscales the segmentation map to produce an upsampled segmentation map having a size equal to the first size of the image received at operation 302. Upsampling may include one or more known image upscaling methods, such as adaptive interpolation or non-adaptive interpolation, including nearest neighbor approximation, bilinear interpolation, bicubic smoothing, bicubic sharpening, etc.

[0061] At operation 314 , the image processing system refines the ROI boundary of the upsampled segmentation map based on the intensity values ​​from the image received at operation 302 , thereby generating a refined segmentation map. Figure 5 and Figure 6Two different embodiments of methods for refining ROI boundaries are provided and discussed in more detail below. Briefly, the ROI boundary is a location or region in the segmentation map where a pixel or pixel cluster classified as belonging to the ROI touches one or more pixels classified as non-ROI. Figure 2 As can be seen from the upsampled segmentation map 214 in FIG. 3 , the ROI boundary (where an area of ​​bright pixels touches an area of ​​black pixels) may be pixelated or rough, and may therefore provide an unclear indication of the end position of the ROI. By upsampling the segmentation map generated at operation 310 to generate an upsampled segmentation that matches the first size of the image received at operation 302, the original pixel intensity data of the image can be more effectively utilized to provide a refined boundary location for each ROI identified in the upsampled segmentation map, because there is a one-to-one (1-to-1) correspondence between the pixel labels in the upsampled segmentation map and the pixels in the image. Specifically, in methods 500 and 600, the intensity values ​​of the image are obtained from the area within a threshold distance of the ROI boundary identified in the upsampled segmentation map, and therefore, the intensity values ​​within the threshold distance of the ROI boundary (which includes higher resolution information than the upsampled segmentation map, which is generated by compressed / downsampled intensity data) can be used to more accurately locate the boundary between the ROI region and the non-ROI region.

[0062] At operation 316 , method 300 optionally includes an image processing system displaying the refined segmentation map generated at operation 314 to a user via a display device.

[0063] At operation 318, the method 300 optionally includes an image processing system that determines one or more of the length, width, depth, volume, shape, and orientation of the ROI based on the refined segmentation map. In some embodiments, the image processing system may employ principal component analysis of the refined segmentation map to determine one or more spatial parameters of one or more segmented ROIs of the refined segmentation map. After operation 318, the method 300 may end.

[0064] In this way, method 300 is able to generate a segmentation map using a depth-reduced CNN, wherein the segmentation map includes ROI boundaries that are substantially similar in accuracy to ROI boundaries generated by a computationally expensive conventional CNN. Since the depth-reduced CNN includes fewer convolutional layers than a conventional CNN, and therefore includes a greatly reduced number of total network parameters, the segmentation map generated by implementing method 300 can consume a portion of the computational resources of a conventional CNN and can be generated in a portion of the time of a conventional CNN. The threshold receptive field size of the convolutional filter in the first convolutional layer is set based on the expected area / volume occupied by the desired ROI in the downsampled image, so that the convolutional filter in the first convolutional layer covers most of the ROI to be identified and segmented, and the technical effect obtained in this way is that the desired ROI can be identified using a significantly reduced number of convolutional filters. In addition, the image is downsampled before segmenting one or more ROIs in the image, the segmentation map generated from the downsampled image is upsampled, and the ROI boundaries in the upsampled segmentation map are refined based on pixel intensity data from the original full-size image. The technical effect achieved in this way is that a segmentation map with substantially similar accuracy to that generated in conventional methods can be generated in a shorter time using reduced computing resources.

[0065] Now turn to Figure 4 , a flow chart of an exemplary method 400 for classifying an image using a deep reduced CNN is shown. Method 400 may be implemented by an image processing system such as image processing system 100 to determine a standard view of an image. In some embodiments, method 400 may include determining to which of a finite number of standard views a medical image belongs. In short, in medical imaging, images of an anatomical region of interest are acquired in one of a finite number of orientations and / or with a predetermined set of acquisition parameters, each different orientation / acquisition parameter set is referred to as a standard view, and in a medical imaging workflow, identifying the standard view to which a medical image belongs may inform downstream analysis and processing. Similar to the advantages that method 300 has in the case of image segmentation, method 400 is able to achieve reduced computational complexity and increased classification speed, but the method is applied to image classification.

[0066] At operation 402, an image processing system receives an image including a standard view of an anatomical ROI. In some embodiments, the image processing system receives a medical image via wired or wireless communication with a medical imaging device, the medical image including a standard view of an anatomical region of interest of an imaging object. In some embodiments, the image may include a 2D image or a 3D image, or a time sequence of 2D images or 3D images. The image may include metadata indicating the type of anatomical region of interest captured by the image, what imaging modality was used to acquire the image, the size of the image, and one or more acquisition settings / parameters used during acquisition of the image.

[0067] At operation 404, the image processing system downsamples the image received at operation 402 to produce a downsampled image having a predetermined size, wherein the predetermined size is less than the first size. The downsampling ratio may be dynamically determined based on a ratio between the first size and the predetermined size. The downsampling may include pooling pixel intensity data of the image received at operation 402, compressing the pixel intensity data, and / or decimating the pixel intensity data to produce the downsampled image of the predetermined size. In some embodiments, the predetermined size may be dynamically selected from a list of predetermined sizes based on one or more metadata associated with the image received at operation 402.

[0068] At operation 406, the image processing system feeds the downsampled image generated at operation 404 to a trained depth reduction CNN, wherein a first convolutional layer of the trained depth reduction CNN includes a plurality of first convolutional filters having a receptive field size that is 50% larger than a size (area or volume) of the downsampled image generated at operation 404. In other words, the receptive field size of each of the plurality of convolutional filters in the first convolutional layer of the trained depth reduction CNN is greater than a receptive field size threshold, wherein the receptive field size threshold is at least 50% of the area of ​​the downsampled image generated at operation 404. In some embodiments, the image processing system may select a trained depth reduction CNN from the plurality of trained depth reduction CNNs based on one or more metadata associated with the image received at operation 402. In some embodiments, a first number of the plurality of first convolutional filters of the first convolutional layer is greater than a number of convolutional filters in any of one or more subsequent layers of the trained depth reduction CNN. In some embodiments, the first number of the plurality of first convolutional filters is in the range of 100 to 800 (inclusive), or any integer therebetween. In some embodiments, none of the subsequent layers of the trained depth-reduced CNN includes a convolution filter having a receptive field size greater than a threshold receptive field size. In some embodiments, the trained depth-reduced CNN includes less than 6 convolutional layers. In some embodiments, the trained depth-reduced CNN includes a single convolutional layer. In some embodiments, the receptive field size threshold may be 50% to 100% of the size of the downsampled image. In some embodiments, the receptive field size threshold may be at least 80% of the area / volume of the downsampled image.

[0069] At operation 408, the image processing system identifies one or more features in the downsampled image using a plurality of first convolutional filters of the trained deep downsampled CNN. Since the receptive field size of each of the plurality of first convolutional filters in the first convolutional layer occupies a majority (e.g., 50%-100%) of the downsampled image, the features extracted at operation 408 include more "holistic" features than the features identified by the first convolutional layer of a conventional CNN. Specifically, in contrast to a conventional CNN, where the convolutional filters in the first convolutional layer may detect atomic features such as lines, edges, corners, the features identified by the first layer of the trained deep downsampled CNN may include "holistic" or "global" features such as the relative positioning of sub-regions within the downsampled image, the orientation of anatomical features, the overall image brightness, etc. In short, the convolutional filters identify / extract patterns by calculating the dot product between the filter weights of the convolutional filters and the pixel intensity values ​​(or feature values ​​if the input is a feature map) of the downsampled image over the receptive field of the filter, the larger the magnitude of the dot product, the higher the match between the filter and the pixel intensity pattern in the downsampled image. The dot product can be fed into an activation function and then output to a feature map that records the matching degree and spatial information of the areas in the downsampled image where matches are found.

[0070] At operation 410, the image processing system maps the one or more identification features identified at operation 408 into an image classification of the downsampled image using one or more subsequent layers of the trained depth-reduced CNN. In some embodiments, the one or more subsequent layers of the trained depth-reduced CNN may include one or more additional convolutional layers configured to receive the feature map generated by the first convolutional layer. In some embodiments, the receptive field size of the convolutional filters in each of the subsequent convolutional layers is less than the threshold receptive field size, and the number of convolutional filters in each of the subsequent layers is less than the number of convolutional filters in the first convolutional layer.

[0071] At operation 412, method 400 optionally includes an image processing device that displays a graphical user interface (GUI) via a display device, wherein the GUI is selected based on the image classification determined at operation 410. In some embodiments, the image processing workflow may include displaying a GUI based on a standard view of the image, for example, if the image classification indicates that the image includes a first anatomical region imaged at a first orientation, a GUI including features / tools specific to analysis of the first anatomical region at the first orientation may be automatically displayed at operation 412, thereby simplifying the image analysis and processing workflow. After operation 412, method 400 may end.

[0072] Using a reduced resolution / downsampled image (where the view type in the image is still identifiable), and then using multiple convolutional filters in the first layer of a deep reduced CNN with a receptive field size that is 50% larger than the size of the input image, the technical effect obtained is that a reduced number of convolutional layers can be used to identify global orientation and position features of sub-regions within the input image, so that the deep reduced CNN has higher accuracy and greater speed in classifying standard views of the input image.

[0073] In this manner, method 400 can quickly and accurately detect standard views of acquired medical images. In some embodiments, method 400 can be used in conjunction with real-time image acquisition to dynamically adjust a GUI displayed to a medical practitioner performing a scan of an anatomical region of interest to a patient.

[0074] Go to Figure 5 , a flow chart of a first embodiment of a method 500 for refining the ROI boundary of a segmentation map is shown. The method 500 can be implemented by the image processing system 100 to improve the accuracy of the boundary position of the ROI boundary of the segmentation map generated by the depth reduction CNN. When the depth reduction CNN proposed in this article receives an input downsampled image, the corresponding resolution of the output segmentation map is also reduced, which improves the speed and computational efficiency of implementing such a depth reduction CNN, and by implementing one or more operations of the method 500 or the method 600 discussed below, the accuracy and smoothness of the ROI boundary can be substantially equivalent to the ROI boundary generated using a slower and less computationally efficient CNN.

[0075] At operation 502, an image processing system receives a segmentation map (such as upsampled segmentation map 214) and a corresponding image (such as image 202), wherein the segmentation map includes a segmented anatomical region of interest. In some embodiments, the segmentation map and the image each have a first size such that for each pixel label of the segmentation map, there is a corresponding pixel at a corresponding location of the image. In some embodiments, the image processing system may receive the segmentation map and the image from a location in a non-transitory memory or from a wired or wireless communication with a remotely located computing system.

[0076] At operation 504, the image processing system determines an intensity profile of the medical image along one or more lines that pass through and are substantially perpendicular to a boundary of the segmented anatomical region of interest, wherein the one or more lines are each substantially bisected by the ROI boundary (i.e., a center of the one or more lines each substantially coincides / intersects a portion of the ROI boundary). Figure 8 , a conceptual diagram of operation 504 is shown. Figure 8As can be seen in , multiple intensity profiles such as intensity profile 802 are sampled along lines of a threshold length that pass substantially perpendicular to the boundary of the segmented ROI. The segmentation map is used to determine the position of each of the one or more lines, and then pixel intensity values ​​are sampled from the image at the corresponding position. The number of lines used may be pre-selected based on the expected shape / geometry of the ROI to be segmented. For example, as the expected complexity of the ROI boundary increases, the number of lines used at operation 504 may be increased accordingly. The threshold length of the line may be selected based on the desired computational complexity, wherein longer lines correspond to increased computational complexity and shorter lines correspond to reduced computational complexity. In addition, the threshold length of the line may be selected based on the size of the downsampled image used to generate the segmentation map, wherein as the size of the downsampled image from which the segmentation map is generated decreases, the length of the line of operation 504 may be increased to compensate for the increased pixelation that may occur in the resulting upsampled segmentation map. In an embodiment where the segmentation map includes a 3D segmentation map, method 500 and Figure 8 The line may be replaced by a plane, wherein substantially half of the plane area is within the ROI and substantially half of the plane area is outside the ROI.

[0077] At operation 506, the image processing system updates the position of the ROI boundary along each of the one or more lines based on the intensity distribution map of the image along the one or more lines, thereby generating a refined segmentation map of the anatomical region of interest. In some embodiments, the one or more lines extend from a threshold distance inside the ROI boundary to a threshold distance outside the ROI boundary. In some embodiments, one or more conventional edge detection algorithms may be used to determine the updated ROI boundary position along each of the one or more lines using the corresponding intensity distribution map of the one or more lines obtained from the original full-size image. In short, the edge detection algorithm may evaluate the change or discontinuity of the pixel intensity data along the one or more lines to determine the updated ROI boundary position. In some embodiments, each of the intensity distribution maps along each of the one or more lines may be fed to a trained neural network that is trained to map a one-dimensional intensity vector to an edge position.

[0078] In this way, method 500 can convert pixelated ROI boundaries in a segmentation map produced by a depth-reduced CNN into smooth and accurate ROI boundaries by utilizing pixel intensity data from an intelligently selected subset of regions within the original full-size image from which the segmentation map was produced.

[0079] Go to Figure 6, a flow chart of a second embodiment of a method 600 for refining ROI boundaries in a segmentation map generated by a depth reduction CNN is shown. The method 600 can be implemented by an image processing system such as the image processing system 100 to improve the smoothness and accuracy of ROI boundaries in a segmentation map generated by a depth reduction CNN.

[0080] At operation 602, an image processing system receives a segmentation map (such as upsampled segmentation map 214) and a corresponding image (such as image 202), wherein the segmentation map includes a segmented anatomical region of interest. In some embodiments, the segmentation map and the image each have a first size such that for each pixel label of the segmentation map, there is a corresponding pixel at a corresponding location of the image. In some embodiments, the image processing system may receive the segmentation map and the image from a location in a non-transitory memory or from a wired or wireless communication with a remotely located computing system.

[0081] At operation 604, the image processing system divides the medical image into one or more sub-regions, wherein each of the plurality of sub-regions includes a portion of a boundary of a segmented anatomical region of interest. Fig. 9 , a conceptual diagram of operation 604 is shown. Fig. 9 As can be seen, multiple sub-regions of pixel intensity values, such as sub-region 902, are sampled from the image received at operation 602. In some embodiments, the sub-regions are square or rectangular, and in other embodiments, the sub-regions may be circular or rectangular. Each sub-region may have a threshold area, wherein substantially half of the area of ​​the sub-region is within the ROI (within the initial estimated ROI boundary of the upsampled segmentation map), and substantially half of the area of ​​the plane is outside the ROI (outside the initial estimated ROI boundary of the upsampled segmentation map received at operation 602). The location of each sub-region is determined using the segmentation map received at operation 602, and then pixel intensity values ​​are sampled from the image received at operation 602 at the location corresponding to the sub-region. The number of sub-regions used may be pre-selected based on the expected shape / geometry of the ROI to be segmented. For example, as the expected complexity of the ROI boundary increases, the number of sub-regions used at operation 604 may be increased accordingly, and the coverage size / area of ​​the sub-regions may be reduced accordingly. In an embodiment where the segmentation map includes a 3D segmentation map, method 600 and Fig. 9 The sub-region may be replaced by a three-dimensional sub-region, such as a plane of a cube, a cuboid, a sphere, etc., wherein substantially half of the volume of the 3D sub-region is within the ROI, and substantially half of the volume of the 3D sub-region is outside the ROI.

[0082] At operation 606, the image processing system feeds the one or more sub-regions to a trained CNN, wherein the trained CNN is configured to map the matrix of pixel intensity values ​​corresponding to each sub-region into a corresponding edge segmentation map, indicating an updated position of the ROI boundary along a line (or along a plane in the case of a 3D image and a 3D segmentation map) for each sub-region. In some embodiments, identification of the ROI boundary within the one or more sub-regions may include one or more conventional edge detection algorithms.

[0083] At operation 608, the image processing system updates the location of the ROI boundary in the one or more sub-regions based on the one or more segmentation maps generated at operation 606. After operation 608, the method 600 can end. In this manner, the method 600 can convert the pixelated ROI boundary in the segmentation map generated by the depth reduction CNN into a smooth and accurate ROI boundary by utilizing the pixel intensity data from the intelligently selected region subset from within the original full-size image from which the segmentation map was generated.

[0084] Go to Figure 7 , shows a method for training a deep reduction CNN (such as Figure 2 Flowchart of an exemplary method 700 for inferring a segmentation map of one or more ROIs from an input downsampled image using a deep downsampled CNN 221 as shown. The method 700 may be performed by one or more of the systems discussed above. In some embodiments, the method 700 may be performed by Figure 1 The method 700 is implemented by the image processing system 100 shown. In some embodiments, the method 700 may be implemented by the training module 112 stored in the non-transitory memory 106 of the image processing device 102.

[0085] At operation 702, a training data pair from a plurality of training data pairs is fed to an input layer of a depth reduction CNN, wherein the training data pair comprises an image and a corresponding ground-truth segmentation map. An image processing system may intelligently select the training data pair based on one or more metadata associated with the training data pair. In one embodiment, the method 700 may be employed to train the depth reduction CNN to identify one or more predetermined types of ROIs, and operation 702 may include selecting an image processing system comprising a training data pair comprising an image, wherein the image comprises one or more of the predetermined types of ROIs, and wherein the training data pair further comprises a ground-truth segmentation map of the one or more ROIs in the image. In some embodiments, the ground-truth segmentation map may be generated by an expert, such as a radiologist.

[0086] In some embodiments, the training data pair and the plurality of training data pairs may be stored in the image processing device, such as in the image data module 114 of the image processing device 102. In other embodiments, the training data set may be obtained via a communication coupling between the image processing system and an external storage device, such as via an Internet connection to a remote server.

[0087] At operation 704, the image of the training data pair is mapped into a predicted segmentation map using a deep reduced CNN. In some embodiments, operation 704 may include inputting pixel / voxel intensity data of the image into an input layer of the deep reduced CNN, identifying features present in the image using at least a first convolutional layer including a plurality of first convolutional filters, wherein each of the plurality of convolutional filters has a receptive field size greater than a threshold receptive field size, and mapping the features extracted by the first convolutional layer into a predicted segmentation map using one or more subsequent layers. In some embodiments, the one or more subsequent layers include at least one classification layer.

[0088] At operation 706, the image processing system calculates the loss of the deep reduced CNN based on the difference between the predicted segmentation map and the true segmentation map. In other words, operation 706 includes the image processing system using the true segmentation map and the loss / cost function to determine the error of the predicted segmentation map. In some embodiments, operation 706 includes the image processing system determining a plurality of pixel classification label differences between a plurality of pixels / voxels of the predicted segmentation map and a plurality of pixels / voxels of the true segmentation map, and inputting the plurality of pixel classification label differences into a predetermined loss / cost function (e.g., an MSE function or other loss functions known in the field of machine learning). In some embodiments, the loss function may include a DICE score, a mean square error, an absolute distance error, or a weighted combination of one or more of the foregoing. In some embodiments, operation 706 may include determining the DICE score of the predicted segmentation map using the true segmentation map according to the following equation:

[0089] DICE=(S∩T) / (S∪T),

[0090] where S is the true segmentation map, and where T is the predicted segmentation map.

[0091] At operation 708, the weights and biases of the reduced depth CNN are updated based on the loss determined at operation 706. In some embodiments, the loss is back-propagated through the layers of the reduced depth CNN, and the parameters of the reduced depth CNN may be updated according to a gradient descent algorithm based on the back-propagated loss. The loss may be back-propagated through the layers of the reduced depth CNN to update the weights (and biases) of each layer. In some embodiments, back-propagation of the loss may occur according to a gradient descent algorithm, wherein the gradient (first-order derivative or an approximation of the first-order derivative) of the loss function is determined for each weight and bias of the reduced depth CNN. Each weight (and bias) of the reduced depth CNN is then updated by adding the negative of the gradient product determined (or approximated) for the weight (or bias) to a predetermined step size according to the following equation:

[0092]

[0093] Where P i+1 is the updated parameter value, P i is the previous parameter value, Step is the step size, and is the partial derivative of the loss with respect to the previous parameters.

[0094] After operation 708, method 600 may end. It should be noted that method 700 may be repeated until the weights and biases of the depth-reduced CNN converge, a threshold loss is obtained (for the training data or on a separate validation dataset), or the rate of change of the weights and / or biases of the depth-reduced CNN is below a threshold rate of change for each iteration of method 700. In this manner, method 700 enables the depth-reduced CNN to be trained to infer segmentation maps of one or more ROIs from the downsampled image.

[0095] When introducing the elements of various embodiments of the present disclosure, the words "one", "a kind of" and "the" are intended to mean that there are one or more elements in these elements. The terms "first", "second", etc. do not represent any order, amount or importance, but are used to distinguish one element from another element. The terms "comprise", "include" and "have" are intended to be inclusive, and mean that additional elements may also exist in addition to the listed elements. As used herein, the terms "connected to", "connected to", etc., an object (e.g., material, element, structure, member, etc.) can be connected to or connected to another object, regardless of whether the one object is directly connected or connected to another object, or whether there are one or more intervening objects between the one object and another object. In addition, it should be understood that reference to "one embodiment" or "embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that also include the cited features.

[0096] In addition to any previously indicated modifications, numerous other variations and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of the present description, and the appended claims are intended to cover such modifications and arrangements. Therefore, although the information has been described above in detail in conjunction with what are currently considered to be the most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that many modifications may be made, including but not limited to form, function, mode of operation, and use, without departing from the principles and concepts set forth herein. Likewise, as used herein, the examples and embodiments are intended to be illustrative in all respects only and should not be construed as limiting in any way.

Claims

1. An image processing method, the method comprising: receiving an image having a first size; downsampling the image to generate a downsampled image of a predetermined size, wherein the predetermined size is smaller than the first size; feeding the downsampled image to a convolutional neural network, wherein a first convolutional layer of the convolutional neural network comprises a plurality of first convolutional filters, each of the plurality of first convolutional filters having a receptive field size greater than a threshold receptive field size; identifying one or more anatomical structures of the downsampled image using the plurality of first convolutional filters; as well as mapping the one or more anatomical structures into an output using one or more subsequent layers of the convolutional neural network; The output includes a segmentation map of the anatomical region of interest, the method further comprising: Upsampling the segmentation map to generate an upsampled segmentation map of the first size; as well as The boundaries of the upsampled segmentation map are refined based on intensity values ​​of the image within a threshold distance of a boundary of the anatomical region of interest, thereby producing a refined segmentation map. 2 . The method according to claim 1 , wherein the receptive field size threshold is 5% to 100% of the predetermined size. The method of claim 1 , wherein the predetermined size is less than 50% of the first size.

4. The method according to claim 1, wherein the first number of the plurality of first convolution filters is in the range of 100 to 3000, or any integer therebetween. 5 . The method of claim 1 , wherein each of the one or more subsequent layers does not have an input size that is smaller than the predetermined size.

6. The method of claim 1, wherein the image comprises two-dimensional imaging data of an anatomical region of an imaged object, and the threshold receptive field size is a receptive field area threshold.

7. The method of claim 1, wherein the image comprises three-dimensional imaging data of an anatomical region of an imaged object, and the threshold receptive field size is a receptive field volume threshold.

8. The method of claim 1, wherein downsampling the image comprises determining a ratio of the first size to the predetermined size, and dynamically determining a downsampling ratio based on the ratio of the first size to the predetermined size.

9. The method of claim 1, wherein the output further comprises an image classification, the image classification comprising an indication of a standard view of the image, the method further comprising: selecting a graphical user interface based on the standard view; as well as The graphical user interface is displayed via a display device.

10. The method of claim 9, wherein the threshold receptive field size comprises a threshold area or volume, and wherein the threshold area or volume is greater than 20% of the area or volume occupied by the anatomical region of interest in the downsampled image.

11. An image processing system, comprising: A memory storing a convolutional neural network and instructions; and a processor, wherein the processor is communicatively coupled to the memory and when executing the instructions is configured to: receiving a two-dimensional image or a three-dimensional image of a first size; determining a downsampling ratio based on the first size and a predetermined size; downsampling the image using the downsampling ratio to generate a downsampled image of the predetermined size, wherein the predetermined size is smaller than the first size; Feeding the downsampled image to a convolutional neural network, wherein a first convolutional layer of the convolutional neural network includes a plurality of first convolutional filters, each of the plurality of first convolutional filters having a receptive field size greater than a threshold size; identifying one or more anatomical structures of the downsampled image using the plurality of first convolutional filters; and mapping the one or more anatomical structures into an output using one or more subsequent layers of the convolutional neural network, wherein each of the one or more subsequent layers does not include a pooling operation; wherein the output comprises a segmentation map of the anatomical region of interest, and wherein when executing the instructions, the processor is further configured to: Upsampling the segmentation map to generate an upsampled segmentation map of the first size; and The boundary of the upsampled segmentation map is refined based on intensity values ​​of the image within a threshold distance of a boundary of the anatomical region of interest, thereby producing a refined segmentation map.

12. The image processing system according to claim 11, further comprising a display device, and wherein when executing the instructions, the processor is further configured to: The refined segmentation map is displayed via the display device.

13. The image processing system of claim 11, wherein the output comprises an image classification indicating a standard view of a finite list of standard views to which the image belongs.

14. An image processing method, the method comprising: receiving a medical image including an anatomical region of interest, wherein the medical image has a first size; determining a downsampling ratio based on the first size and a predetermined size; downsampling the medical image using the downsampling ratio to generate a downsampled image of the predetermined size, wherein the predetermined size is less than 50% of the first size; feeding the downsampled image to a convolutional neural network, wherein a first convolutional layer of the convolutional neural network comprises a plurality of first convolutional filters, each of the plurality of first convolutional filters having a receptive field configured to receive data from a predetermined fraction of the downsampled image, wherein the predetermined fraction is 5% to 100% of the area or volume of the downsampled image; identifying one or more features of the downsampled image using the plurality of first convolutional filters; and Mapping the one or more features into an output using one or more subsequent layers of the convolutional neural network; Wherein the output comprises a two-dimensional segmentation map or a three-dimensional segmentation map of the anatomical region of interest, the method further comprises: Upsampling the segmentation map to generate an upsampled segmentation map; Refining the boundary of the anatomical region of interest in the upsampled segmentation map to generate a refined segmentation map; and One or more of a length, a width, a shape, and an orientation of the anatomical region of interest is determined based on the refined segmentation map.

15. The method of claim 14, wherein refining the boundary of the anatomical region of interest in the upsampled segmentation map comprises: determining a plurality of intensity profiles of the medical image along a plurality of lines passing through and substantially perpendicular to a boundary of the anatomical region of interest; as well as A location of a boundary of the anatomical region of interest in the upsampled segmentation map is updated based on the plurality of intensity distribution maps.

16. The method of claim 15, wherein updating the location of the boundary of the anatomical region of interest in the upsampled segmentation map based on the plurality of intensity distribution maps comprises: mapping each of the plurality of intensity profiles to a corresponding boundary location using a trained neural network; as well as The position of the boundary along each of the plurality of lines is updated to the corresponding boundary position.

17. The method of claim 14, wherein refining the boundary of the anatomical region of interest in the upsampled segmentation map comprises: dividing the medical image into a plurality of sub-regions, wherein each of the plurality of sub-regions includes a portion of a boundary of the anatomical region of interest; Mapping each of the plurality of sub-regions into a corresponding segmentation map using a second trained convolutional neural network; as well as The position of the boundary in each of the plurality of sub-regions is updated based on the corresponding segmentation map.

Images