Image processing method and system based on convolutional neural network

Abstract

An image processing method based on a convolutional neural network (CNN) is provided. The method includes the steps of receiving an input image, performing a plurality of feature extraction operations using a plurality of convolution layers of the CNN based on the input image to respectively generate a plurality of output feature maps, and generating an output image for the input image based on the plurality of output feature maps of the plurality of convolution layers. In particular, for each of the plurality of feature extraction operations, performing the feature extraction operation using the convolution layer includes the steps of generating an output feature map of the convolution layer based on the input feature map received by the convolution layer and the plurality of weighted coordinate maps, generating a plurality of weighted coordinate maps based on the plurality of coordinate maps and a spatial attention map, and generating a spatial attention map based on the input feature map received by the convolution layer and modifying the coordinate information in each of the plurality of coordinate maps to generate the plurality of weighted coordinate maps. A corresponding image processing system based on a CNN is also provided.

Description

[Technical field]

[0001] The present invention generally relates to image processing methods and systems based on convolutional neural networks (CNNs). [Background technology]

[0002]

[0002] Convolutional Neural Networks (CNNs) are a class of artificial neural networks well known in the art, which have been applied in various domains for prediction purposes, particularly in image processing for various predictive applications, such as image segmentation and image classification. Although CNNs may be generally understood to be applicable in various domains for various predictive applications, the use of CNNs in various predictive applications may not always provide satisfactory prediction results (e.g., not always accurate enough in image segmentation or image classification), and it may be difficult or challenging to obtain satisfactory prediction results.

[0003]

[0003] As an example, medical ultrasound imaging is a safe, non-invasive, real-time imaging diagnostic modality that uses high-frequency sound waves to provide images of structures of the human body. Compared with other imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI), ultrasound imaging is relatively cheap, portable, and more widespread, and is widely considered to be the stethoscope of the 21st century. However, ultrasound images may be obtained from hand-held probes and are therefore operator-dependent, and prone to numerous artifacts such as large speckle noise, shadowing, and blurred boundaries. This makes it more difficult to segment tissue structures of interest (e.g., anatomical structures) from adjacent tissues. Several conventional methods (e.g., active contours, graph cuts, superpixels, and deep models (e.g., fully convolutional networks (FCN), U-Net, etc.) have been proposed and adapted for ultrasound image segmentation. However, due to the noisiness of ultrasound images, such conventional methods usually produce poor results. Although deep models have achieved significant improvements over conventional methods, accurately segmenting soft tissue structures from ultrasound images remains a challenging task.

[0004]

[0004] Another problem with using a single deep model to segment ultrasound images is that it generally results in highly biased results due to blurred boundaries and textures, and has high variance due to noise and heterogeneity. To reduce both bias and variance, multi-model ensemble methods such as bagging and boosting have been proposed. However, training multiple models for ensembling is computationally expensive. To address this, it has been proposed to train a model in one pass while saving multiple sets of model weights along an optimization path through learning rate annealing. However, such methods still require multiple runs of the inference process. In an attempt to address this issue, multiple multi-stage prediction-refining deep models (e.g., HourglassNet, CU-Net, RNet, etc.) have been proposed to predict and gradually refine the segmentation results through serial modules. 3-Net, BASNet) have been developed. While such strategies may be able to reduce the splitting bias, their impact on variance is limited, meaning that while the average performance over the entire dataset may look good, they are unlikely to yield stable predictions for a variety of input images. Summary of the Invention [Problem to be solved by the invention]

[0005]

[0005] There is therefore a need to provide a CNN-based image processing method and system that overcomes or at least ameliorates one or more problems associated with conventional CNN-based image processing methods and systems, and in particular aims to enhance or improve the predictive capabilities (e.g., accuracy of prediction results) associated with CNN-based image processing, such as, but not limited to, image segmentation. Against this background, the present invention has been developed. [Means for solving the problem]

[0006] According to a first aspect of the present invention, there is provided a CNN-based image processing method using at least one processor, the method comprising: receiving an input image; performing a plurality of feature extraction operations using a plurality of convolutional layers of the CNN based on the input image to generate a plurality of output feature maps, respectively; generating an output image for the input image based on a plurality of output feature maps of the plurality of convolutional layers; For each of the plurality of feature extraction operations, performing the feature extraction operation using a convolutional layer, generating an output feature map of the convolutional layer based on the input feature map and the plurality of weighted coordinate maps received by the convolutional layer; generating a plurality of weighted coordinate maps based on the plurality of coordinate maps and the spatial attention map; creating a spatial attention map based on the input feature map received by the convolutional layer, and modifying coordinate information in each of the plurality of coordinate maps to create a plurality of weighted coordinate maps.

[0007]

[0007] According to a second aspect of the present invention, there is provided a CNN-based image processing system comprising a memory and at least one processor communicatively coupled to the memory and configured to perform the CNN-based image processing method according to the above-mentioned first aspect of the present invention.

[0008]

[0008] According to a third aspect of the present invention, there is provided a computer program product embodied in one or more non-transitory computer-readable storage media and including instructions executable by at least one processor to perform the CNN-based image processing method according to the above-mentioned first aspect of the present invention.

[0009] According to a fourth aspect of the present invention, there is provided a method of segmenting tissue structures in an ultrasound image using CNN, using at least one processor, the method comprising: performing a CNN-based image processing method according to the above-mentioned first aspect of the present invention, the input image is an ultrasound image including tissue structure; The output image has segmented tissue structures and is the result of inference on the input image using CNN.

[0010]

[0010] According to a fifth aspect of the present invention, there is provided an image processing system based on CNN, the system comprising a memory and at least one processor communicatively coupled to the memory and configured to execute a method for segmenting tissue structures in an ultrasound image using a CNN according to the above-mentioned fourth aspect of the present invention.

[0011]

[0011] According to a sixth aspect of the present invention, there is provided a computer program product embodied in one or more non-transitory computer-readable storage media and including instructions executable by at least one processor to perform a method for segmenting tissue structures in ultrasound images using CNN according to the above-mentioned fourth aspect of the present invention.

[0012]

[0012] Embodiments of the present invention will be better understood and readily apparent to those skilled in the art from the following description, given by way of example only, in conjunction with the drawings in which: [Brief description of the drawings]

[0013] [Figure 1] 1 is a schematic flow diagram of a CNN-based image processing method according to various embodiments of the present invention; [Diagram 2] FIG. 1 is a schematic block diagram of a CNN-based image processing system according to various embodiments of the present invention. [Diagram 3] FIG. 1 is a schematic block diagram of an exemplary computer system that can be used to realize or implement a CNN-based image processing system in accordance with various embodiments of the present invention. [Figure 4A] FIG. 1 illustrates an example network architecture of an example CNN in accordance with various example embodiments of the present invention. [Figure 4B] FIG. 1 illustrates an example network architecture of an example CNN in accordance with various example embodiments of the present invention. [Diagram 5] 1 is a table (Table 1) illustrating an exemplary detailed configuration of a prediction module and a refinement module of an exemplary CNN according to various exemplary embodiments of the present invention. [Figure 6] FIG. 2 is a schematic block diagram of a residual U block (RSU) according to various exemplary embodiments of the present invention; [Figure 7A] FIG. 7A is a schematic block diagram of a residual block according to various exemplary embodiments. [Figure 7B] FIG. 7B is a schematic block diagram of an RSU according to various exemplary embodiments. [Figure 8A] FIG. 8A is a schematic block diagram of an original coordinate convolution (CoordConv) according to various exemplary embodiments of the present invention. [Figure 8B] FIG. 8B is a schematic block diagram of Attention Coordinate Convolution (AC-Conv) according to various exemplary embodiments of the present invention. [Figure 9A] FIG. 9A is a schematic block diagram of a conventional series smelting module and a parallel smelting module in accordance with various exemplary embodiments of the present invention. [Figure 9B] FIG. 9B is a schematic block diagram of a conventional series smelting module and a parallel smelting module in accordance with various exemplary embodiments of the present invention. [Figure 10] FIG. 1 is a schematic diagram illustrating a thyroid gland and ultrasound scanning protocol according to various exemplary embodiments of the present invention, along with a corresponding ultrasound image with a manually labeled thyroid lobe overlay. [Figure 11] 2 is a table (Table 2) showing volume numbers and corresponding slices (images) in each subset of ultrasound images according to various exemplary embodiments of the present invention. [Figure 12] 3 is a table (Table 3) showing a quantitative evaluation or comparison of an exemplary CNN according to various exemplary embodiments of the present invention with other state-of-the-art segmentation models on transverse (TRX) and sagittal (SAG) plane test sets. [Figure 13A] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13B] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13C] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13D] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13E] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13F] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13G] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13H] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13I] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13J] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13K] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 13L] FIG. 1 illustrates sample segmentation results on TRX thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14A] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14B] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14C] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14D] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14E] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14F] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14G] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14H] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14I] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14J] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14K] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 14L] FIG. 1 illustrates sample segmentation results in SAG thyroid images using an exemplary CNN in accordance with various exemplary embodiments of the present invention. [Figure 15A] 11 is a success rate curve graph of exemplary CNN and other state of the art models according to various exemplary embodiments of the present invention in TRX images. [Figure 15B] 11 is a success rate curve graph of exemplary CNN and other state of the art models according to various exemplary embodiments of the present invention in SAG images. [Figure 16] Table 4 shows ablation studies performed with different convolutional blocks and refinement architectures. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014]

[0013] Various embodiments of the present invention provide a method and system for image processing based on convolutional neural networks (CNN), more specifically deep CNN. CNN is a class or type of artificial neural network, sometimes referred to as a CNN model, or simply a model. For example, as described in the background, CNN may be generally understood to be applicable in various areas of various predictive applications, but using CNN in various predictive applications does not always provide satisfactory predictive results (e.g., not always accurate enough in image segmentation or image classification), and satisfactory predictive results may be difficult or challenging to obtain. As an example, ultrasound images containing tissue structures (e.g., anatomical structures, or other types of tissue structures such as tumors) are found to be noisy, and conventional methods for segmenting such ultrasound images based on CNNs have been found to produce poor results. Accordingly, various embodiments of the present invention provide a method and system for CNN-based image processing that aims to overcome or at least improve one or more problems associated with conventional methods and systems for CNN-based image processing, particularly but not limited to, enhancing or improving the predictive capabilities (e.g., accuracy of prediction results) associated with CNN-based image processing, such as image segmentation.

[0015]

[0014] Figure 1 shows a schematic flow diagram of a CNN-based image processing method 100 using at least one processor according to various embodiments of the present invention. The method 100 includes receiving an input image (102), performing feature extraction operations using a plurality of convolution layers of the CNN based on the input image and generating a plurality of output feature maps (104), and generating an output image for the input image based on the plurality of output feature maps of the plurality of convolution layers (106). In particular, for each of the plurality of feature extraction operations, performing the feature extraction operation using the convolution layer includes generating an output feature map of the convolution layer based on the input feature map and the plurality of weighted coordinate maps received by the convolution layer, generating a plurality of weighted coordinate maps based on the plurality of coordinate maps and the spatial attention map, and generating a spatial attention map based on the input feature map received by the convolution layer and modifying the coordinate information in each of the plurality of coordinate maps to generate a plurality of weighted coordinate maps.

[0016]

[0015] Accordingly, it has been found that the image processing method 100 advantageously enhances or improves prediction capabilities, particularly in relation to image segmentation, and more particularly in relation to ultrasound image segmentation. In particular, by performing the feature extraction operation using a corresponding convolutional layer as described above, not only does the associated convolutional operation have access to coordinate information (coordinate map (by using an extra coordinate channel)), but also allows the associated convolutional operation to pay more attention (i.e., additional attention) to specific coordinates that may be beneficial for the feature extraction operation (by using a spatial attention map, which may also be simply referred to as an attention map), whereby such additional attention (i.e., additional attention) is guided by the input feature map received by the convolutional layer, via a spatial attention map derived from the input feature map. Accordingly, not only does the associated convolutional operation know the spatial location (e.g., in Cartesian space), but the associated convolutional operation also knows the location to pay more attention to via the spatial attention map. For example, the spatial attention map can add extra weight to certain coordinates that may require more attention or attention and reduce weight to certain coordinates that may require less attention or attention, depending on the guidance of the input feature map (e.g., more important parts of the input feature map can thus receive more attention in the feature extraction operation), thereby causing the associated convolution operation of the convolution layer to advantageously have attention coordinate guidance. Accordingly, such feature extraction operation using such a convolution layer with attention coordinate guidance can be referred to as attention coordinate guided convolution (AC-Conv), and such a convolution layer with attention coordinate guidance can be referred to as an AC-Conv layer. In this regard, it has been found that with attention coordinate guidance, the image processing method 100 advantageously enhances or improves prediction capabilities. As the image processing method 100 and the corresponding image processing system are described in more detail by various embodiments and exemplary embodiments of the present invention, these advantages or technical effects, and / or other advantages or technical effects will become more apparent to those skilled in the art.

[0017]

[0016] In various embodiments, the above-mentioned step of creating a spatial attention map includes a step of performing a first convolution operation based on an input feature map received by the convolutional layer to create a convolutional feature map, and a step of applying an activation function based on the convolutional feature map to create the spatial attention map.

[0018] In various embodiments, the activation function is a sigmoid activation function.

[0018] In various embodiments, the above-mentioned step of creating a plurality of weighted coordinate maps includes multiplying each of the plurality of coordinate maps with the spatial attention map to modify the coordinate information in each of the plurality of coordinate maps.

[0019]

[0019] In various embodiments, the multiple coordinate maps include a first coordinate map including coordinate information for a first dimension and a second coordinate map including coordinate information for a second dimension, the first and second dimensions being two dimensions along which the first convolution operation is configured to be performed.

[0020]

[0020] In various embodiments, the above-mentioned step of creating an output feature map of the convolutional layer includes a step of concatenating the input feature map and multiple weighted coordinate maps received by the convolutional layer for each channel to form a concatenated feature map, and a step of performing a second convolution operation based on the concatenated feature map to create the output feature map of the convolutional layer.

[0021]

[0021] In various embodiments, the CNN includes a predictive sub-network including at least one convolutional layer of the plurality of convolutional layers of the CNN. In this regard, the method 100 further includes a step of generating a set of predictive feature maps using the predictive sub-network based on the input image, and the above-mentioned step of generating the set of predictive feature maps includes a step of performing at least one feature extraction operation of the plurality of feature extraction operations using at least one convolutional layer of the predictive sub-network. Furthermore, the plurality of predictive feature maps of the set of predictive feature maps have different spatial resolution levels.

[0022]

[0022] In various embodiments, the prediction sub-network has an encoder-decoder structure including a set of encoder blocks and a set of decoder blocks. The set of encoder blocks of the prediction sub-network includes a plurality of encoder blocks, and the set of decoder blocks of the prediction sub-network includes a plurality of decoder blocks. In this regard, the method 100 further includes, for each of the plurality of encoder blocks of the prediction sub-network, creating a downsample feature map using the encoder block based on the input feature map received by the encoder block, and for each of the plurality of decoder blocks of the prediction sub-network, creating an upsample feature map using the decoder block based on the input feature map and the downsample feature map created by the encoder block corresponding to the decoder block and received by the decoder block.

[0023]

[0023] In various embodiments, the above-mentioned step of creating a set of predicted feature maps using a prediction sub-network includes creating multiple predicted feature maps based on multiple upsampled feature maps produced by multiple decoder blocks, respectively.

[0024]

[0024] In various embodiments, the above-mentioned step of creating a downsampled feature map using an encoder block of a prediction sub-network includes extracting multi-scale features based on an input feature map received by the encoder block, and creating a downsampled feature map based on the extracted multi-scale features extracted by the encoder block. In various embodiments, the above-mentioned step of creating an upsampled feature map using a decoder block of a prediction sub-network includes extracting multi-scale features based on an input feature map and a downsampled feature map created by an encoder block corresponding to the decoder block and received by the decoder block, and creating an upsampled feature map based on the extracted multi-scale features extracted by the decoder block.

[0025]

[0025] In various embodiments, each of the plurality of encoder blocks of the prediction sub-network includes at least one convolutional layer of the plurality of convolutional layers of the CNN, and the above-mentioned step of creating a downsampled feature map using the encoder block of the prediction sub-network includes performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the encoder block. In various embodiments, each of the plurality of decoder blocks of the prediction sub-network includes at least one convolutional layer of the plurality of convolutional layers of the CNN, and the above-mentioned step of creating an upsampled feature map using the decoder block of the prediction sub-network includes performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the decoder block.

[0026] In various embodiments, each convolutional layer of each of the plurality of encoder blocks of the prediction sub-network is one of the plurality of convolutional layers of the CNN. In various embodiments, each convolutional layer of each of the plurality of decoder blocks of the prediction sub-network is one of the plurality of convolutional layers of the CNN.

[0027] In various embodiments, each of the plurality of encoder blocks of the prediction sub-network is configured as a residual block.In various embodiments, each of the plurality of decoder blocks of the prediction sub-network is configured as a residual block.

[0028]

[0028] In various embodiments, the CNN further includes a refinement sub-network including at least one convolution layer of the plurality of convolution layers of the CNN. In this regard, the method 100 further includes a step of creating a set of refinement feature maps using the refinement sub-network based on the fusion feature map, and the above-mentioned step of creating the set of refinement feature maps includes a step of performing at least one feature extraction operation of the plurality of feature extraction operations using at least one convolution layer of the refinement sub-network. Furthermore, the plurality of refinement feature maps of the set of refinement feature maps have different spatial resolution levels.

[0029] In various embodiments, the method 100 further includes concatenating the set of predicted feature maps to create a fused feature map.

[0030] In various embodiments, the refinement sub-network includes a plurality of refinement blocks each configured to create a plurality of refinement feature maps, and each of the plurality of refinement blocks has an encoder-decoder structure including a set of encoder blocks and a set of decoder blocks. The set of encoder blocks of the refinement sub-network includes a plurality of encoder blocks, and the set of decoder blocks of the refinement sub-network includes a plurality of decoder blocks. In this regard, the method 100 further includes, for each of the plurality of refinement blocks, creating a downsample feature map using the encoder block based on an input feature map received by the encoder block for each of the plurality of encoder blocks of the refinement block, and creating an upsample feature map using the decoder block based on the input feature map and the downsample feature map created by the encoder block corresponding to the decoder block and received by the decoder block.

[0030]

[0031] In various embodiments, the encoder-decoder structures of the refining blocks have different heights.

[0032] In various embodiments, the above-mentioned step of creating a set of refined feature maps using a refinement sub-network includes, for each of a plurality of refinement blocks, creating a refined feature map for the refinement block based on a fused feature map received by the refinement block and an upsampled feature map created by a first decoder block of a plurality of decoder blocks of the refinement block.

[0031]

[0033] In various embodiments, the above-mentioned step of creating a downsampled feature map using the encoder block of the refinement block includes extracting multi-scale features based on an input feature map received by the encoder block, and creating a downsampled feature map based on the extracted multi-scale features extracted by the encoder block. In various embodiments, the above-mentioned step of creating an upsampled feature map using the decoder block of the refinement block includes extracting multi-scale features based on the input feature map and a downsampled feature map created by the encoder block of the refinement block corresponding to the decoder block and received by the decoder block, and creating an upsampled feature map based on the extracted multi-scale features extracted by the decoder block.

[0032]

[0034] In various embodiments, for each of the plurality of refinement blocks, each of the plurality of encoder blocks of the refinement block includes at least one convolutional layer of the plurality of convolutional layers of the CNN, and the above-mentioned step of creating a downsampled feature map using the encoder block of the refinement block includes performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the encoder block. In various embodiments, for each of the plurality of refinement blocks, each of the plurality of decoder blocks of the refinement block includes at least one convolutional layer of the plurality of convolutional layers of the CNN, and the above-mentioned step of creating an upsampled feature map using the decoder block of the refinement block includes performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the decoder block.

[0033]

[0035] In various embodiments, each convolutional layer of each of the plurality of encoder blocks of the refinement block is one of the plurality of convolutional layers of the CNN. In various embodiments, each convolutional layer of each of the plurality of decoder blocks of the refinement block is one of the plurality of convolutional layers of the CNN.

[0034]

[0036] In various embodiments, for each of a plurality of refinement blocks, each of a plurality of encoder blocks of the refinement block is configured as a residual block, and each of a plurality of decoder blocks of the refinement block is configured as a residual block.

[0035]

[0037] In various embodiments, the output image is created based on a set of refined feature maps.

[0038] In various embodiments, the output image is created based on an average of a set of refined feature maps.

[0036]

[0039] In various embodiments, the above-mentioned step of receiving an input image (102) includes receiving a plurality of input images, each of the plurality of input images being a labeled image for training the CNN to obtain a trained CNN, in this regard, performing, for each of the plurality of input images, a plurality of feature extraction operations using respective plurality of convolutional layers of the CNN based on the input image to generate a respective plurality of output feature maps, and generating an output image for the input image based on the respective plurality of output feature maps of the respective plurality of convolutional layers.

[0037]

[0040] In various embodiments, the labeled image is a labeled ultrasound image that includes tissue structures.

[0041] In various embodiments, the output image is the result of inference on the input image using a CNN.

[0038]

[0042] In various embodiments, the input image is an ultrasound image that includes tissue structures.

[0043] 2 shows a schematic block diagram of a CNN-based image processing system 200 according to various embodiments of the present invention, corresponding to the image processing method 100 described above with reference to FIG. 1 according to various embodiments of the present invention. The system 200 comprises a memory 202 and at least one processor 204 communicatively coupled to the memory 202 and configured to perform the image processing method 100 described herein according to various embodiments of the present invention. Accordingly, in various embodiments, the at least one processor 204 is configured to receive an input image, perform a plurality of feature extraction operations using a plurality of convolutional layers of the CNN based on the input image, create a plurality of output feature maps, respectively, and create an output image for the input image based on the plurality of output feature maps of the plurality of convolutional layers. In particular, as described above, for each of the plurality of feature extraction operations, the step of performing the feature extraction operation using a convolutional layer includes the steps of: creating an output feature map of the convolutional layer based on an input feature map received by the convolutional layer and the plurality of weighted coordinate maps; creating a plurality of weighted coordinate maps based on the plurality of coordinate maps and a spatial attention map; and creating a spatial attention map based on the input feature map received by the convolutional layer and modifying the coordinate information in each of the plurality of coordinate maps to create the plurality of weighted coordinate maps.

[0039]

[0044] It will be understood by those skilled in the art that the at least one processor 204 can be configured to perform various functions or operations by a set of instructions (e.g., software modules) executable by the at least one processor 204 to perform various functions or operations. Accordingly, as shown in FIG. 2, the system 200 can include an input image receiving module (or input image receiving circuit) 206 configured to receive an input image, a feature extraction module (or feature extraction circuit) 208 configured to perform a plurality of feature extraction operations using a plurality of convolution layers of a CNN based on the input image and create a plurality of output feature maps, respectively, and an output image creation module (or output image creation circuit) 210 configured to create an output image for the input image based on the plurality of output feature maps of the plurality of convolution layers.

[0040]

[0045] Those skilled in the art will appreciate that the above-mentioned modules are not necessarily separate modules, and that one or more of the modules may be realized by or implemented as a single functional module (e.g., a circuit or software program) as desired or appropriate without departing from the scope of the present invention. For example, two or more of the input image receiving module 206, the feature extraction module 208, and the output image creating module 210 may be realized (e.g., compiled together) as a single executable software program (e.g., referred to as a software application, or simply an "app") that may be stored, for example, in memory 202 and executable by at least one processor 204 to perform various functions / operations described herein according to various embodiments of the present invention.

[0041]

[0046] In various embodiments, the image processing system 200 corresponds to the image processing method 100 described above with reference to Figure 1 according to various embodiments, and thus various functions or operations configured to be performed by the at least one processor 204 may correspond to various steps or operations of the image processing method 100 described above according to various embodiments, and thus need not be repeated with respect to the image processing system 200 for the sake of clarity and brevity. In other words, various embodiments described herein in the context of a method are analogously valid for the corresponding system, and vice versa.

[0042]

[0047] For example, in various embodiments, the memory 202 may store an input image receiving module 206, a feature extraction module 208, and / or an output image creation module 210, which correspond respectively to various steps (or operations or functions) of the image processing method 100 described herein according to various embodiments and are executable by at least one processor 204 to perform the corresponding functions or operations described herein.

[0043]

[0048] In various embodiments, a method is provided for segmenting tissue structures in an ultrasound image using CNN using at least one processor according to various embodiments of the present invention, the method includes performing the CNN-based image processing method 100 described above according to various embodiments, whereby an input image is an ultrasound image including tissue structures and an output image has segmented tissue structures and is a result of inference on the input image using CNN.

[0044]

[0049] In various embodiments, the CNN is trained as previously described by various embodiments, i.e., the CNN is a pre-trained CNN as described above.

[0050] In various embodiments, a system for segmenting tissue structures in an ultrasound image using a CNN according to various embodiments is provided, which corresponds to the above-described method for segmenting tissue structures in an ultrasound image according to various embodiments. The system comprises a memory and at least one processor communicatively coupled to the memory and configured to execute the above-described method for segmenting tissue structures in an ultrasound image. In various embodiments, the system for segmenting tissue structures in an ultrasound image can be the same as the image processing system 200, whereby the input image is an ultrasound image including tissue structures and the output image has segmented tissue structures and is a result of inference on the input image using the CNN.

[0045]

[0051] According to various embodiments of the present disclosure, a computing system, controller, microcontroller, or any other system that provides processing capabilities can be provided. Such a system can be considered to include one or more processors and one or more computer-readable storage media. For example, the image processing system 200 described above can include a processor (or controller) 204 and a computer-readable storage medium (or memory) 202 used in various processes, for example, as described herein. The memory or computer-readable storage medium used in various embodiments can be a volatile memory, such as DRAM (Dynamic Random Access Memory), or a non-volatile memory, such as PROM (Programmable Read Only Memory), EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, such as a floating gate memory, a charge trap memory, MRAM (Magnetoresistive Random Access Memory), or PCRAM (Phase Change Random Access Memory).

[0046]

[0052] In various embodiments, a "circuit" can be understood to be any kind of logic implementation entity, and can be a special purpose circuit or a processor running software stored in memory, firmware, or any combination thereof. Thus, in one embodiment, a "circuit" can be a hardwired logic circuit or a programmable logic circuit, such as a programmable processor, for example, a microprocessor (e.g., a complex instruction set computer (CISC) processor or a reduced instruction set computer (RISC) processor). A "circuit" can also be a processor running software, for example, any kind of computer program, for example, a computer program using virtual machine code, for example, Java. According to various embodiments, any other kind of implementation of the respective functions can also be understood to be a "circuit". Similarly, a "module" can be a part of a system according to various embodiments, can include the "circuit" described above, or can be understood to be any kind of logic implementation entity.

[0047]

[0053] Some portions of this disclosure are explicitly or implicitly presented in terms of algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here generally conceived to be a self-consistent sequence of steps leading to a desired result. These steps involve physical manipulations of physical quantities, such as electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

[0048]

[0054] Unless specifically stated otherwise, as will be apparent hereinafter, throughout this specification, descriptions or discussions utilizing terms such as "receive," "execute," "create," "multiply," "concatenate," "extract," and the like will be understood to refer to operations and processes of a computer system or similar electronic device that manipulates and transforms data represented as physical quantities within a computer system into other data similarly represented as physical quantities within a computer system or other information storage, transmission, or display device.

[0049]

[0055] This specification also discloses systems (e.g., which may be embodied as devices or apparatuses) such as image processing system 200 for performing various operations / functions of various methods described herein. Such systems may be specially constructed for the required purposes, or may comprise a general-purpose computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose machines may be used with computer programs in accordance with the teachings of this specification. Alternatively, more specialized apparatus constructions for performing various method steps may also be appropriate.

[0050]

[0056] In addition, this specification also at least implicitly discloses computer programs or software / functional modules, since it should be clear to one skilled in the art that the individual steps of the various methods described herein can be implemented by computer code. The computer program is not intended to be limited to any particular programming language and its implementation. It will be understood that various programming languages ​​and their coding can be used to implement the teachings of the present disclosure contained herein. Furthermore, the computer program is not intended to be limited to any particular control flow. There are many other variations of computer programs that can use different control flows without departing from the scope of the present invention. It will be understood by one skilled in the art that the various modules described herein (e.g., input image receiving module 206, feature extraction module 208, and / or output image creating module 210) can be software modules realized by a computer program or instruction set executable by a computer processor to perform the required functions, or can be hardware modules that are functional hardware units designed to perform the required functions. It will also be understood that a combination of hardware and software modules can be implemented.

[0051]

[0057] Moreover, one or more of the steps of the computer programs / modules or methods described herein may be executed in parallel rather than sequentially. Such computer programs may be stored on any computer-readable medium. The computer-readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interacting with a general-purpose computer. The computer programs, when loaded and executed on such a general-purpose computer, substantially produce an apparatus that implements the steps of the methods described herein.

[0052]

[0058] In various embodiments, a computer program product is provided that includes instructions embodied in one or more computer-readable storage media (non-transitory computer-readable storage media) and executable by one or more computer processors to perform the image processing method 100 described herein with reference to Figure 1 according to various embodiments (e.g., input image receiving module 206, feature extraction module 208, and / or output image creation module 210). Accordingly, various computer programs or modules described herein may be stored in a computer program product that is receivable by a system, such as the image processing system 200 shown in Figure 2, for execution by at least one processor 204 of the system 200 to perform various functions.

[0053]

[0059] In various embodiments, a computer program product is provided that is embodied in one or more computer-readable storage media (non-transitory computer-readable storage media) and includes instructions executable by one or more computer processors to perform the above-described methods for segmenting tissue structures in ultrasound images according to various embodiments. Accordingly, various computer programs or modules described herein can be stored in a computer program product that can be received by a system, such as the above-described system for segmenting tissue structures in ultrasound images, for execution by at least one processor of the system to perform various functions.

[0054]

[0060] The software or functional modules described herein can also be implemented as hardware modules. More specifically, in the hardware sense, a module is a functional hardware unit designed for use with other components or modules. For example, a module can be implemented using individual electronic components or can form part of an overall electronic circuit, such as an application specific integrated circuit (ASIC). Many other possibilities exist. Those skilled in the art will understand that the software or functional modules described herein can also be implemented as a combination of hardware and software modules.

[0055]

[0061] In various embodiments, the image processing system 200 can be realized by any computer system (e.g., a desktop or portable computer system) including at least one processor and memory, such as the computer system 300 shown diagrammatically in FIG. 3 by way of example and not limitation. The various methods / steps or functional modules can be implemented as software, such as computer programs, running within the computer system 300 and instructing the computer system 300 (particularly the processor or processors therein) to perform various functions or operations described herein according to various embodiments. The computer system 300 can comprise a computer module 302, input modules such as a keyboard and / or touch screen 304 and a mouse 306, and multiple output devices such as a display 308 and a printer 310. The computer module 302 can be connected to a computer network 312 via a suitable transceiver device 314 to enable access to other network systems, such as the Internet or a local area network (LAN) or / and / or (WAN). The computer module 302 of this example may include a processor 318 for executing various instructions, a random access memory (RAM) 320, and a read only memory (ROM) 322. The computer module 302 may also include a number of input / output (I / O) interfaces, for example an I / O interface 324 to the display 308, and an I / O interface 326 to the keyboard 304. The components of the computer module 302 typically communicate via an interconnected bus 328, in a manner known to those skilled in the art.

[0056]

[0062] Those skilled in the art will appreciate that the terminology used herein is for the purpose of describing various embodiments only and is not intended to limit the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0057]

[0063] Unless otherwise stated or required by context, references herein to elements or features using designations such as "first," "second," etc., do not limit the quantity or order of such elements or features. For example, such designations may be used herein as a convenient method of distinguishing between two or more elements or instances of an element. Thus, references to a first and a second element do not necessarily imply that only two elements can be used or that the first element must precede the second element. In addition, phrases referring to "at least one" of a list of items refer to any single item therein, or any combination of two or more items therein.

[0058]

[0064] In order that the present invention may be readily understood and practically practiced, various exemplary embodiments of the present invention are described below by way of example only and not by way of limitation. However, those skilled in the art will appreciate that the present invention may be embodied in a variety of different forms or configurations and should not be construed as being limited to the exemplary embodiments set forth below. On the contrary, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0059]

[0065] In particular, and without limitation or loss of generality, in order to better understand the present invention, various exemplary embodiments of the present invention will now be described with respect to the input image being an ultrasound image and the image processing being for ultrasound image segmentation, i.e., a CNN-based image processing method for segmenting tissue structures within an ultrasound image. While such a particular application (i.e., ultrasound image segmentation) may be preferred for various exemplary embodiments, those skilled in the art will appreciate that the present invention is not limited to such a particular application, and that the image processing method can be implemented, as desired or appropriate, in other types of applications, such as, but not limited to, image classification (e.g., applications in which the input image is relatively noisy and / or the structures of interest generally have similar positions and / or shapes within the input image).

[0060]

[0066] Ultrasound image segmentation is a challenging task due to the presence of artifacts inherent to diagnostic modalities such as attenuation, shadowing, speckle noise, non-uniform textures, and blurred boundaries. In this regard, various exemplary embodiments provide a predictive refined attention network (CNN) for segmentation of soft tissue structures in ultrasound images, which is herein referred to as ACU. 2 The prediction refinement attention network may be referred to as E-Net, or simply as this CNN or model. The prediction refinement attention network corresponds to a prediction module or block (e.g., the prediction sub-network previously described in accordance with various embodiments, and is referred to herein as ACU) that includes an attention coordinate convolution (AC-Conv). 2The present invention comprises a multi-head residual refinement module or block (e.g., corresponding to the refinement sub-networks described above by various embodiments, which may be referred to as MH-RRM or E-Module herein) including multiple (e.g., three) parallel residual refinement modules or blocks (e.g., corresponding to the multiple refinement blocks described above by various embodiments). In various exemplary embodiments, the AC-Conv is configured or designed to improve segmentation accuracy by perceiving shape and position information of the target anatomical structure. By integrating residual refinement and ensemble strategies, the MH-RRM has been found to advantageously reduce both segmentation bias and variance, and avoid the multi-pass training and inference commonly found in ensemble methods. To demonstrate the effectiveness of the present CNN-based image processing method for segmenting tissue structures in ultrasound images according to various exemplary embodiments, a dataset of thyroid ultrasound scans was collected and the present CNN was evaluated against state-of-the-art segmentation methods. Comparison with state-of-the-art models demonstrates competitive or improved performance of the present CNN in both transverse and sagittal thyroid images. For example, ablation studies show that the AC-Conv and MH-RRM modules improve the segmentation Dice score of the baseline model from 79.62% to 80.97% and 83.92%, while reducing the variance from 6.12% to 4.67% and 3.21%.

[0061]

[0067] As described in the background, ultrasound images may be obtained from a handheld probe and are therefore operator dependent and subject to numerous artifacts such as large speckle noise, shadows, and blurred boundaries. This makes it more difficult to segment tissue structures of interest (e.g., anatomical structures) from adjacent tissues. Several conventional methods (e.g., active contours, graph cuts, superpixels, and deep models (e.g., fully convolutional networks (FCN), U-Net, etc.) have been proposed and adapted for ultrasound image segmentation. However, due to the noisiness of ultrasound images, such conventional methods usually produce poor results. Although deep models have achieved significant improvements over conventional methods, accurately segmenting soft tissue structures from ultrasound images remains a challenging task.

[0062]

[0068] In relation to ultrasound image segmentation, in various exemplary embodiments, it should be noted that unlike the general objects of various shapes and positions in natural image segmentation, tissue structures (e.g., anatomical structures) in ultrasound images have similar positions and shape patterns. However, these geometric features are difficult to represent and encode, and are therefore rarely used in segmentation deep models. Accordingly, traditionally, how to use the unique geometric constraints of soft tissue structures in ultrasound images remains a challenge. Another problem with segmenting ultrasound images using a single deep model is that it generally results in highly biased results due to blurred boundaries and textures, and has high variance due to noise and inhomogeneity.

[0063]

[0069] Accordingly, to overcome these challenges, various exemplary embodiments provide the above-mentioned attention-based predictive refinement architecture (i.e., the present CNN), which comprises a prediction module built on the above-mentioned AC-Conv and a multi-head residual refinement module (MH-RRM). Such attention-based predictive refinement architecture advantageously leverages anatomical location and shape constraints presented in the ultrasound image to reduce bias and variance in the segmentation results and avoid multi-pass training and inference. Accordingly, the present CNN contributions include (a) an AC-Conv configured to improve segmentation accuracy by perceiving both geometric information (e.g., shape and location information) from the ultrasound image, and / or (b) a predictive refinement architecture with MH-RRM that improves segmentation accuracy by integrating both ensemble and predictive refinement strategies together. As described below, the present CNN-based image processing method for ultrasound image segmentation according to various exemplary embodiments has been tested on a dataset of thyroid ultrasound scans and achieved improved performance (e.g., accuracy) compared to conventional models.

[0064] CNN Architecture

[0070] 4A and 4B together illustrate an example network architecture of an example CNN 400 according to various example embodiments of the present invention. As also previously mentioned, the example CNN 400 includes a prediction module or block (ACU) 2 In various exemplary embodiments, the prediction module 410 comprises a MH-RRM (MH-Net) 410 (FIG. 4A) and a MH-RRM 450 (FIG. 4B). 2 U-Net: Going Deeper with nested U-structure for salient object detection,” Pattern Recognition, 106:107404, 2020 (herein referred to as reference Qin, the contents of which are incorporated by reference in their entirety for all purposes). 2 -Net based U 2By replacing each simple convolutional layer in the U-Net with an AC-Conv layer as described herein according to various exemplary embodiments, attention coordinate guidance U 2 -Net(ACU 2 In various exemplary embodiments, the refinement module 450 may be configured to form a prediction module (ACU) that is a 2 -Net) arranged in parallel (e.g., to create refined feature maps with different spatial resolution levels). As an example, as shown in FIG. 4B, the refining module 450 may include three refining heads or blocks arranged in parallel (three ACUs for creating refined feature maps with different spatial resolution levels). 2 -Net variants) 454-1, 454-2, and 454-3, each of which is an ACU 2 -Net-Ref7, ACU 2 -Net-Ref5, and ACU 2 4B as -Net-Ref3 (e.g., looks like the letter "E," and thus such an exemplary configuration of the refinement module 450 may be referred to herein as an E-Module). In the legend shown in FIG. 4A, the term AC-CBR refers to AC-Conv+BacthNorm+ReLU.

[0065]

[0071] For illustrative purposes and by way of example only, and not limitation, FIG. 5 illustrates a table (Table 1) showing an example detailed configuration of the prediction module 410 and the refinement module 450 of an example CNN 400 according to various exemplary embodiments. Blank cells in Table 1 indicate the absence of such a stage. In addition, “I”, “M”, and “O” denote the input channels (C in ), the intermediate channels, and the output channel (C out5. "En_i" and "De_j" denote the number of AC-RSU stages. "En_i" and "De_j" denote the encoder and decoder stages, respectively. The number "L" in "AC-RSU-L" denotes the height of the AC-RSU block. Those skilled in the art will appreciate that the present invention is not limited to a CNN having the exemplary detailed configurations (or parameters) shown in FIG. 5, which are provided by way of example only for illustrative purposes and not for limitation. The parameters of the CNN may be, if desired or appropriate, configured as, but not limited to, an ACU. 2 Those skilled in the art will appreciate that these may be changed or modified for various purposes, such as the desired height of the encoder-decoder structure of the -Net, the desired different spatial resolution levels (and / or the desired number of different spatial resolution levels) of the predicted feature maps to be created, the desired different spatial resolution levels (and / or the desired number of different spatial resolution levels) of the refined feature maps to be created, the desired number of layers in the encoder or decoder block, the desired number of channels in the encoder or decoder block, etc.

[0066]

[0072] References Qin, A Deep Network Architecture for Salient Object Detection (U 2 The company has disclosed a new system for managing the U-Net. 2 The network architecture of N-Net is a two-level nested U-structure. The network architecture has the advantages of (1) being able to capture more contextual information from different scales due to the mixing of receptive fields of different sizes in the residual U-blocks (RSU blocks, sometimes simply called RSUs), and (2) being able to increase the overall depth of the architecture without significantly increasing the computational cost due to the pooling operations used within these RSU blocks. Such a network architecture makes it possible to train deep networks from scratch without using a backbone from image classification tasks. In particular, the U-structure allows for the training of deep networks from scratch without using a backbone from image classification tasks. 2 U-Net is a two-level nested U-structure designed for SOD without using a pre-trained backbone from image classification. 2The U-Net can be trained from scratch to achieve competitive performance. Moreover, this network architecture allows the network to go deeper and achieve higher resolution without significantly increasing memory and computational costs. This is achieved by a nested U-structure, whereby at the lowest level, RSU blocks are configured that can extract intra-stage multi-scale features without degrading the feature map resolution, and at the top level, there is a U-Net-like structure (encoder-decoder structure), where each stage is filled with RSU blocks. The two-level structure results in a nested U-structure, and an example of a nested U-structure (encoder-decoder structure) according to various exemplary embodiments shown in FIG. 4A, whereby, as previously described, the U-Net-like structure (encoder-decoder structure) can be obtained. 2 Each simple convolutional layer in the -Net is replaced by an AC-Conv layer as described herein according to various exemplary embodiments to obtain the ACU 2 - Form Net410.

[0067]

[0073] In summary, multi-level deep feature integration methods mainly focus on developing better multi-level feature aggregation strategies. On the other hand, methods in the field of multi-scale feature extraction target the design of new modules to extract both local and global information from the features obtained by the backbone network. In this regard, U 2 -Net or ACU 2 The network architecture of -Net410 is configured to directly extract multi-scale features at each stage.

[0068] Residual U Block (RSU) / Attention Coordinate Guidance Residual U Block (AC-RSU)

[0074] Both local and global context information are important for salient object detection and other segmentation tasks. In modern CNN designs such as VGG, ResNet, and DenseNet, small convolution filters with sizes of 1×1 or 3×3 are the most frequently used building blocks for feature extraction. These convolution filters are preferred because they require small storage space and are computationally efficient. For example, the output feature maps of shallow layers only accommodate local features because the receptive fields of 1×1 or 3×3 filters are too small to capture global information. To achieve more global information in high-resolution feature maps from shallow layers, the most straightforward concept is to enlarge the receptive field. However, performing multiple dilation convolutions (especially at early stages) on input feature maps at the original resolution requires too many computational and memory resources. To reduce the computational cost, a parallel configuration can be adapted from the pyramid pooling module (PPM), which uses small kernel filters on downsampled feature maps rather than dilation convolutions on feature maps of the original size. However, direct upsampling and fusing features at different scales by concatenation (or addition) can lead to degradation of high-resolution features.

[0069]

[0075] Accordingly, an RSU block is provided to capture intra-stage multi-scale features, as described in reference Qin. By way of example only and not by way of limitation, RSU-L(C in , M.C. out An exemplary structure of a ) block 600 is shown in FIG. 6, where L is the number of layers in the encoder and C in , C out denotes input and output channels, and M denotes the number of channels in an inner layer of the RSU block 600. Those skilled in the art will appreciate that the RSU-L block 600 is not limited to the specific dimensions (e.g., the number of layers L) shown in Figure 6, which is merely an example and not a limitation. Accordingly, the RSU block 600 comprises three components:

[0070] (i) Input feature map x(H×W×C in ) to C out An input convolutional layer that converts the input vector into an intermediate map F1(x) with channels. This is a simple convolutional layer for local feature extraction.

[0071] (ii) A U-Net-like symmetric encoder-decoder structure with height L that takes the intermediate feature map F1(x) as input and learns to extract and encode multi-scale context information U(F1(x)). U represents the U-Net-like structure shown in Figure 6. A larger L leads to deeper residual U blocks (RSUs), more pooling operations, larger range of receptive fields, and richer local and global features. Configuring this parameter enables the extraction of multi-scale features from input feature maps with any spatial resolution. Multi-scale features are extracted from the gradually downsampled feature maps by progressive upsampling, concatenation, and convolution, and encoded into high-resolution feature maps. This process mitigates the loss of fine details caused by direct upsampling of large scales.

[0072] (iii) Residual connection, which fuses local and multi-scale features by addition F1(x)+U(F1(x)).

[0076] For better understanding, Figures 7A and 7B show schematic diagrams of the original residual block 700 (Figure 7A) and the residual U block (RSU) 720 (Figure 7B) for comparison. The operations in the original residual block 700 can be summarized as H(x) = F2(F1(x)) + x, where H(x) denotes the desired mapping of input features x, and F2, F1 represent weight layers, which in this setting are convolution operations. The main design difference between the RSU block 720 and the original residual block 700 is that in the RSU block 720, the simple single-stream convolution is replaced by a U-Net-like structure 600, and the original features are fed into the weight layer H RSUThe main difference between the RSU block 720 and the original residual block 700 is that the local features transformed by F1(x)=U(F1(x))+F1(x) are replaced by the local features transformed by F1(x), where U represents the multi-layer U structure 600 as shown in Fig. 6. Such a difference between the RSU block 720 and the original residual block 700 enables the network to extract features from multiple scales directly from each RSU block. Moreover, the computation overhead caused by the U structure is small because most operations are applied on the downsampled feature maps.

[0073]

[0077] In various exemplary embodiments, the AC-RSU block may be formed based on (e.g., the same as or similar to) the RSU block 720 described above (without being limited to any particular dimensions, such as the number of layers L, which may be changed or modified as desired or appropriate), whereby each simple convolutional layer in the RSU block 720 is replaced with an AC-Conv layer as described herein according to various exemplary embodiments.

[0074] ACU 2 -Net Architecture

[0078] According to various exemplary embodiments, the ACU n- A U-Net is disclosed, whereby multiple U-Net-like structures are stacked on top of each other in a nested manner. In particular, the exponential notation refers to a nested U-structure, rather than a serial stack. In theory, the exponent n can be set as any positive integer to realize single-level or multi-level nested U-structures. However, architectures with too many nesting levels are too complex to implement and use in real applications. For example, setting n to 2 results in an ACU 2 -Net can be formed. ACU 2 The -Net has a two-level nested U-structure, and FIG. 4A illustrates an exemplary ACU that forms a prediction module 410 according to various exemplary embodiments. 24A shows a schematic block diagram of the AC-RSU-Net. The top level is a U-structure that includes multiple stages (multiple cubes in FIG. 4A), e.g., but not limited to, 14 stages. Each stage is filled by a constructed AC-RSU block (the bottom-level U-structure). Accordingly, the nested U-structure enables the extraction of intra-stage multi-scale features and the aggregation of inter-stage multi-level features more efficiently.

[0075]

[0079] As shown in Figure 4A, the prediction module (ACU 2 The Prediction Module 410 includes a Prediction-Net 410 having an encoder-decoder structure that includes a set of encoder blocks 420 and a set of decoder blocks 430. By way of example only and not limitation, the Prediction Module 410 includes three parts: (1) a multi-stage (e.g., seven-stage) encoder structure 420, (2) a multi-stage (e.g., seven-stage) decoder structure 430, and (3) a feature map fusion module or block 440 coupled or attached to the decoder stage 430.

[0076]

[0080] For the encoder stage 420, an exemplary configuration of a set of encoder blocks 420 is shown in Table 1 of FIG. 5. For the decoder stage 430, an exemplary configuration of a set of decoder blocks 430 is also shown in Table 1 of FIG. 5. As mentioned above, “7”, “6”, “5”, and “4” indicate the height (L) of the AC-RSU block. For example, L can be configured according to the spatial resolution of the input feature map. For feature maps with large heights and widths, a larger L can be used to capture larger scale information. For example, the resolution of the feature maps in En_6 and En_7 is relatively low, and further downsampling of these feature maps will result in loss of useful context. Therefore, in both En_6 and En_7 stages, AC-RSU-4F is used, where “F” indicates that the AC-RSU block is a dilated version, e.g., pooling and upsampling operations are replaced by dilated convolutions. In this case, all of the intermediate feature maps of AC-RSU-4F have the same resolution as their input feature maps.

[0077]

[0081] With respect to the decoder stage 430, an exemplary configuration of a set of decoder blocks (AC-RSU) is also shown in Table 1 of FIG. 5. In various exemplary embodiments, the decoder stage 430 may have a structure similar to or corresponding to the symmetric or corresponding encoder stage 420. For example, the dilated AC-RSU-4F is also used for the decoder blocks De_6 and De_7, which are similar to or corresponding to those used for the symmetric or corresponding encoder blocks En_6 and En_7. As shown in FIG. 4A, each decoder stage may be configured to take as input a concatenation of an upsampled feature map from its immediately preceding stage and a downsampled feature map from its symmetric or corresponding encoder stage.

[0078]

[0082] In various exemplary embodiments, the prediction module 410 can be configured to generate multiple predicted feature maps based on the upsampled feature maps created by the decoder stage 430. By way of example only and not by way of limitation, the exemplary configuration shown in FIG. 4A generates seven predicted feature maps (e.g., side output saliency probability maps) S from decoder stages De_1, De_2, De_3, De_4, De_5, De_6, and De_7, respectively, based on a 3×3 convolutional layer and a sigmoid function. (1) side , S (2) side , S (3) side , S (4) side , S (5) side , S (6) side , S (7) side The prediction module 410 then upsamples the logits of the side output saliency map (the convolution output before the sigmoid function) to the input image size and fuses them in a concatenation operation followed by a 1×1 convolution layer and a sigmoid function to produce a fused feature map (e.g., a final saliency probability map) S fuse 444 can be generated.

[0079]

[0083] In response, A.C.U. 2 The construction of ACU-Net allows for a deep architecture with rich multi-scale features and relatively low computational and memory costs. 2 The -Net architecture is built based on the AC-RSU block without using a pre-trained backbone adapted from image classification, making it flexible and easy to adapt to different working environments without significant performance loss.

[0080]

[0084] Accordingly, in various exemplary embodiments, the prediction module 410 has an encoder-decoder structure including a set of encoder blocks (e.g., En_1-En_7) 420 and a set of decoder blocks (e.g., De_1-De_7) 430. As shown in FIG. 4A, for each of a plurality of encoder blocks (e.g., En_1-En_5) of the set of encoder blocks, a downsample feature map can be created using the encoder block based on the input feature map received by the encoder block. Furthermore, as shown in FIG. 4A, for each of a plurality of decoder blocks (e.g., De_1-De_5) of the set of decoder blocks, an upsample feature map can be created using the decoder block based on the input feature map and the downsample feature map created by the encoder block corresponding to the decoder block and received by the decoder block. Accordingly, in various exemplary embodiments, the plurality of predicted feature maps created based on the plurality of decoder blocks have different spatial resolution levels.

[0081]

[0085] In various exemplary embodiments, a plurality of predicted feature maps are generated based on a plurality of upsampled feature maps generated by a plurality of decoder blocks, respectively. Attention coordinate convolution (AC-Conv)

[0086] It should be noted that in various exemplary embodiments, soft tissue structures such as the thyroid gland in medical images are believed to have predictable location and shape patterns, which can be used to aid in the segmentation process. To solve the coordinate transformation problem, coordinate convolution (CoordConv) is disclosed as shown in FIG. 8A (see Liu et al., "An intriguing failing of convolution neural networks and the CoordConv solution," NIPS, pp. 9605-9616, 2018, herein referred to as reference Liu, the contents of which are incorporated herein by reference in their entirety for all purposes). In particular, FIG. 8A shows a schematic block diagram of an original CoordConv layer 800. The input feature map M in Considering (h×ω×c) 804, CoordConv is out =conv(cat(M in ,M i ,M j )), where M i 806 and M j 808 show the row and column coordinate maps, respectively. However, in various exemplary embodiments of the invention, the coordinate maps (M i ,M j ) is almost constant, we can use these as feature maps M in Note that directly concatenating these value scales to the feature map M in8B , which is a schematic block diagram of the AC-Conv 850 according to various exemplary embodiments of the present invention. The AC-Conv 850 adds a spatial attention-like operation before the concatenation (per channel) of the input feature map 854 and the coordinate maps 856′, 858′ (corresponding to the multiple weighted coordinate maps described above according to various embodiments).

[0082]

number

[0083] where σ is the sigmoid function.

[0087] Accordingly, in various exemplary embodiments, performing a feature extraction operation using a convolutional (AC-Conv) layer 850 includes creating an output feature map 870 of the convolutional layer 850 based on an input feature map 854 and a plurality of weighted coordinate maps 856′, 858′ received by the convolutional layer 850, creating a plurality of weighted coordinate maps 856′, 858′ based on the plurality of coordinate maps 856, 858 and a spatial attention map 860, and creating the spatial attention map 860 based on the input feature map 854 received by the convolutional layer 850 and modifying the coordinate information in each of the plurality of coordinate maps 856, 858 to create the plurality of weighted coordinate maps 856′, 858′. In various exemplary embodiments, creating the spatial attention map 860 includes performing a first convolution operation 862 based on the input feature map 854 received by the convolution layer 850 to create a convoluted feature map, and applying an activation function 864 based on the convoluted feature map to create the spatial attention map 860. In various exemplary embodiments, creating the multiple weighted coordinate maps 856′, 858′ includes multiplying each of the multiple coordinate maps 856, 858 with the spatial attention map 860 to modify coordinate information in each of the multiple coordinate maps 856, 858. In various exemplary embodiments, creating the output feature map 870 of the convolutional layer 850 includes concatenating, for each channel, the input feature map 854 and the multiple weighted coordinate maps 856′, 858′ received by the convolutional layer 850 to form a concatenated feature map 866, and performing a second convolution operation 868 based on the concatenated feature map 866 to create the output feature map 870 of the convolutional layer 850.

[0084]

[0088] Behavior such as spatial attention is i) M in and {M i ,M j}, and ii) reweighting of all pixel coordinates rather than using a constant coordinate map to capture more significant geometric information with the guidance of an attention map 860 derived from the current input feature map 854. For example, for two coordinates i and j, an i coordinate map (or i coordinate channel) 856 and a j coordinate map (or j coordinate channel) 858 can be provided. For example, the i coordinate map 856 can be a rank-1 matrix of h×ω, whose first row is filled with 0s, whose second row is filled with 1s, whose third row is filled with 2s, and so on. The j coordinate map 858 can be the same or similar to the i coordinate map 856, but with columns rather than rows filled with the values ​​mentioned above. As previously described by various exemplary embodiments, U 2 The RSU 720 used in the -Net can be modified or adapted to create or construct an AC-RSU according to various exemplary embodiments by replacing the convolutional layer with an AC-Conv layer 850 according to various exemplary embodiments. For example, compared to the RSU 720, the AC-RSU is capable of extracting both texture and geometric features from different receptive fields. In various exemplary embodiments, the prediction module ACU 2 -Net410, as well as three sub-network ACUs in the refined E-Module450 2 -Net-Ref7, ACU 2 -Net-Ref5, and ACU 2 -Net-Ref3 is entirely built on AC-RSU.

[0085] Parallel Multi-Head Residual Refinement Module (MH-RRM)

[0089] In an attempt to further improve accuracy, several conventional predictive refinement models have been proposed, as shown in FIG.

[0086]

number

[0087] As a result, the coarse results are recursively or progressively refined by serial sub-networks (serial refinement modules).

[0088]

number

[0089] is theoretically the most accurate output and is therefore usually utilized as the final result. This serial refinement strategy can reduce bias in the segmentation results. However, various exemplary embodiments have found that segmentation of soft tissues in ultrasound images using such networks often has high variance in practice due to poor image quality and blurred boundaries. To reduce bias and variance in predictions, a multi-model ensemble strategy can be used. However, various exemplary embodiments have found that direct ensembling of multiple deep models requires high computational and time costs. To address these issues with conventional techniques, various exemplary embodiments embed an ensemble strategy in the refinement module. In particular, various exemplary embodiments of the present invention provide a simple and effective parallel multi-head residual refinement module (MH-RRM) 450 as shown in FIG. 4B. By way of example only and not limitation, the number of MH-RRM heads 454-1, 454-2, 454-3 (e.g., corresponding to the multiple refinement blocks described above by various embodiments) according to various exemplary embodiments may be 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 45, 46, 47, 48, 49, 50, 51, 52, 44, 45, 46, 47, 48, 49, 53, 49, 54, 55, 56, 57, 58, 59, 60, 61,

[0090]

number

[0091] As previously described, the three refinement heads or blocks 454-1, 454-2, 454-3 each include an ACU configured to generate refinement feature maps having different spatial resolution levels based on the fusion feature map 444. 2-Net. In various exemplary embodiments, the refinement blocks 454-1, 454-2, 454-3 generate a plurality of refinement feature maps 464-1, 464-2, 464-3, respectively. Accordingly, in various exemplary embodiments, the refinement feature maps 464-1, 464-2, 464-3 have different spatial resolution levels.

[0092]

[0090] In various exemplary embodiments, each of the refinement blocks 454-1, 454-2, 454-3 has an encoder-decoder structure including a plurality of encoder blocks and a plurality of decoder blocks. For each refinement block and each of the plurality of encoder blocks of the refinement block, a downsample feature map can be created using the encoder block based on the input feature map received by the encoder block, as shown in FIG. 4B. Furthermore, for each refinement block and each of the plurality of decoder blocks of the refinement block, a decoder block can be used to create an upsample feature map based on the input feature map and the downsample feature map created by the encoder block corresponding to the decoder block and received by the decoder block, as shown in FIG. 4B. In various exemplary embodiments, the plurality of encoder-decoder structures of the refinement blocks have different heights.

[0093]

[0091] In various exemplary embodiments, as shown in Figure 4B, for each refinement block, a refinement feature map for the refinement block may be created based on the fused feature map 444 received by the refinement block and the upsample feature map created by a first decoder block 458-1, 458-2, 458-3 of the refinement block's multiple decoder blocks. In various exemplary embodiments, the output image of the exemplary CNN 400 is created based on the average of the set of refinement feature maps 464-1, 464-2, 464-3.

[0094] Accordingly, in various exemplary embodiments, given an input image X, the final segmentation result of the exemplary CNN 400 is

[0095]

number

[0096] It can be expressed as:

[0093] Figure 9B illustrates the semantic workflow of the prediction refinement architecture of an exemplary CNN 400 with the parallel refinement modules described above. In Figures 9A and 9B, bold indicates the final prediction result.

[0097] Training and Inference

[0094] In the training process, the three refined outputs R (1) 464-1, R (2) 464-2, and R (3) 464-3, as shown in FIG. 4A and FIG. 4B, has seven side outputs

[0098]

number

[0099] and one fused output S from the prediction module 410 fuse The whole model is supervised by a binary cross entropy (BCE) loss, which is calculated independently along with the

[0100]

number

[0101] It can be trained end-to-end by

[0102]

number

[0103] is the total loss,

[0104]

number

[0105]

number

[0106] and

[0107]

number

[0108] are the corresponding losses of the side output, fusion output, and refinement output,

[0109]

number

[0110]

number

[0111] and

[0112]

number

[0113] are the corresponding weights for emphasizing different outputs. In the experiments conducted by various exemplary embodiments, the λ weights are all set to 1.0. In the inference process, R (1) 464-1, R (2) 464-2, and R (3) The average of 464-3 is used as the final prediction result (eg, corresponding to the output image of the CNN described above according to various embodiments).

[0114] experiment

[0095] The thyroid gland is a butterfly-shaped organ located at the base of the neck just above the clavicle, with left and right lobes connected by a thin band of tissue in the middle called the isthmus (see FIG. 10). In particular, FIG. 10 shows a schematic diagram of the thyroid gland and ultrasound scanning protocol along with corresponding ultrasound images with manually labeled thyroid lobe overlays 1010. The dotted arrows in the top row of images in FIG. 10 indicate the scanning direction of the ultrasound probe in the transverse (TRX) and sagittal (SAG) planes. The bottom row of images in FIG. 10 shows sample TRX images (left) and SAG images (right) along with the manually labeled thyroid lobe overlays 1010.

[0115] To diagnose thyroid abnormalities, a clinician can manually segment the thyroid gland from an acquired ultrasound scan to assess its size. For illustrative purposes and not for limitation, by way of example only, the exemplary CNN 400 was evaluated as a case study on the problem of thyroid tissue segmentation.

[0116] Dataset

[0097] None of the existing public datasets appear to be suitable for large-scale learning-based methods. To enable large-scale clinical applications, a comprehensive thyroid ultrasound segmentation dataset was collected with the approval of the Health Research Ethics Committees of the participating institutions.

[0117]

[0098] Regarding the collection of ultrasound scans, 777 ultrasound scans were retrospectively collected from 700 patients aged 18 to 82 years found at thyroid ultrasound examinations in 12 different imaging facilities. The scans were divided into transverse (TRX) and sagittal (SAG) planes according to the scanning direction of the ultrasound probe (see, for example, FIG. 10). Thus, two partitions (TRX and SAG sets) were available. Each partition was further randomly divided into three subsets based on patient IDs for training, validation, and testing, so that the same patient was not included in two different subsets. FIG. 11 shows a table (Table 2) showing the volume numbers and corresponding slices (images) in each subset. In particular, Table 2 shows the number of TRX and SAG thyroid scans in the thyroid dataset, whereby "Vol#" and "Slice#" indicate the volume number and the corresponding labeled image, respectively.

[0118] Regarding annotation or labeling, the images in the dataset were manually labeled by five experienced sonographers and verified by three radiologists. Considering the availability of a fairly large number of images in total, to save labeling time, ultrasound scans in the training set were labeled every third or fifth slice. However, the validation and testing sets were labeled slice by slice for accurate volumetric estimation.

[0119]

[0100] Regarding implementation details, the exemplary CNN400 was implemented by PyTorch. The performance of the exemplary CNN400 was evaluated using the specified training, evaluation, and test sets. In the training process, the input images were first resized to 160x160x3 and then randomly cropped to 144x144x3. Online random horizontal and vertical flips were used to augment the dataset. The training batch size was set to 12. The model weights were initialized by default He's uniform initialization (see, for example, He et al., "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification," Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015). The Adam optimizer (see, e.g., Kingma, "Adam: A method for stochastic optimization", arXiv preprint arXiv:1412.6980, 2014) was used with a learning rate of le-3 and no weight decay. The training loss converged after about 50,000 iterations, which took about 24 hours. In the testing process, the input images were resized to 160x160x3 and fed into the example CNN. In both the downsampling and upsampling processes, bilinear interpolation was used. Both the training and testing processes were performed on a 12-core, 24-thread PC with an AMD Ryzen Threadripper 2920x 4.3 GHz CPU (128 GB RAM) and an NVIDIA GTX 1080 Ti GPU.

[0120]

[0101] Regarding the evaluation metrics, we used two measures to evaluate the overall performance of our method: the volume Dice (see, e.g., Popovic et al., "Statistical validation metric for accuracy assessment in medical image segmentation," IJCARS, 2(2-4):169-181, 2007) and its standard deviation σ. The Dice score is

[0121]

number

[0122] where P and G are the predicted segmentation mask sweep (h × ω × c) and the ground truth mask sweep (h × ω × c), respectively. The standard deviation of the Dice score is

[0123]

number

[0124] where N is the number of test volumes and Dice μ is the average volumetric Dice score across all test sets. In the experiments conducted, the average Dice for each test set was reported along with its standard deviation (σ).

[0125]

[0102] An exemplary CNN (ACU 2E-Net)400 is a convolutional network based on 11 state-of-the-art (SOTA) models, including U-Net (Ronneberger et al., "U-net: Convolutional networks for biomedical image segmentation," MICCAI, pp. 234-241, 2015), as well as Res U-Net (see, for example, Xiao et al., "Weighted Res-UNet for high-quality retina vessel segmentation," ITME, pp. 327-331, 2018), Dense U-Net (see, for example, Guan et al., "Fully Dense UNet for 2-D Sparse Photoacoustic Tomography Artifact Removal," IEEE JBHI, 24(2):568-576, 2019), and Attention U-Net (see, for example, Xiao et al., "Weighted Res-UNet for high-quality retina vessel segmentation," ITME, pp. 327-331, 2018). U-Net (see, e.g., Oktay et al., “Attention u-net: Learning where to look for the pancreas,” arXiv preprint arXiv:1804:03999, 2018), U-Net++ (see, e.g., Zhou et al., “Unet++: A nested u-net architecture for medical image segmentation,” MICCAI-W, pp. 3–11, 2018), and U 2 -Net (e.g., Qin et al., "U2-Net: Going Deeper with nested U-structure for salient object detection (U 2We have also developed a number of 3D models, including Stacked HourglassNet (see, e.g., Newell et al., "Stacked hourglass networks for human pose estimation," ECCV, 483-499, 2016), SRM (see, e.g., Wang et al., "A stagewise refinement model for detecting salient objects in images," ICCV, 4019-4028, 2017), CU-Net (see, e.g., Tang et al., "Quantized densely connected u-nets for efficient landmark localization," ECCV, 339-354, 2018), and five of its variants. 3 -Net(Deng et al., “R3net:Recurrent residual refinement network for saliency detection(R 3 net: Recurrent Residual Refinement Networks for Saliency Detection, AAAI, 2018), and BASNet (Qin et al., "Basnet: Boundary-aware salient object detection," CVPR, pp. 7479-7489, 2019).

[0126] FIG. 12 shows a table (Table 3) that shows a quantitative evaluation or comparison of the exemplary CNN400 with other state-of-the-art segmentation models on the TRX and SAG test sets. The top part of Table 3 includes a comparison against the traditional U-Net and its variants such as Attention U-Net, and the bottom part of the table shows the performance of R 3We show a comparison against models with predictive refinement strategies such as CNN-Net. It can be observed that the exemplary CNN 400 produces the highest Dice scores in both TRX and SAG images. Furthermore, the parallel refinement module 450 compares the best models (BASNet and R 3 Compared to other refined module designs such as -Net, it significantly improves the Dice score by 2.55% and 1.22%, and reduces the standard deviation σ by 31.99% and 7.51%.

[0127] 13A-13L and 14A-14L show sample segmentation results in TRX and SAG thyroid images. In particular, 13A-13L show a qualitative comparison of the ground truth (white dotted line) and segmentation results (white solid line) for different methods in a sampled TRX slice of a homogeneous thyroid, and 14A-14L show a qualitative comparison of the ground truth (white dotted line) and segmentation results (white solid line) for different methods in a sampled SAG slice of a heterogeneous thyroid. As can be seen, the exemplary CNN 400 was able to provide improved (more accurate) segmentation results. Specifically, 13A-13L show a homogeneous TRX thyroid lobe with large sparkle noise and blurred boundaries. Res U-Net, U-Net++, SRM, CU-Net, R 3 -Net, and BASNet cannot capture the exact boundaries. 2 Other models, such as CNN-Net and Stacked HourglassNet, are unable to segment the elongated upper left region of the thyroid gland. Figures 14A-14L show segmentation results of the thyroid gland in a heterogeneous SAG image containing several complex nodules. Accordingly, as can be seen, the exemplary CNN 400 provides relatively better results than the other models.

[0128] To further evaluate the robustness of the exemplary CNN400, the success rate curves of the exemplary CNN400 and other eleven state-of-the-art models in TRX and SAG images are plotted in Fig. 15A and Fig. 15B, respectively. The success rate is defined as the ratio of the number of scan predictions (with a score higher than a certain Dice threshold) to the total number of scans. A higher success rate indicates better performance, and thus the upper curve (ACU 2 E-Net) outperforms the other 11 state-of-the-art models it is compared to. Accordingly, as can be seen, the exemplary CNN400 outperforms the other models by a large margin on both the TRX and SAG test sets.

[0129] To verify the validity of AC-Conv according to various exemplary embodiments, an adapted U 2We have proposed a number of new convolutional networks that use simple convolutions in convolutional neural networks (SimpleConv) (LeCun et al., "Gradient-based learning applied to document recognition," Proceedings of IEEE, 86(11):2278-2324, 1998), SE-Conv (Hu et al., "Squeeze-and-excitation networks," CVPR, pp. 7132-7141, 2018), which explicitly models channel interdependence through its squeeze-and-excitation blocks; CBAM-Conv (Woo et al., "Cbam: Convolutional block attention module," ECCV, pp. 3-19, 2018), which refines feature maps through its channel and spatial attention blocks; and CoordConv (Liu et al., "An intriguing failing of convolutional neural networks and the CoordConv Ablation studies were performed by replacing the 1-D convolutional neural network with ... 2 These experiments were performed on the TRX thyroid test set. The results on the TRX test set are shown in the top part of Table 4. As can be seen, the ACU with AC-Conv 2 -Net gives the best results in terms of both Dice score and standard deviation σ, which further demonstrates that a combined strategy of jointly perceiving geometric and spatial information is more effective than a single spatial attention-based method (CBAM) or coordinate-based method (CoordConv).

[0130] To validate the performance of MH-RRM (E-Module), ablation studies were also conducted on different refinement configurations, including serial RRM Ref3 (Ref5 (Ref7)), three identical RRMs with parallel RRMs avg(Refk,Refk,Refk) {k=3,5,7}, and fused parallel RRM conv(Ref7,Ref5,Ref3), where the parallel refinement outputs were fused by convolutional layers instead of averaging in inference. The bottom part of Table 4 shows the ablation results on RRMs, showing that serial RRM, parallel RRMs with the same branches, and fused parallel RRMs are all inferior to MH-RRM according to various exemplary embodiments.

[0131] Accordingly, the various exemplary embodiments advantageously employ an Attention-Based Predictive Refinement Network (ACU) for segmentation of soft tissue structures in ultrasound images. 2 E-Net) 400. In particular, ACU 2 E-Net is built based on (a) an attention coordinate convolution (AC-Conv)850, which fully exploits the geometric information of the thyroid gland in ultrasound images, and (b) a parallel multi-head refinement module (MH-RRM)450, which refines the segmentation results by integrating an ensemble strategy with a residual refinement method.

[0132]

[0109] Extensive ablation studies and comparisons with the aforementioned state-of-the-art models demonstrate the effectiveness and robustness of the exemplary CNN 400 without complicating the training and inference process. Although the exemplary CNN 400 has been described with respect to segmenting thyroid tissue from ultrasound images, it will be understood that the exemplary CNN 400, as well as AC-Conv850 and MH-RRM450, are not limited to application to segment thyroid tissue from ultrasound images, and may also be applied to segment other types of tissue from ultrasound images, such as, but not limited to, liver, spleen, and kidney, as well as tumors (e.g., hepatocellular carcinoma (HCC) in the liver or subcutaneous masses), as desired or appropriate.

[0133]

[0110] Although the embodiments of the present invention have been particularly shown and described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and detail can be made therein without departing from the scope of the invention as defined by the appended claims. The scope of the present invention is therefore indicated by the appended claims, and it is therefore intended to embrace all changes that come within the meaning and range of equivalence of the claims.

Claims

1. 1. A method for image processing based on a convolutional neural network (CNN) using at least one processor, comprising: receiving an input image; performing a plurality of feature extraction operations using a plurality of respective convolutional layers of the CNN based on the input image to generate a plurality of respective output feature maps; generating an output image for the input image based on the output feature maps of the multiple convolutional layers; Including, for each of the plurality of feature extraction operations, performing the feature extraction operation using the convolutional layer, generating an output feature map of the convolutional layer based on an input feature map and a plurality of weighted coordinate maps received by the convolutional layer; generating the plurality of weighted coordinate maps based on a plurality of coordinate maps and a spatial attention map; generating the spatial attention map based on an input feature map received by the convolutional layer and modifying coordinate information in each of the plurality of coordinate maps to generate the plurality of weighted coordinate maps; A method comprising:

2. The step of generating the spatial attention map further comprises: performing a first convolution operation based on an input feature map received by the convolution layer to create a convolutional feature map; applying an activation function based on the convolutional feature map to create the spatial attention map; The method of claim 1 , comprising:

3. The method of claim 2 , wherein the activation function is a sigmoid activation function.

4. The method of claim 2 or 3, wherein creating the plurality of weighted coordinate maps comprises multiplying each of the plurality of coordinate maps with the spatial attention map to modify the coordinate information in each of the plurality of coordinate maps.

5. 5. The method of claim 2, wherein the plurality of coordinate maps comprises a first coordinate map comprising coordinate information for a first dimension and a second coordinate map comprising coordinate information for a second dimension, the first dimension and the second dimension being two dimensions along which the first convolution operation is configured to be performed.

6. generating the output feature map of the convolutional layer, concatenating, for each channel, an input feature map received by the convolutional layer and the plurality of weighted coordinate maps to form a concatenated feature map; performing a second convolution operation based on the connected feature map to generate the output feature map of the convolution layer; The method according to any one of claims 1 to 5, comprising:

7. the CNN includes a prediction sub-network including at least one convolutional layer of the plurality of convolutional layers of the CNN; the method further comprising: generating a set of predicted feature maps using the predictive sub-network based on the input image, the generating the set of predicted feature maps comprising performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the predictive sub-network; a plurality of predicted feature maps in the set of predicted feature maps having different spatial resolution levels; The method according to any one of claims 1 to 6.

8. the prediction sub-network has an encoder-decoder structure including a set of encoder blocks and a set of decoder blocks, the set of encoder blocks of the prediction sub-network including a plurality of encoder blocks, and the set of decoder blocks of the prediction sub-network including a plurality of decoder blocks; The method further comprising: for each of the plurality of encoder blocks of the prediction sub-network, creating a downsampled feature map using the encoder block based on an input feature map received by the encoder block; for each of the plurality of decoder blocks of the prediction sub-network, creating an upsample feature map using the decoder block based on an input feature map and the downsample feature map created by the encoder block corresponding to the decoder block and received by the decoder block; The method of claim 7 further comprising:

9. 9. The method of claim 8, wherein creating the set of predicted feature maps using the prediction sub-network comprises creating the plurality of predicted feature maps based on the plurality of upsampled feature maps created by the plurality of decoder blocks, respectively.

10. creating the downsampled feature map using the encoder block of the prediction sub-network, extracting multi-scale features based on the input feature map received by the encoder block; creating the downsampled feature map based on the extracted multi-scale features extracted by the encoder block; Including, generating the upsampled feature map using the decoder blocks of the prediction sub-network, extracting multi-scale features based on the input feature map and the downsampled feature map produced by the encoder block corresponding to the decoder block and received by the decoder block; creating the upsampled feature map based on the extracted multi-scale features extracted by the decoder block; Including, 10. The method according to claim 8 or 9.

11. Each of the plurality of encoder blocks of the prediction sub-network includes at least one convolutional layer of a plurality of convolutional layers of the CNN, and creating the downsampled feature map using the encoder blocks of the prediction sub-network includes performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the encoder blocks; each of the plurality of decoder blocks of the prediction sub-network includes at least one convolutional layer of the plurality of convolutional layers of the CNN, and creating the upsampled feature map using the decoder blocks of the prediction sub-network includes performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the decoder blocks; The method according to any one of claims 8 to 10.

12. Each convolutional layer of each of the plurality of encoder blocks of the prediction sub-network is one of the plurality of convolutional layers of the CNN; Each convolutional layer of each of the plurality of decoder blocks of the prediction sub-network is one of a plurality of convolutional layers of the CNN. The method of claim 11.

13. each of the plurality of encoder blocks of the prediction sub-network is configured as a residual block; each of the plurality of decoder blocks of the prediction sub-network is configured as a residual block; The method according to any one of claims 8 to 12.

14. The CNN further includes a refinement sub-network including at least one convolutional layer of the plurality of convolutional layers of the CNN; The method further includes generating a set of refined feature maps using the refinement sub-network based on the fusion feature map, the generating the set of refined feature maps including performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the refinement sub-network; A plurality of refinement feature maps of the set of refinement feature maps have different spatial resolution levels. The method according to any one of claims 7 to 13.

15. The method of claim 14 , further comprising concatenating the set of predicted feature maps to create the fused feature map.

16. The refinement sub-network includes a plurality of refinement blocks configured to respectively generate the plurality of refinement feature maps, each of the plurality of refinement blocks having an encoder-decoder structure including a set of encoder blocks and a set of decoder blocks, the set of encoder blocks of the refinement sub-network includes a plurality of encoder blocks, and the set of decoder blocks of the refinement sub-network includes a plurality of decoder blocks; The method further comprises the steps of: for each of the plurality of refining blocks: for each of the plurality of encoder blocks of the refinement block, creating a downsampled feature map using the encoder block based on an input feature map received by the encoder block; For each of the plurality of decoder blocks of the refinement block, creating an upsample feature map using the decoder block based on an input feature map and the downsample feature map created by the encoder block corresponding to the decoder block and received by the decoder block; Further comprising:

16. The method according to claim 14 or 15.

17. The method of claim 16 , wherein the encoder-decoder structures of the refining blocks have different heights.

18. The method of claim 16 or 17, wherein the step of creating the set of refinement feature maps using the refinement sub-network includes, for each of the plurality of refinement blocks, creating the refinement feature map for the refinement block based on the fused feature map received by the refinement block and the upsample feature map created by a first decoder block of the plurality of decoder blocks of the refinement block.

19. creating the downsampled feature map using the encoder block of the refinement block; extracting multi-scale features based on the input feature map received by the encoder block; creating the downsampled feature map based on the extracted multi-scale features extracted by the encoder block; Including, creating the upsampled feature map using the decoder block of the refinement block; extracting multi-scale features based on the input feature map and the downsampled feature map generated by the encoder block of the refinement block corresponding to the decoder block and received by the decoder block; creating the upsampled feature map based on the extracted multi-scale features extracted by the decoder block; Including, The method according to any one of claims 16 to 18.

20. For each of the plurality of refining blocks, Each of the plurality of encoder blocks of the refinement block includes at least one convolutional layer of a plurality of convolutional layers of the CNN, and creating the downsampled feature map using the encoder blocks of the refinement block includes performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the encoder blocks; Each of the plurality of decoder blocks of the refinement block includes at least one convolutional layer of a plurality of convolutional layers of the CNN, and the step of creating the upsampled feature map using the decoder blocks of the refinement block includes a step of performing at least one feature extraction operation of the plurality of feature extraction operations using the at least one convolutional layer of the decoder blocks.

20. The method according to any one of claims 16 to 19.

21. Each convolutional layer of each of the plurality of encoder blocks of the refinement block is one of a plurality of convolutional layers of the CNN; Each convolutional layer of each of the plurality of decoder blocks of the refinement block is one of a plurality of convolutional layers of the CNN; 21. The method of claim 20.

22. For each of the plurality of refining blocks, Each of the plurality of encoder blocks of the refinement block is configured as a residual block; Each of the plurality of decoder blocks of the refinement block is configured as a residual block. The method according to any one of claims 16 to 21.

23. The method of any one of claims 14 to 21, wherein the output image is generated based on the set of refined feature maps.

24. The method of claim 23 , wherein the output image is created based on an average of the set of refined feature maps.

25. receiving the input image includes receiving a plurality of input images, each of the plurality of input images being a labeled image for training the CNN to obtain a trained CNN; For each of the plurality of input images, performing the feature extraction operations using the plurality of convolutional layers of the CNN based on the input image to generate the plurality of output feature maps, respectively; generating an output image for the input image based on the output feature maps of the convolutional layers; Including, 25. The method according to any one of claims 1 to 24.

26. The method of claim 25 , wherein the labeled image is a labeled ultrasound image including tissue structures.

27. The method of any one of claims 1 to 24, wherein the output image is a result of inference on the input image using the CNN.

28. 28. The method of claim 27, wherein the input image is an ultrasound image containing tissue structures.

29. A CNN-based image processing system, comprising: Memory, At least one processor communicatively coupled to the memory and configured to perform the CNN-based image processing method of any one of claims 1 to 28; A system comprising:

30. A computer program product embodied in one or more non-transitory computer-readable storage media and comprising instructions executable by at least one processor to perform the CNN-based image processing method of any one of claims 1 to 28.

31. 1. A method for segmenting tissue structures in an ultrasound image using a CNN, using at least one processor, comprising:

25. The method for image processing based on CNN according to claim 1, the input image is the ultrasound image including the tissue structure; The method, wherein the output image has the tissue structure segmented and is the result of inference on the input image using the CNN.

32. 32. The method of claim 31 , wherein the CNN is trained as described in claim 25 or 26.

33. 1. A system for segmenting tissue structures in ultrasound images using a CNN, comprising: Memory, At least one processor communicatively coupled to the memory and configured to execute a method for segmenting tissue structures in ultrasound images using a CNN according to claim 31 or 32; A system comprising:

34. 33. A computer program product embodied in one or more non-transitory computer readable storage media and comprising instructions executable by at least one processor to perform the method for segmenting tissue structures in ultrasound images using a CNN according to claim 31 or 32.

Images