Image segmentation method using dual attention and device using the same

Abstract

The image segmentation method of the present invention may include the steps of: extracting spatial attention from input data; extracting channel attention from the input data; generating merging data based on the spatial attention and the channel attention; and obtaining result data including component-specific indices based on the merging data.

Description

Technology Field

[0001] The present invention relates to an image segmentation method, and more specifically, to a method for segmenting an image using dual attention. Background Technology

[0002] Medical image segmentation is a field that enables rapid and easy analysis of medical images and accurate diagnosis by distinguishing lesions such as tumors and segmenting various organs. In particular, for complex medical images such as 3D CT and MRI, it is important to identify useful spatial and channel information from feature data obtained through deep neural network models to accurately distinguish multiple organs.

[0003] Existing U-Net-based deep neural network studies have limitations in obtaining information for accurate medical image segmentation because they focus solely on spatial information, despite the increase in the number of channels as the model size grows. Therefore, to additionally secure contextual information regarding medical images, a technique utilizing dual attention is required that extracts and fuses channel information in addition to spatial information. The problem to be solved

[0004] One objective of the present invention is to an image segmentation method using dual attention. means of solving the problem

[0005] An image segmentation method according to one embodiment, in an image segmentation method performed by at least one processor, may include: a step of extracting spatial attention from input data; a step of extracting channel attention from the input data; a step of generating merging data based on the spatial attention and the channel attention; and a step of obtaining result data including component-specific indices based on the merging data.

[0006] Here, the spatial attention includes shape information related to the component, and the channel attention may include context information related to the component.

[0007] Here, the step of extracting the spatial attention may be a step using a swin transformer.

[0008] Here, the step of extracting the spatial attention comprises: a step of extracting a first spatial attention based on a patch of a first size and a number of channels of a first value generated by the input data; and a step of extracting a second spatial attention based on a patch of a second size and a number of channels of a second value generated by the input data, wherein the second size may be larger than the first size and the first value may be smaller than the second value.

[0009] Here, the step of extracting the channel attention may include: a step of extracting the first channel attention based on the number of channels of the first value and the patch of the first size; and a step of extracting the second channel attention based on the number of channels of the second value and the patch of the second size.

[0010] Here, the step of generating the merging data may include: a step of generating first bottom-up data based on a first spatial attention, a first channel attention, a second spatial attention, and a second channel attention; and a step of generating second bottom-up data based on the first bottom-up data, a third spatial attention, and a third channel attention.

[0011] Here, the step of generating the merging data may include: a step of generating a first top-down data based on the first bottom-up data, the second bottom-up data, the second spatial attention, and the second channel attention; and a step of generating a second top-down data based on the first top-down data, the first spatial attention, and the first channel attention.

[0012] Here, the step of obtaining the result data comprises: a step of generating first decoding data based on a third size patch and a channel number of a third value generated by the merging data; and a step of generating second decoding data based on a fourth size patch and a channel number of a fourth value generated by the merging data, wherein the fourth size may be smaller than the third size and the third value may be larger than the fourth value.

[0013] Here, the step of obtaining the result data may include: a step of setting a first index corresponding to a first component in a first pixel based on the first decoding data and the second decoding data; a step of setting a second index corresponding to a second component in a second pixel based on the first decoding data and the second decoding data; and a step of setting a first region including a pixel where the first index is set and a second region including a pixel where the second index is set.

[0014] Here, the step of generating the first decoding data and the step of generating the second decoding data may be a step of using a swin transformer.

[0015] Here, a computer program recorded on a computer-readable recording medium may be provided to execute the image segmentation method.

[0017] An electronic device according to one embodiment comprises: a memory; and at least one processor, wherein, when executed by the at least one processor, the memory may store spatial attention extracted from input data, store channel attention extracted from the input data, store merging data generated based on the spatial attention and the channel attention, and store result data including a component-specific index obtained based on the merging data.

[0019] An image segmentation device according to one embodiment may include an encoding unit that extracts spatial attention and channel attention from input data; a merging unit that generates merging data based on the spatial attention and the channel attention; and a decoding unit that generates result data including component-specific indices based on the merging data. Effects of the invention

[0020] According to one embodiment of the present invention, an image segmentation method using dual attention may be provided. Brief explanation of the drawing

[0021] FIG. 1 is a flowchart of an image segmentation method according to one embodiment. FIG. 2 is a diagram illustrating an image segmentation method according to one embodiment. FIG. 3 is a diagram illustrating a spatial attention and channel attention merging method according to one embodiment. FIGS. 4 to 9 are drawings for explaining the results according to the image segmentation method of the present invention in comparison with the prior art. Specific details for implementing the invention

[0022] The embodiments described in this specification are intended to clearly explain the concept of the present invention to those skilled in the art to which the present invention belongs. Therefore, the present invention is not limited to the embodiments described in this specification, and the scope of the present invention should be interpreted to include modifications or variations that do not deviate from the concept of the present invention.

[0023] The terms used in this specification have been selected to be as widely used as possible, taking into account their functions in the present invention; however, these terms may vary depending on the intent of those skilled in the art to which the present invention pertains, case law, or the emergence of new technologies. However, if a specific term is defined and used with an arbitrary meaning, the meaning of that term will be described separately. Accordingly, the terms used in this specification should be interpreted based on their actual meaning and the content throughout this specification, rather than merely their names.

[0024] The drawings attached to this specification are intended to facilitate the explanation of the present invention. Since the shapes depicted in the drawings may be exaggerated as necessary to aid in understanding the present invention, the present invention is not limited by the drawings.

[0025] In this specification, if it is determined that a detailed description of known configurations or functions related to the present invention may obscure the essence of the present invention, such detailed description will be omitted as necessary.

[0027] The present invention relates to a method and apparatus for segmenting an image using extracted dual attention. Specifically, the present invention relates to an electronic device and an image segmentation device capable of extracting spatial attention and channel attention from input data. The electronic device or the image segmentation device may include at least one processor as a computing device, comprising a memory and a processor capable of executing said memory. The image segmentation method described below may be interpreted as being performed by at least one processor included in the electronic device or the image segmentation device.

[0029] FIG. 1 is a flowchart of an image segmentation method according to one embodiment.

[0030] Referring to FIG. 1, an image segmentation method according to one embodiment may include a step of extracting spatial attention and channel attention (S100), a step of generating merging data (S200), and a step of generating result data (S300). FIG. 1 illustrates steps S100 to S300 being performed sequentially, but is not limited thereto, and new steps may be added or some steps may be merged and performed simultaneously.

[0031] An electronic device or image segmentation device may include an encoding unit, a merging unit, and a decoding unit. The encoding unit may extract spatial attention and channel attention from input data. The merging unit may generate merging data based on the spatial attention and channel attention extracted by the encoding unit. Specifically, the merging unit may generate merging data by merging spatial attention and channel attention. The decoding unit may generate result data including component-specific indices based on the merging data generated by the merging unit. The encoding unit may be the subject of step S100, the merging unit may be the subject of step S200, and the decoding unit may be the subject of step S300.

[0032] The image segmentation method will be described in detail below with reference to FIGS. 2 and FIGS. 3.

[0034] FIG. 2 is a diagram illustrating an image segmentation method according to one embodiment.

[0035] Referring to FIG. 2, an image segmentation method according to one embodiment may include an encoding process (100), a merging process (200), and a decoding process (300) using input data (10). The encoding process (100) may include a process (110) for extracting spatial attention from input data (10) and a process (120) for extracting channel attention from input data (10). The merging process (200) may include a top-down merging process (210) and a bottom-up merging process (220).

[0036] The encoding process (100) may include a step (S100) of extracting spatial attention and channel attention from input data (10). Before extracting spatial attention and channel attention, the processor may perform a process of dividing the input data (10) into a plurality of patches. At this time, the size of the patches and the number of channels related to the information extracted from the patches may vary depending on the extraction process.

[0037] A processor can extract spatial attention from input data (10) divided into multiple patches using a spatial stream (e.g., a spatial attention extraction module). At this time, the spatial attention may include shape information or location information related to a component (e.g., an organ such as a stomach, liver, etc.). Additionally, the spatial attention may also include context information contained in channel attention. Specifically, the processor can extract a first spatial attention based on a patch of a first size generated by the input data (10) and the number of channels of a first value. For example, the processor can extract a first spatial attention based on 32 channels for a patch having a height, width, and depth of 4, 4, and 2, respectively.

[0038] Additionally, the processor can extract a second spatial attention based on a second size patch generated by the input data (10) and the number of channels of a second value. At this time, the second size may be larger than the first size, and the first value may be smaller than the second value. For example, the processor can extract a second spatial attention based on 64 channels for a patch having a height, width, and depth of 8, 8, and 4, respectively. In this way, the processor has a plurality of extraction processes and can extract spatial attention based on a patch of a different size and a different number of channels for each process.

[0039] As the processor performs multiple extraction processes, the patch size increases and spatial attention is extracted using large-value channels, allowing for the acquisition of a greater amount of information as the process progresses. Since spatial features are compressed as multiple extraction processes are conducted, the processor can acquire increasingly global information from local information. At this stage, to compensate for the information lost due to spatial compression, the channel information is increased to acquire various global contextual information regarding the feature map.

[0040] Additionally, the method of increasing channels can be performed by shifting a portion of the spatial information before compression toward the channels and then linearly transforming the channel features. Accordingly, each channel information may possess arbitrary spatial information. For example, the first channel may contain information about a part of the liver, and the second channel may contain information about a part of the heart. The processor can extract channel attention to focus on information related to components (e.g., organs such as the liver and heart) among the channel information. Details regarding channel attention will be discussed later.

[0041] The processor can utilize a swin transformer when extracting spatial attention. In other words, the step of extracting spatial attention can be a step that utilizes a swin transformer. The swin transformer is a model that stacks transformer encoders hierarchically, and it can perform attention between windows through a shifted window algorithm.

[0042] The image segmentation method of the present invention requires a large amount of computation because it performs multiple extraction processes while reducing the patch size and increasing the number of channels. Accordingly, the present invention enables more computation by utilizing a Swin Transformer, which uses linearity rather than a Vision Transformer, when extracting spatial attention, thereby allowing for the efficient acquisition of local and global information compared to conventional methods.

[0043] The processor can extract channel attention from input data (10) divided into multiple patches using a channel stream. At this time, the channel attention may include context information related to the components. The channel stream increases the number of channels to prevent information loss as the resolution is reduced at each stage of encoding. Accordingly, the channel attention may include the characteristics of the corresponding components and context information between other components. For example, the channel attention may include color, sharpness, intensity, association with other components, etc. As a specific example, the channel attention may include the context of an image that can be recognized at the computer level, such as a context focused on the liver, or a context regarding the liver and pancreas.

[0044] Specifically, the processor can extract a first channel attention based on a first size patch generated by input data (10) and the number of channels of a first value. For example, the processor can extract a first channel attention based on 32 channels for a patch having a height, width, and depth of 4, 4, and 2, respectively.

[0045] Additionally, the processor can extract second channel attention based on the number of channels of a second size patch and a second value generated by the input data (10). At this time, the second size may be larger than the first size, and the first value may be smaller than the second value. For example, the processor can extract second channel attention based on 64 channels for a patch having a height, width, and depth of 8, 8, and 4, respectively.

[0046] As the processor performs multiple extraction processes, the patch size increases and spatial attention is extracted using channels with large values; thus, a larger amount of information can be acquired as the extraction process progresses. For example, based on a patch of size 1 and a channel size of value 1, the processor can extract first spatial attention using a Swin Transformer and extract first channel attention in parallel using a C-MSA (Channel-Specific Attention Module). However, this is not limited to this, and variations are also possible when extracting channel attention, such as using a Swin Transformer or extracting first channel attention after extracting first spatial attention.

[0047] The merging process (200) may include a step of merging spatial attention extracted from a spatial stream and channel attention extracted from a channel stream. Specifically, the merging process (200) may include a top-down merging process (210) that merges sequentially according to the order of multiple extraction processes and a bottom-up merging process (220) that merges in the order opposite to the order of multiple extraction processes. By performing two mergings using the top-down merging process (210) and the bottom-up merging process (220), the processor can fuse local information from the top side and global information from the bottom side to obtain a large amount of information.

[0048] The merging process (200) may be a step (S200) for generating merged data. The step (S200) for generating merged data may include a top-down merging process (210) and a bottom-up merging process (220). The top-down merging process (210) may be a process of merging spatial attention and channel attention in the order of the step in which spatial attention and channel attention were extracted. The bottom-up merging process (220) may be a process of merging spatial attention and channel attention in the reverse order of the step in which spatial attention and channel attention were extracted. The merged data may include top-down data and bottom-up data generated in the top-down merging process (210) and the bottom-up merging process (220).

[0049] Before performing the top-down merging process (210) and the bottom-up merging process (220), a step of generating fusion data by merging spatial attention and channel attention may be performed first. For example, the processor may generate first fusion data by merging the first spatial attention and the first channel attention, and generate second fusion data by merging the second spatial attention and the second channel attention. Additionally, the processor may generate third fusion data by merging the third spatial attention and the third channel attention. At this time, the third fusion data may be data generated based on the third spatial attention and the third channel attention. Also, at this time, the third spatial attention and the third channel attention may be data extracted by increasing the patch size and the number of channels after the second spatial attention and the second channel attention, respectively.

[0050] The generated fusion data can be merged with each other in the top-down merging process (210) and the bottom-up merging process (220). However, it is not limited to this, and spatial attention and channel attention may be merged directly in the merging process (200) without generating fusion data. Below, the merging process (200) using fusion data is described in detail.

[0051] The top-down merging process (210) and / or bottom-up merging process (220) may include a process for adjusting patches of different sizes (expanding or merging) in the process of extracting spatial attention and channel attention, a fusion method called Fnf (Fast normalized fusion) which is a weighted sum of a learnable scalar and a feature map and then divided by the sum of scalars, and a convolution method (DSConv).

[0052] The processor can generate first bottom-up data by merging second fusion data and third fusion data in the bottom-up merging process (220). Subsequently, the processor continues to generate next-stage bottom-up data by merging fusion data and previous-stage bottom-up data. For example, the processor can generate second bottom-up data by merging first bottom-up data and first fusion data.

[0053] A top-down merging process (210) may be performed after a bottom-up merging process (220). FIG. 2 illustrates an example in which a top-down merging process (210) is performed after a bottom-up merging process (220), but is not limited thereto, and a bottom-up merging process (220) may also be performed after a top-down merging process (210). In the top-down merging process (210), the processor may generate first top-down data by merging first bottom-up data, second bottom-up data, and second fusion data. Subsequently, the processor continues to merge bottom-up data, fusion data, and top-down data from the previous stage to generate top-down data for the next stage. For example, the processor may generate second top-down data by merging first top-down data and third fusion data.

[0054] FIG. 3 is a diagram illustrating a method for extracting spatial attention and channel attention according to one embodiment.

[0055] Referring to Fig. 3, the process of extracting spatial attention using a Swin Transformer in the Swin Block and the process of extracting channel attention in the C-MSA Block can be seen. The Swin Block and the C-MSA module can each be merged N times (e.g., 3 times). As N increases, the degree of merging increases, and a large amount of information can be generated. For example, the merging can use the DSConv method, which is an efficient convolution method that performs Depthwise Convolution followed by Pointwise Convolution, but is not limited thereto.

[0056] Referring again to FIG. 2, the processor may perform a decoding process (300) after the encoding process (100) and the merging process (200). The decoding process (300) may be a step (S300) of generating result data. Specifically, the step (S300) of obtaining result data may include a step of generating first decoding data based on a patch of a third size and a channel number of a third value generated by the merging data (specifically, the second top-down data). For example, the processor may decode (patch expanding) the merging data based on 128 channels for a patch having a height, width, and depth of 16, 16, and 8, respectively.

[0057] Additionally, the step of obtaining result data (S300) may include the step of generating second decoding data based on a fourth size patch generated by the merging data (specifically, the first top-down data) and the number of channels of the fourth value. For example, the processor may decode the merging data based on 64 channels for a patch having a height, width, and depth of 8, 8, and 4, respectively. The processor may output result data (20) by performing decoding based on the patch size and the number of channels that decrease as the decoding process progresses. At this time, the step of generating the first decoding data and the step of generating the second decoding data may be steps using a swin transformer, but are not limited thereto.

[0058] Additionally, the step of obtaining result data (S300) may include the step of setting a first index corresponding to a first component in a first pixel based on the first decoding data and the second decoding data. Additionally, the step of obtaining result data (S300) may include the step of setting a second index corresponding to a second component in a second pixel based on the first decoding data and the second decoding data.

[0059] For example, the processor may set a first index (e.g., 1) related to the liver if the first pixel included in the result data (20) is related to the liver. Also, for example, the processor may set a second index (e.g., 2) related to the heart if the second pixel included in the result data (20) is related to the heart.

[0060] The processor can set a first region based on pixels with a first index set, and set a second region including pixels with a second index set. For example, the processor can set a first region, which is a region related to the liver, by changing the values ​​of pixels set to 1 using a first color. Also, for example, the processor can set a second region, which is a region related to the liver, by changing the values ​​of pixels set to 2 using a second color. In this way, the processor obtains result data (20), which is the result of decoding, and can perform segmentation by component using it.

[0062] FIGS. 4 to 9 are drawings for explaining the results according to the image segmentation method of the present invention in comparison with the prior art.

[0063] Figure 4 is a table showing Dice, HD95, Params, and FLOPs for each component (organ) using the prior art and the present invention.

[0064] Referring to FIG. 4, it can be seen that the image segmentation method (DS-UNETR) according to the present invention has the best or top-tier accuracy for each component compared to the prior art. Specifically, it can be seen that the image segmentation method according to the present invention has achieved a high score, particularly for the spleen. In addition, as can be seen from Params and FLOPs, it can be seen that the hardware (memory, etc.) occupancy and computational amount are much lower than those of the prior art.

[0065] Figure 5 is a diagram showing the image segmentation results using the prior art and the present invention.

[0066] Referring to Fig. 5, it can be seen that the image segmentation method (DS-UNETR) of the present invention captured the right kidney as shown in the first row. Additionally, it can be seen that the present invention captured the gallbladder located between the livers as shown in the second row. Furthermore, it can be seen that the present invention effectively segmented the stomach overlapping between the livers as shown in the third row, and that there are no overlapping areas with other organs as shown in the fourth row. Accordingly, it can be confirmed that the present invention has high image segmentation performance for the stomach and pancreas.

[0067] Figure 6 is a diagram showing the division result in a 3D form.

[0068] Referring to Fig. 6, it can be seen that the present invention (DS-UNETR) showed the most similar results to Ground Truth compared to other technologies.

[0069] Figure 7 is a table comparing the performance of the conventional and various variations of the present invention.

[0070] Referring to FIG. 7, it can be seen that the present invention and its variations (e.g., those without the merging Bi-ASF) exhibit better performance than the prior art Swin-Unet and Swin UNETR.

[0071] Figure 8 is a table showing the performance of the right ventricle, myocardium, and left ventricle.

[0072] Referring to FIG. 8, it can be seen that the present invention demonstrates good performance with fewer parameters and computational load than other technologies.

[0073] Figure 9 is a comparison graph of the conventional and the present invention regarding the number of parameters and die score.

[0074] Referring to FIG. 9, it can be seen that the present invention has a high dice score while having fewer parameters compared to other technologies.

[0076] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0077] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0078] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 An image segmentation method performed by at least one processor, comprising: a step of extracting spatial attention from input data; a step of extracting channel attention from the input data; a step of generating merging data based on the spatial attention and the channel attention; and a step of obtaining result data including component-specific indices based on the merging data, wherein the step of generating merging data comprises: a step of generating first bottom-up data based on a first spatial attention, a first channel attention, a second spatial attention, and a second channel attention; and a step of generating second bottom-up data based on the first bottom-up data, a third spatial attention, and a third channel attention. Claim 2 An image segmentation method according to claim 1, wherein the spatial attention includes shape information related to the component, and the channel attention includes context information related to the component. Claim 3 An image segmentation method according to claim 1, wherein the step of extracting spatial attention is a step using a swin transformer. Claim 4 An image segmentation method performed by at least one processor, comprising: a step of extracting spatial attention from input data; a step of extracting channel attention from the input data; a step of generating merging data based on the spatial attention and the channel attention; and a step of obtaining result data including component-specific indices based on the merging data, wherein the step of extracting spatial attention comprises: a step of extracting a first spatial attention based on a first size patch and a channel number of a first value generated by the input data; and a step of extracting a second spatial attention based on a second size patch and a channel number of a second value generated by the input data, wherein the second size is larger than the first size and the first value is smaller than the second value. Claim 5 An image segmentation method performed by at least one processor, comprising: a step of extracting spatial attention from input data; a step of extracting channel attention from the input data; a step of generating merging data based on the spatial attention and the channel attention; and a step of obtaining result data including component-specific indices based on the merging data, wherein the step of extracting channel attention comprises: a step of extracting a first channel attention based on a patch of a first size and a number of channels of a first value; and a step of extracting a second channel attention based on a patch of a second size and a number of channels of a second value. Claim 6 delete Claim 7 An image segmentation method according to claim 1, wherein the step of generating the merging data comprises: a step of generating first top-down data based on the first bottom-up data, the second bottom-up data, the second spatial attention, and the second channel attention; and a step of generating second top-down data based on the first top-down data, the first spatial attention, and the first channel attention. Claim 8 An image segmentation method performed by at least one processor, comprising: a step of extracting spatial attention from input data; a step of extracting channel attention from the input data; a step of generating merging data based on the spatial attention and the channel attention; and a step of obtaining result data including a component-specific index based on the merging data, wherein the step of obtaining result data comprises: a step of generating first decoding data based on a third size patch and a channel number of a third value generated by the merging data; and a step of generating second decoding data based on a fourth size patch and a channel number of a fourth value generated by the merging data, wherein the fourth size is smaller than the third size and the third value is larger than the fourth value. Claim 9 An image segmentation method according to claim 8, wherein the step of obtaining the result data comprises: a step of setting a first index corresponding to a first component in a first pixel based on the first decoding data and the second decoding data; a step of setting a second index corresponding to a second component in a second pixel based on the first decoding data and the second decoding data; and a step of setting a first region including a pixel where the first index is set and a second region including a pixel where the second index is set. Claim 10 In claim 8, the steps of generating the first decoding data and generating the second decoding data are steps utilizing a swin transformer in an image segmentation method. Claim 11 A computer program recorded on a computer-readable recording medium to execute an image segmentation method described in any one of paragraphs 1 through 5 and paragraphs 7 through 10. Claim 12 An electronic device comprising: a memory; and at least one processor, wherein, when executed by the at least one processor, the memory stores a spatial attention extracted from input data, stores a channel attention extracted from the input data, stores merging data generated based on the spatial attention and the channel attention, stores result data including a component-specific index obtained based on the merging data, stores a first bottom-up data generated based on a first spatial attention, a first channel attention, a second spatial attention, and a second channel attention, and stores a second bottom-up data generated based on the first bottom-up data, a third spatial attention, and a third channel attention. Claim 13 An image segmentation device comprising: an encoding unit that extracts spatial attention and channel attention from input data; a merging unit that generates merging data based on the spatial attention and the channel attention; and a decoding unit that generates result data including component-specific indices based on the merging data, wherein the merging unit generates first bottom-up data based on a first spatial attention, a first channel attention, a second spatial attention, and a second channel attention, and generates second bottom-up data based on the first bottom-up data, a third spatial attention, and a third channel attention.

Images