Image segmentation method using dual attention, and apparatus using same

Abstract

An image segmentation method of the present invention may comprise the steps of: extracting spatial attention from input data; extracting channel attention from the input data; generating merging data on the basis of the spatial attention and the channel attention; and acquiring, on the basis of the merging data, result data including an index for each constituent element.

Description

Image segmentation method using double attention and device using the same

[0001] The present invention relates to an image segmentation method, and more particularly, to a method for segmenting an image using dual attention.

[0002] Medical image segmentation is a field that identifies lesions such as tumors and segments various organs, enabling quick and easy medical image analysis and accurate diagnosis. In particular, in complex medical images such as 3D CT and MRI, it is crucial to extract useful spatial and channel information from feature information obtained through deep neural network models to accurately distinguish multiple organs.

[0003] Existing U-Net-based deep neural network research, despite increasing the number of channels as model sizes increase, focuses solely on spatial information, limiting its ability to obtain information necessary for accurate medical image segmentation. Therefore, to secure additional contextual information for medical images, a dual-attention technique is needed that extracts and fuses both spatial and channel information.

[0004] The present invention is derived from research conducted as part of the SW Computing Industry Core Technology Development of the Ministry of Science and ICT (Project Unique Number: 1711193130, Project Number: 2017-0-00477-007, Project Management Agency: National IT Industry Planning and Evaluation Institute, Research Project Name: (SW Star Lab) Development of In-Memory Distributed DBMS Based on High-Performance Flash Memory Storage for IoT Environment, Project Performing Agency: Yonsei University Industry-Academic Cooperation Foundation, Research Period: 2023.01.01~2023.12.31).

[0005] Meanwhile, the Korean government, which provided the task, has no property interest in any aspect of the present invention.

[0006] One object of the present invention is to provide an image segmentation method using dual attention.

[0007] According to one embodiment of the present invention, an image segmentation method using dual attention can be provided.

[0008] Figure 1 is a flowchart of an image segmentation method according to one embodiment.

[0009] FIG. 2 is a drawing for explaining an image segmentation method according to one embodiment.

[0010] FIG. 3 is a diagram for explaining a method for merging spatial attention and channel attention according to one embodiment.

[0011] Figures 4 to 9 are drawings for explaining the results according to the image segmentation method of the present invention in comparison with the prior art.

[0012] An image segmentation method according to one embodiment may include a step of extracting spatial attention from input data; a step of extracting channel attention from the input data; a step of generating merging data based on the spatial attention and the channel attention; and a step of obtaining result data including an index for each component based on the merging data.

[0013] Here, the spatial attention may include shape information related to the component, and the channel attention may include contextual information related to the component.

[0014] Here, the step of extracting the spatial attention may be a step using a swin transformer.

[0015] Here, the step of extracting the spatial attention includes the step of extracting the first spatial attention based on the number of channels of the first size patch and the first value generated by the input data; and the step of extracting the second spatial attention based on the number of channels of the second size patch and the second value generated by the input data, wherein the second size may be larger than the first size, and the first value may be smaller than the second value.

[0016] Here, the step of extracting the channel attention may include a step of extracting the first channel attention based on the number of channels of the first size patch and the first value; and a step of extracting the second channel attention based on the number of channels of the second size patch and the second value.

[0017] Here, the step of generating the merging data may include a step of generating first bottom-up data based on the first spatial attention, the first channel attention, the second spatial attention, and the second channel attention; and a step of generating second bottom-up data based on the first bottom-up data, the third spatial attention, and the third channel attention.

[0018] Here, the step of generating the merging data may include a step of generating first top-down data based on the first bottom-up data, the second bottom-up data, the second spatial attention, and the second channel attention; and a step of generating second top-down data based on the first top-down data, the first spatial attention, and the first channel attention.

[0019] Here, the step of obtaining the result data includes the step of generating first decoding data based on the number of channels of the third size patch and the third value generated by the merging data; and the step of generating second decoding data based on the number of channels of the fourth size patch and the fourth value generated by the merging data, wherein the fourth size may be smaller than the third size, and the third value may be larger than the fourth value.

[0020] Here, the step of obtaining the result data may include: setting a first index corresponding to a first component in a first pixel based on the first decoding data and the second decoding data; setting a second index corresponding to a second component in a second pixel based on the first decoding data and the second decoding data; and setting a first area including a pixel to which the first index is set and a second area including a pixel to which the second index is set.

[0021] Here, the step of generating the first decoding data and the step of generating the second decoding data may be steps using a swin transformer.

[0022] Here, a computer program recorded on a computer-readable recording medium may be provided to execute the above image segmentation method.

[0023]

[0024] An electronic device according to one embodiment includes a memory; and at least one processor, wherein the memory, when executed by the at least one processor, causes the electronic device to store spatial attention extracted from input data, store channel attention extracted from the input data, store merging data generated based on the spatial attention and the channel attention, and store result data including component-specific indices obtained based on the merging data.

[0025]

[0026] An image segmentation device according to one embodiment may include an encoding unit that extracts spatial attention and channel attention from input data; a merging unit that generates merging data based on the spatial attention and the channel attention; and a decoding unit that generates result data including an index for each component based on the merging data.

[0027] Since the embodiments described in this specification are intended to clearly explain the idea of ​​the present invention to a person having ordinary skill in the art to which the present invention pertains, the present invention is not limited to the embodiments described in this specification, and the scope of the present invention should be interpreted to include modified or altered examples that do not depart from the idea of ​​the present invention.

[0028] The terms used in this specification have been selected from widely used terms, taking into account their functions in the present invention. However, these terms may vary depending on the intentions of those skilled in the art, precedents, or the emergence of new technologies in the technical field to which the present invention pertains. However, if a specific term is defined and used with an arbitrary meaning, the meaning of that term will be described separately. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the term and the overall content of this specification, rather than simply the name of the term.

[0029] The drawings attached to this specification are intended to facilitate explanation of the present invention, and the shapes depicted in the drawings may be exaggerated as necessary to help understand the present invention, and therefore the present invention is not limited by the drawings.

[0030] In cases where it is determined that a specific description of the composition or function of a public notice related to the present invention in this specification may obscure the gist of the present invention, a detailed description thereof will be omitted as necessary.

[0031]

[0032] The present invention relates to a method and device for segmenting an image using extracted dual attention. Specifically, the present invention relates to an electronic device and an image segmentation device capable of extracting spatial attention and channel attention from input data. The electronic device or the image segmentation device may include at least one processor, which may be a computing device and includes a memory and a processor capable of executing the memory. The image segmentation method described below may be interpreted as being performed by at least one processor included in the electronic device or the image segmentation device.

[0033]

[0034] Figure 1 is a flowchart of an image segmentation method according to one embodiment.

[0035] Referring to FIG. 1, an image segmentation method according to one embodiment may include a step of extracting spatial attention and channel attention (S100), a step of generating merging data (S200), and a step of generating result data (S300). Although FIG. 1 illustrates that steps S100 to S300 are performed sequentially, this is not limited to the sequential execution of steps, and new steps may be added or some steps may be merged and performed simultaneously.

[0036] An electronic device or an image segmentation device may include an encoding unit, a merging unit, and a decoding unit. The encoding unit may extract spatial attention and channel attention from input data. The merging unit may generate merged data based on the spatial attention and channel attention extracted by the encoding unit. Specifically, the merging unit may generate merged data by merging spatial attention and channel attention. The decoding unit may generate result data including an index for each component based on the merged data generated by the merging unit. The encoding unit may be the subject of step S100, the merging unit may be the subject of step S200, and the decoding unit may be the subject of step S300.

[0037] Hereinafter, the image segmentation method will be specifically described with reference to FIGS. 2 and 3.

[0038]

[0039] FIG. 2 is a drawing for explaining an image segmentation method according to one embodiment.

[0040] Referring to FIG. 2, an image segmentation method according to one embodiment may include an encoding process (100), a merging process (200), and a decoding process (300) using input data (10). The encoding process (100) may include a process (110) of extracting spatial attention from the input data (10) and a process (120) of extracting channel attention from the input data (10). The merging process (200) may include a top-down merging process (210) and a bottom-up merging process (220).

[0041] The encoding process (100) may include a step (S100) of extracting spatial attention and channel attention from input data (10). Before extracting the spatial attention and channel attention, the processor may perform a process of dividing the input data (10) into a plurality of patches. At this time, the size of the patches and the number of channels related to information extracted from the patches may vary for each extraction process.

[0042] The processor can extract spatial attention from input data (10) divided into a plurality of patches using a spatial stream (e.g., a spatial attention extraction module). At this time, the spatial attention can include shape information or location information related to a component (e.g., an organ such as a stomach or a liver). In addition, the spatial attention can also include context information included in the channel attention. Specifically, the processor can extract the first spatial attention based on the number of channels of a first size patch and a first value generated by the input data (10). For example, the processor can extract the first spatial attention based on 32 channels for a patch having a height, a width, and a depth of 4, 4, and 2, respectively.

[0043] Additionally, the processor may extract second spatial attention based on the number of channels of the second size patch and the second value generated by the input data (10). At this time, the second size may be larger than the first size, and the first value may be smaller than the second value. For example, the processor may extract second spatial attention based on 64 channels for a patch having a height, width, and depth of 8, 8, and 4, respectively. In this way, the processor may have multiple extraction processes, and may extract spatial attention based on patches of different sizes and different numbers of channels for each process.

[0044] As the processor performs multiple extraction steps, the patch size increases and spatial attention is extracted using channels with larger values. This allows for the acquisition of greater amounts of information as the extraction process progresses. Spatial features are compressed as multiple extraction steps are performed, allowing the processor to gradually acquire global information from local features. To compensate for the loss of information due to spatial compression, channel information is increased to acquire diverse global contextual information about the feature map.

[0045] Additionally, the method of increasing the channel can be performed by shifting some of the spatial information in the channel direction before compression, and then linearly transforming the channel features. Thus, each channel information can have arbitrary spatial information. For example, a first channel may have information about a portion of the liver, and a second channel may have information about a portion of the heart. The processor can extract channel attention to focus on information related to components (e.g., organs such as the liver or heart) among the channel information. Channel attention will be described later.

[0046] When extracting spatial attention, the processor can utilize a Swin transformer. That is, the step of extracting spatial attention may involve utilizing a Swin transformer. A Swin transformer is a model that hierarchically stacks transformer encoders and can perform attention between windows using a shifted window algorithm.

[0047] The image segmentation method of the present invention requires a significant amount of computation because it performs multiple extraction processes while reducing the patch size and increasing the number of channels. Therefore, the present invention utilizes a Swin transformer, which utilizes linearity rather than a vision transformer, to extract spatial attention, thereby reducing computational complexity and efficiently acquiring local and global information compared to conventional methods.

[0048] The processor can extract channel attention from input data (10) divided into multiple patches using a channel stream. At this time, the channel attention may include context information related to the component. The channel stream increases the number of channels to prevent information loss as the resolution is reduced at each stage of encoding. Accordingly, the channel attention may include characteristics of the corresponding component information and context information between other components. For example, the channel attention may include color, sharpness, intensity, and correlations with other components. For example, the channel attention may include context of an image that can be recognized at a computer level, such as a context focused on the liver or a context related to the liver and pancreas.

[0049] Specifically, the processor can extract first channel attention based on the number of channels of the first size patch and the first value generated by the input data (10). For example, the processor can extract first channel attention based on 32 channels for a patch having a height, width, and depth of 4, 4, and 2, respectively.

[0050] Additionally, the processor may extract second channel attention based on the number of channels of a second size patch and a second value generated by the input data (10). In this case, the second size may be larger than the first size, and the first value may be smaller than the second value. For example, the processor may extract second channel attention based on 64 channels for a patch having a height, width, and depth of 8, 8, and 4, respectively.

[0051] As the processor performs multiple extraction processes, the size of the patch increases and the spatial attention is extracted using a channel with a large value. Therefore, as the extraction process progresses, a greater amount of information can be obtained. For example, based on a patch of the first size and a channel size of the first value, the processor can extract the first spatial attention using a SW transformer and extract the first channel attention in parallel using a C-MSA (channel-specific attention module). However, this is not limited to this, and variations such as using a SW transformer when extracting channel attention or extracting the first spatial attention first and then extracting the first channel attention are also possible.

[0052] The merging process (200) may include a step of merging spatial attention extracted from a spatial stream and channel attention extracted from a channel stream. Specifically, the merging process (200) may include a top-down merging process (210) that sequentially merges in the order of a plurality of extraction processes and a bottom-up merging process (220) that merges in the order opposite to the order of the plurality of extraction processes. By performing two merges using the top-down merging process (210) and the bottom-up merging process (220), the processor can acquire a lot of information by fusing local information at the top and global information at the bottom.

[0053] The merging process (200) may be a step (S200) of generating merging data. The step (S200) of generating merging data may include a top-down merging process (210) and a bottom-up merging process (220). The top-down merging process (210) may be a process of merging spatial attention and channel attention in the order of the steps of extracting spatial attention and channel attention. The bottom-up merging process (220) may be a process of merging spatial attention and channel attention in the reverse order of the steps of extracting spatial attention and channel attention. The merging data may include top-down data and bottom-up data generated in the top-down merging process (210) and the bottom-up merging process (220).

[0054] Before performing the top-down merging process (210) and the bottom-up merging process (220), a step of generating fusion data by merging spatial attention and channel attention may be performed first. For example, the processor may generate first fusion data by merging first spatial attention and first channel attention, and generate second fusion data by merging second spatial attention and second channel attention. In addition, the processor may generate third fusion data by merging third spatial attention and third channel attention. In this case, the third fusion data may be data generated based on the third spatial attention and the third channel attention. In addition, in this case, the third spatial attention and the third channel attention may be data extracted by increasing the size of the patch and increasing the number of channels after the second spatial attention and the second channel attention, respectively.

[0055] The generated fusion data can be merged in a top-down merging process (210) and a bottom-up merging process (220). However, this is not limited to this, and spatial attention and channel attention can be merged directly in the merging process (200) without generating fusion data. The merging process (200) using fusion data is described in detail below.

[0056] The top-down merging process (210) and / or the bottom-up merging process (220) may include a process for adjusting patches of different sizes (expanding or merging) in the process of extracting spatial attention and channel attention, a fast normalized fusion (FnF) process, which is a fusion method that weights a learnable scalar and a feature map and then divides it by the sum of the scalars, and a convolutional method (DSConv).

[0057] The processor can merge the second fusion data and the third fusion data in the bottom-up merging process (220) to generate the first bottom-up data. Thereafter, the processor continuously merges the fusion data with the bottom-up data of the previous step to generate the bottom-up data of the next step. For example, the processor can merge the first bottom-up data and the first fusion data to generate the second bottom-up data.

[0058] A top-down merging process (210) may be performed after a bottom-up merging process (220). FIG. 2 illustrates an example in which a top-down merging process (210) is performed after a bottom-up merging process (220), but the present invention is not limited thereto, and the bottom-up merging process (220) may be performed after the top-down merging process (210). The processor may merge first bottom-up data, second bottom-up data, and second fusion data in the top-down merging process (210) to generate first top-down data. Thereafter, the processor continuously merges bottom-up data, fusion data, and top-down data of the previous stage to generate top-down data of the next stage. For example, the processor may merge first top-down data and third fusion data to generate second top-down data.

[0059] FIG. 3 is a diagram for explaining a method for extracting spatial attention and channel attention according to one embodiment.

[0060] Referring to Fig. 3, the process of extracting spatial attention using a Swin transformer in a Swin block and the process of extracting channel attention in a C-MSA block can be seen. The Swin block and the C-MSA module can each be merged N times (e.g., 3 times). As N increases, the degree of merging increases, which can generate a large amount of information. For example, merging can use the DSConv method, which is an efficient convolution method that performs pointwise convolution after depthwise convolution, but is not limited thereto.

[0061] Referring back to FIG. 2, the processor may perform a decoding process (300) after the encoding process (100) and the merging process (200). The decoding process (300) may be a step (S300) of generating result data. Specifically, the step (S300) of obtaining the result data may include a step of generating first decoded data based on a third-sized patch generated by the merging data (specifically, the second top-down data) and the number of channels of the third value. For example, the processor may decode (patch expand) the merging data based on 128 channels for a patch having a height, width, and depth of 16, 16, and 8, respectively.

[0062] In addition, the step (S300) of obtaining the result data may include a step of generating second decoding data based on the number of channels of the fourth size patch and the fourth value generated by the merging data (specifically, the first top-down data). For example, the processor may decode the merging data based on 64 channels for patches having sizes of 8, 8, and 4 in height, width, and depth, respectively. The processor may perform decoding based on the size of the patch and the number of channels that decrease as the decoding process progresses, and output the result data (20). At this time, the step of generating the first decoding data and the step of generating the second decoding data may be steps using a SW transformer, but are not limited thereto.

[0063] In addition, the step of obtaining result data (S300) may include a step of setting a first index corresponding to a first component in a first pixel based on the first decoding data and the second decoding data. In addition, the step of obtaining result data (S300) may include a step of setting a second index corresponding to a second component in a second pixel based on the first decoding data and the second decoding data.

[0064] For example, if a first pixel included in the result data (20) is related to the liver, the processor may set a first index (e.g., 1) related to the liver. Also, for example, if a second pixel included in the result data (20) is related to the heart, the processor may set a second index (e.g., 2) related to the heart.

[0065] The processor can set a first region based on pixels having a first index set, and can set a second region including pixels having a second index set. For example, the processor can set a first region, which is an region related to the liver, using a first color by changing the values ​​of pixels having a first index set to 1. Also, for example, the processor can set a second region, which is an region related to the liver, using a second color by changing the values ​​of pixels having a second index set to 2. In this way, the processor can obtain result data (20), which is a result of decoding, and perform segmentation by component using the same.

[0066]

[0067] Figures 4 to 9 are drawings for explaining the results according to the image segmentation method of the present invention in comparison with the prior art.

[0068] Figure 4 is a table showing Dice, HD95, Params, and FLOPs for each component (organ) using the prior art and the present invention.

[0069] Referring to Figure 4, the image segmentation method (DS-UNETR) according to the present invention demonstrates the best or highest component-specific accuracy compared to prior art. Specifically, the image segmentation method according to the present invention achieves particularly high scores for the spleen. Furthermore, as can be seen from the Params and FLOPs, the hardware (e.g., memory) footprint and computational load are significantly lower than those of prior art.

[0070] Figure 5 is a drawing showing the results of image segmentation using conventional technology and the present invention.

[0071] Referring to FIG. 5, it can be confirmed that the image segmentation method (DS-UNETR) of the present invention captured the right kidney, as shown in the first row. In addition, it can be confirmed that the present invention captured the gallbladder located between the liver, as shown in the second row. In addition, the present invention successfully segmented the stomach overlapping the liver, as shown in the third row, and it can be confirmed that there is no area overlapping with other organs, as shown in the fourth row. Accordingly, it can be confirmed that the present invention has high image segmentation performance for the stomach and pancreas.

[0072] Figure 6 is a drawing showing the segmentation result in 3D form.

[0073] Referring to Figure 6, it can be seen that the present invention (DS-UNETR) shows results most similar to Ground Truth compared to other technologies.

[0074] Figure 7 is a table comparing the performance of various modifications of the prior art and the present invention.

[0075] Referring to FIG. 7, it can be confirmed that the present invention and its modifications (e.g., without the merging part Bi-ASF, etc.) show better performance than the prior art Swin-Unet and Swin UNETR.

[0076] Figure 8 is a table showing the performance for the right ventricle, myocardium, and left ventricle.

[0077] Referring to FIG. 8, it can be confirmed that the present invention shows good performance with a smaller number of parameters and computational amount than other technologies.

[0078] Figure 9 is a comparative graph of the conventional and the present invention in terms of the number of parameters and dice scores.

[0079] Referring to FIG. 9, it can be confirmed that the present invention has a high dice score while having fewer parameters compared to other technologies.

[0080]

[0081] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0082] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0083] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In an image segmentation method performed by at least one processor, A step of extracting spatial attention from input data; A step of extracting channel attention from the above input data; A step of generating merging data based on the spatial attention and the channel attention; and A step of obtaining result data including a component-specific index based on the above merging data. Image segmentation method.

2. In paragraph 1, The above spatial attention includes shape information related to the components, The above channel attention contains contextual information related to the component. Image segmentation method.

3. In paragraph 1, The step of extracting the above spatial attention is a step using a swin transformer. Image segmentation method.

4. In paragraph 1, The step of extracting the above spatial attention is: A step of extracting a first spatial attention based on a first size patch generated by the input data and a number of channels of a first value; and A step of extracting second spatial attention based on the number of channels of the second size patch and the second value generated by the above input data, The second size is greater than the first size, and the first value is less than the second value. Image segmentation method.

5. In paragraph 1, The step of extracting the above channel attention is: A step of extracting first channel attention based on a first size patch and a number of channels of a first value; and A step of extracting second channel attention based on a second size patch and a second value channel number. Image segmentation method.

6. In paragraph 1, The step of generating the above merging data is: A step of generating first bottom-up data based on first spatial attention, first channel attention, second spatial attention, and second channel attention; and A step of generating second bottom-up data based on the first bottom-up data, third spatial attention, and third channel attention. Image segmentation method.

7. In paragraph 6, The step of generating the above merging data is: A step of generating first top-down data based on the first bottom-up data, the second bottom-up data, the second spatial attention, and the second channel attention; and A step of generating second top-down data based on the first top-down data, the first spatial attention, and the first channel attention. Image segmentation method.

8. In paragraph 1, The step of obtaining the above result data is: A step of generating first decoding data based on the number of channels of a third size patch and a third value generated by the above merging data; and A step of generating second decoding data based on the number of channels of the fourth size patch and the fourth value generated by the above merging data, The fourth size is smaller than the third size, and the third value is larger than the fourth value. Image segmentation method.

9. In paragraph 8, The step of obtaining the above result data is: A step of setting a first index corresponding to a first component in a first pixel based on the first decoding data and the second decoding data; A step of setting a second index corresponding to a second component in a second pixel based on the first decoding data and the second decoding data; and A step of setting a first area including a pixel for which the first index is set and a second area including a pixel for which the second index is set. Image segmentation method.

10. In paragraph 8, The step of generating the first decoding data and the step of generating the second decoding data are steps using a swin transformer. Image segmentation method.

11. A non-transitory computer-readable recording medium having recorded thereon a program for executing the image segmentation method described in paragraph 1.

12. In electronic devices, memory; and Contains at least one processor, The above memory, when executed by the at least one processor, causes the electronic device to: Store the spatial attention extracted from the input data, Store the channel attention extracted from the above input data, Store the merging data generated based on the above spatial attention and the above channel attention, Storing result data including component-specific indexes obtained based on the above merging data Electronic devices.

13. An encoding unit that extracts spatial attention and channel attention from input data; A merging unit that generates merging data based on the spatial attention and the channel attention; and A decoding unit that generates result data including an index for each component based on the above merging data. Image segmentation device.

Images