Vision transformer-based apparatus and method for diagnosing medical image

Abstract

The present invention relates to a vision transformer-based apparatus and method for diagnosing a medical image, and the apparatus comprises: a medical image receiving unit for receiving the original medical image; a convolutional neural network processing unit that generates a local feature map through extraction of local information based on convolution-based shift window attention for the original medical image, and a global feature map through extraction of global information based on distortion attention; and a medical image transformer unit that performs a segmentation task and a classification task for the original medical image through at least one of the local feature map and the global feature map.

Description

Vision Transformer-based Medical Imaging Diagnostic Device and Method

[0001] The present invention relates to medical imaging diagnostic technology, and more specifically, to a vision transformer-based medical imaging diagnostic device and method capable of improving diagnostic performance by utilizing the induced bias of a transformer structure for limited medical images.

[0002]

[0003] In the medical field, research utilizing artificial intelligence (AI) for medical imaging diagnosis is actively underway. The use of AI in medicine and healthcare offers the benefits of improving diagnostic accuracy and accelerating the decision-making process for medical professionals. Recent medical imaging incorporates various visual modalities, such as X-rays, magnetic resonance imaging (MRI), and computed tomography (CT). Consequently, integrating AI into medicine can potentially improve cost-efficiency by reducing diagnostic inaccuracies among medical experts and streamlining diagnostic procedures.

[0004] In machine learning for medical imaging, Convolutional Neural Network (CNN) architectures have been primarily used for disease diagnosis. This is a significant advantage given the scarcity of data in the medical field, as it offers relatively excellent generalization performance even with limited data. CNNs extract image features using convolutional filters that slide across the image. Here, CNNs assume locality and stationarity, which posits that adjacent pixels within an image are related and that these relationships are consistently maintained throughout the image. Consequently, CNNs are prone to performance degradation when objects are occluded or when locality conditions are not met. On the other hand, in general computer vision, recent research is advancing toward using Vision Transformer (ViT) models to utilize attention-based mechanisms that focus on important parts of an image. While ViT models can improve performance by capturing global context through attention operations on the entire image, they have the disadvantage of requiring more training time and data compared to CNNs that utilize locality and stationarity.

[0005]

[0006] [Prior Art Literature]

[0007] [Patent Literature]

[0008] Korean Patent Publication No. 10-2023-0044123 (April 3, 2023)

[0009]

[0010] One embodiment of the present invention aims to provide a vision transformer-based medical imaging diagnostic device and method capable of improving diagnostic performance by utilizing the induced bias of a transformer structure for limited medical images.

[0011] One embodiment of the present invention aims to provide a vision transformer-based medical imaging diagnostic device and method that can effectively process a wider variety of visual elements in medical imaging diagnostics by fusing shift window attention, distortion attention, and convolution block attention modules to facilitate the induced bias of the vision transformer model.

[0012]

[0013] Among the embodiments, a vision transformer-based medical imaging diagnostic device comprises: a medical image receiving unit that receives an original medical image; a convolutional neural network processing unit that generates a local feature map through the extraction of local information based on convolutional shift window attention and a global feature map through the extraction of global information based on distortion attention for the original medical image; and a medical image transformer unit that performs a segmentation task and a classification task for the original medical image through at least one of the local feature map and the global feature map.

[0014] The above convolutional neural network processing unit can generate a preprocessed map as a first convolutional layer through preprocessing operations on the original medical image prior to the generation of the local feature map.

[0015] The above convolutional neural network processing unit may include a stem block module that outputs the first convolutional layer by performing patch-wise encoding on the original medical image using a plurality of 2D convolutional layers (Conv2d) as the preprocessing operation.

[0016] The above convolutional neural network processing unit includes a shift window attention block (SWA Block) module, and the shift window attention block module includes a first convolutional block attention module layer (CBAM). For a first input feature map derived based on the original medical image, object presence detection through channel-unit attention of the first convolutional block attention module layer (CBAM) and object location detection through spatial-unit attention are performed, and the local feature map can be output as a second convolutional layer.

[0017] The above shift window attention block module further includes a first local attention layer (LA) and a shift window attention layer (SWA), and can output the first input feature map by performing window-unit division of the first local attention layer (LA) and division window-unit shift of the shift window attention layer (SWA) on the first convolution layer of the original medical image.

[0018] The above convolutional neural network processing unit can increase the feature dimension by repeatedly connecting the above shift window attention block modules in series.

[0019] The above convolutional neural network processing unit includes a distortion attention block (DA Block) module, and the distortion attention block (DA Block) module includes a second convolutional block attention module layer (CBAM). For a second input feature map derived based on the local feature map, object presence detection through channel-unit attention of the second convolutional block attention module layer (CBAM) and object location detection through spatial-unit attention can be performed to output the global feature map as a third convolutional layer.

[0020] The above distortion attention block (DA Block) module further includes a second local attention layer (LA) and a distortion attention layer (DA), and can output the second input feature map by performing window-unit division of the second local attention layer (LA), sampling of reference points in the division window unit of the distortion attention layer (DA), and offset of the sampling points for the second convolution layer of the local feature map.

[0021] The above convolutional neural network processing unit can increase the feature dimension up to the target feature dimension by repeatedly connecting the above distortion attention block (DA Block) modules in series.

[0022] The medical image transformer unit can fuse the local feature map and the global feature map to generate a fused feature map and perform the segmentation task on the fused feature map.

[0023] The medical image transformer unit can fuse N (where N is a natural number) local feature maps output according to the iterative connection of the shift window attention block module and M (where M is a natural number) global feature maps output according to the iterative connection of the distortion attention block (DA Block) module.

[0024] The medical image transformer unit can perform the classification task by processing Global Average Pooling (GAP) on the global feature map.

[0025] Among the embodiments, a vision transformer-based medical image diagnostic method is a vision transformer-based medical image diagnostic method performed in a medical image diagnostic device, comprising: a medical image receiving step of receiving an original medical image; a convolutional neural network processing step of generating a local feature map through the extraction of local information based on convolutional shift window attention and a global feature map through the extraction of global information based on distortion attention for the original medical image; and a medical image transformer step of performing a segmentation task and a classification task for the original medical image through at least one of the local feature map and the global feature map.

[0026] The above convolutional neural network processing step may include a step of generating a preprocessed map as a first convolutional layer through preprocessing operations on the original medical image prior to the generation of the local feature map.

[0027] The above convolutional neural network processing step may include a step of detecting the presence of an object through channel-unit attention and detecting the location of an object through spatial-unit attention of a first convolutional block attention module layer (CBAM) on a first input feature map derived based on the original medical image, and outputting the local feature map as a second convolutional layer.

[0028] The above convolutional neural network processing step may include a step of detecting the presence of an object through channel-unit attention and detecting the location of an object through spatial-unit attention of a second convolutional block attention module layer (CBAM) on a second input feature map derived based on the local feature map, and outputting the global feature map as a third convolutional layer.

[0029] The medical image transformer step may include the step of fusing the local feature map and the global feature map to generate a fused feature map and performing the segmentation task on the fused feature map.

[0030] The medical image transformer step may include a step of performing the classification task by processing Global Average Pooling (GAP) on the global feature map.

[0031]

[0032] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0033] A vision transformer-based medical imaging diagnostic device and method according to one embodiment of the present invention can improve diagnostic performance by utilizing the induced bias of a transformer structure for limited medical images.

[0034] A vision transformer-based medical imaging diagnostic device and method according to one embodiment of the present invention can effectively process a wider variety of visual elements in medical imaging diagnostics by fusing shift window attention, distortion attention, and convolution block attention modules to facilitate the induced bias of the vision transformer model.

[0035]

[0036] FIG. 1 is a drawing illustrating a medical imaging diagnostic system according to the present invention.

[0037] Figure 2 is a diagram illustrating the system configuration of the medical imaging diagnostic device of Figure 1.

[0038] Figure 3 is a diagram illustrating the functional configuration of the medical imaging diagnostic device of Figure 1.

[0039] FIG. 4 is a diagram illustrating a convolutional neural network architecture according to the present invention.

[0040] FIG. 5 is a flowchart illustrating a vision transformer-based medical imaging diagnostic method according to the present invention.

[0041] Figure 6 is a diagram illustrating an example of a layer configuration in the convolutional neural network architecture of Figure 4.

[0042] FIG. 7 is a diagram illustrating an example of a convolution block attention module layer (CBAM) in the convolution neural network architecture of FIG. 4.

[0043] FIGS. 8 to 10 are drawings illustrating experimental results regarding a medical imaging diagnostic method according to the present invention.

[0044]

[0045] The description of the present invention is merely an example for structural or functional explanation, and therefore the scope of the present invention should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific example must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0046] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0047] Terms such as "first," "second," etc., are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0048] When it is stated that one component is "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when it is stated that one component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0049] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the implemented features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0050] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps; the steps may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0051] The present invention may be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0052] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0053]

[0054] FIG. 1 is a drawing illustrating a medical imaging diagnostic system according to the present invention.

[0055] Referring to FIG. 1, the medical imaging diagnostic system (100) may include a user terminal (110), a medical imaging diagnostic device (130), and a database (150).

[0056] The user terminal (110) may correspond to a computing device capable of transmitting data or utilizing various services using a network, and may be implemented as a smartphone, laptop, or computer, but is not necessarily limited thereto and may also be implemented as various devices such as a tablet PC. For example, the user terminal (110) may correspond to a terminal device capable of collecting or capturing original medical images from a user for the performance of the vision transformer-based medical image diagnosis method according to the present invention. In this case, the user terminal (110) may be implemented including a medical image capturing device or may operate in conjunction with a medical database.

[0057] Additionally, the user terminal (110) can be implemented as one device constituting the medical imaging diagnostic system (100) according to the present invention, and the medical imaging diagnostic system (100) can be implemented in various modified forms depending on the purpose of medical imaging diagnostic based on a vision transformer.

[0058] A user terminal (110) can be connected to a medical imaging diagnostic device (130) via a network, and multiple user terminals (110) can be connected to the medical imaging diagnostic device (130) simultaneously. The user terminal (110) can install and run a dedicated program or application for interoperability with the medical imaging diagnostic device (130).

[0059] The medical imaging diagnostic device (130) may be implemented as a computer or server that performs a vision transformer-based medical imaging diagnostic method according to the present invention. Here, the vision transformer-based medical imaging diagnostic method according to the present invention may correspond to a series of processes that construct a diagnostic model for medical imaging diagnosis using a publicly available medical imaging dataset, and perform segmentation and classification operations on the original medical image using the constructed diagnostic model.

[0060] Additionally, the medical imaging diagnostic device (130) can be connected to a user terminal (110) via a wired network or a wireless network such as Bluetooth, WiFi, LTE, etc., and can transmit and receive data with the user terminal (110) through the network. Additionally, the medical imaging diagnostic device (130) can be implemented to operate in connection with an independent external system (not shown in FIG. 1). For example, the medical imaging diagnostic device (130) can operate in conjunction with a data collection system that collects original medical images from medical equipment, an artificial intelligence system that performs learning and verification of an artificial intelligence model, and a service server that provides services related to medical imaging diagnosis.

[0061] The database (150) may correspond to a storage device that stores various information required during the operation of the medical imaging diagnostic device (130). The database (150) may store a dataset regarding medical imaging data, and may store algorithms and software for model learning and verification, but is not necessarily limited thereto, and may store information collected or processed in various forms during the process in which the medical imaging diagnostic device (130) performs a vision transformer-based medical imaging diagnostic method.

[0062] In addition, in FIG. 1, the database (150) is shown as a device independent of the medical imaging diagnostic device (130), but it is not necessarily limited thereto and can be implemented as a logical storage device included in the medical imaging diagnostic device (130).

[0063]

[0064] Figure 2 is a diagram illustrating the system configuration of the medical imaging diagnostic device of Figure 1.

[0065] Referring to FIG. 2, the medical imaging diagnostic device (130) may include a processor (210), memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290).

[0066] The processor (210) can execute a vision transformer-based medical imaging diagnostic procedure according to an embodiment of the present invention, manage memory (230) that is read or written during this process, and schedule the synchronization time between volatile memory and non-volatile memory in memory (230). The processor (210) can control the overall operation of the medical imaging diagnostic device (130) and is electrically connected to the memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290) to control the data flow between them. The processor (210) can be implemented as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the medical imaging diagnostic device (130).

[0067] The memory (230) may include an auxiliary storage device implemented as non-volatile memory such as an SSD (Solid State Disk) or HDD (Hard Disk Drive) and used to store all data required for the medical imaging diagnostic device (130), and may include a main memory implemented as volatile memory such as RAM (Random Access Memory). Additionally, the memory (230) may store a set of instructions that execute the vision transformer-based medical imaging diagnostic method according to the present invention by being executed by an electrically connected processor (210).

[0068] The user input / output unit (250) includes an environment for receiving user input and an environment for outputting specific information to the user, and may include an input device including an adapter such as a touch pad, touch screen, virtual keyboard, or pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input / output unit (250) may correspond to a computing device connected via remote access, and in such case, the medical imaging diagnostic device (130) may be performed as an independent server.

[0069] The network input / output unit (270) provides a communication environment for connecting to a user terminal (110) through a network and may include an adapter for communication such as a LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), and VAN (Value Added Network). Additionally, the network input / output unit (270) may be implemented to provide short-range communication functions such as WiFi and Bluetooth, or wireless communication functions of 4G or higher for wireless transmission of data.

[0070] The communication port section (290) is a hardware interface for connecting to external hardware, for example, the external hardware may include a printer, a mouse, and USB hardware. The communication port section (290) can detect the connection of specific USB hardware and perform the role of a medical imaging diagnostic device (130).

[0071]

[0072] Figure 3 is a diagram illustrating the functional configuration of the medical imaging diagnostic device of Figure 1.

[0073] Referring to FIG. 3, the medical imaging diagnostic device (130) can perform a vision transporter-based medical imaging diagnostic method according to the present invention. To this end, the medical imaging diagnostic device (130) may include a medical image receiving unit (310), a convolutional neural network processing unit (330), a medical image transformer unit (350), and a control unit (370).

[0074] At this time, embodiments of the present invention are not required to include all of the above components simultaneously; depending on each embodiment, some of the above components may be omitted, or some or all of the above components may be selectively included. The operation of each component will be described in detail below.

[0075] The medical image receiving unit (310) can receive the original medical image. The original medical image may include various visual modalities such as CT, MRI, X-ray, and ultrasound.

[0076] In one embodiment, the medical image receiving unit (310) can perform preprocessing operations on the received original medical image. For example, the medical image receiving unit (310) can receive the original medical image from a PACS (Picture Archiving and Communication System) or medical database within the hospital, and can perform various preprocessing operations such as format conversion, noise removal, and contrast enhancement of the received medical image.

[0077] The convolutional neural network processing unit (330) can generate a local feature map through the extraction of local information based on convolutional shift window attention and a global feature map through the extraction of global information based on distortion attention for the original medical image. The convolutional neural network processing unit (330) can be implemented with a convolutional neural network architecture as shown in FIG. 4 to utilize the induced bias of the transformer structure.

[0078] Referring to FIG. 4, the convolutional neural network processing unit (330) may include a Stem Block module (410), a Shift Window Attention Block (SWA Block) module (430), and a Distortion Attention Block (DA Block) module (450).

[0079] The convolutional neural network processing unit (330) can generate a preprocessed map as a first convolutional layer by performing a preprocessing operation on the original medical image before generating a local feature map. The convolutional neural network processing unit (330) can output a first convolutional layer by performing patch-wise encoding on the original medical image using a plurality of 2D convolutional operation layers (Conv2d) as a preprocessing operation through the stem block module (410). The Vision Transporter (ViT) model can learn rich representations by dividing the input image into fixed-size patches and linearly embedding each, so that the sequence of embedded patches is input into the Transformer encoder. Here, the stem block module (410) can use a plurality of 2D convolutional operation layers as a preprocessor to encode low-level features (e.g., edges, textures, boundaries, etc.) of the medical image. For example, the stem block module (410) can use two 3×3 convolution layers as preprocessing operations.

[0080] The shift window attention block (SWA Block) module (430) includes a first convolution block attention module layer (CBAM) and can output a local feature map as a second convolution layer by performing object presence detection through channel-unit attention of the first convolution block attention module layer (CBAM) and object location detection through spatial-unit attention on a first input feature map derived based on the original medical image. The shift window attention block (SWA Block) module (430) further includes a first local attention layer (LA) and a shift window attention layer (SWA), and can output a first input feature map by performing window-unit division of the first local attention layer (LA) and division window-unit shift of the shift window attention layer (SWA) on the first convolution layer of the original medical image.

[0081] The convolutional neural network processing unit (330) can increase the feature dimension by repeatedly connecting shift window attention block (SWA Block) modules (430) in series as shown in FIG. 4 (e.g., C→2C→4C···).

[0082] The distortion attention block (DA Block) module (450) includes a second convolution block attention module layer (CBAM), and can output a global feature map as a third convolution layer by performing object detection through channel-unit attention of the second convolution block attention module layer (CBAM) and object location detection through spatial-unit attention on a second input feature map derived based on a local feature map. The distortion attention block (DA Block) module (450) further includes a second local attention layer (LA) and a distortion attention layer (DA), and can output a second input feature map by performing window-unit division of the second local attention layer (LA), sampling of reference points by division window-unit of the distortion attention layer (DA), and offset of the sampling points on the second convolution layer of the local feature map.

[0083] As shown in FIG. 4, the convolutional neural network processing unit (330) can increase the feature dimension to the target feature dimension by repeatedly connecting distortion attention block (DA Block) modules (450) in series (e.g., C→2C→4C···4C→8C→8C···).

[0084] The medical image transformer unit (350) can perform segmentation and classification tasks on the original medical image through at least one of a local feature map and a global feature map. The medical image transformer unit (350) can fuse the local feature map and the global feature map to generate a fused feature map and perform a segmentation task on the fused feature map. In one embodiment, the medical image transformer unit (350) can fuse N (where N is a natural number) local feature maps output according to the iterative connection of the shift window attention block (SWA Block) module (430) and M (where M is a natural number) global feature maps output according to the iterative connection of the distortion attention block (DA Block) module (450).

[0085] The medical image transformer unit (350) can perform a classification task by processing global average pooling (GAP) on a global feature map.

[0086] The control unit (370) controls the overall operation of the medical image diagnostic device (130) and can manage the control flow or data flow between the medical image receiving unit (310), the convolutional neural network processing unit (330), and the medical image transformer unit (350).

[0087]

[0088] FIG. 5 is a flowchart illustrating a vision transformer-based medical imaging diagnostic method according to the present invention.

[0089] Referring to FIG. 5, the medical image diagnostic device (130) can perform a medical image receiving step of receiving an original medical image through a medical image receiving unit (310) (step S510). The medical image diagnostic device (130) can perform a convolutional neural network processing step of generating a local feature map through the extraction of local information based on convolutional shift window attention and a global feature map through the extraction of global information based on distortion attention for the original medical image through a convolutional neural network processing unit (330) (step S530). The convolutional neural network processing step (S530) may perform the steps of: generating a preprocessed map as a first convolutional layer through preprocessing operations on an original medical image; outputting a local feature map as a second convolutional layer by performing object presence detection through channel-unit attention and object location detection through spatial-unit attention of a first convolutional block attention module layer (CBAM) on a first input feature map derived based on the original medical image; and outputting a global feature map as a third convolutional layer by performing object presence detection through channel-unit attention and object location detection through spatial-unit attention of a second convolutional block attention module layer (CBAM) on a second input feature map derived based on the local feature map.

[0090] The medical imaging diagnostic device (130) can perform a medical imaging transformer step that performs a segmentation task and a classification task on the original medical image through at least one of a local feature map and a global feature map via the medical imaging transformer part (350) (step S550).

[0091]

[0092] Figure 6 is a diagram illustrating an example of a layer configuration in the convolutional neural network architecture of Figure 4.

[0093] Referring to FIG. 6, the medical imaging diagnostic device (130) can generate a local feature map and a global feature map for the original medical image through a convolutional neural network processing unit (330). The convolutional neural network processing unit (330) can increase the feature dimension to a target feature dimension by serially connecting at least one shift window attention block (SWA Block) module (430) and a distortion attention block (DA Block) module (450) to each.

[0094] General Vision Transformer (ViT) models tend not to effectively utilize local induced biases; consequently, achieving comparable performance requires significantly more training data compared to Convolutional Neural Networks (CNNs). However, data collection in the medical and healthcare fields is extremely difficult due to strict regulatory requirements and the need for expert guidance from medical professionals. Therefore, to effectively apply Vision Transformer (ViT) models to medical imaging diagnosis, an architecture capable of effectively utilizing induced biases is required.

[0095] Accordingly, the medical imaging diagnostic device (130) can induce the effect of local induced bias by using a shift window attention block (SWA Block) module (430) in the convolutional neural network processing unit (330). Assuming that the size of each window is M×M, the computational complexity of the shift window attention block (SWA Block) module (430) is calculated as shown in Equation 1 below.

[0096] [Mathematical Formula 1]

[0097]

[0098] Here, C is the feature dimension, and the input image size is H×W.

[0099] The shift window attention block (SWA Block) module (430) consists of a local attention layer (LA), a shift window attention layer (SWA), and a convolution block attention module layer (CBAM).

[0100] The local attention layer (LA), as shown in FIG. 6 (a), divides the feature map using a window of a predetermined size and generates attention weights accordingly. During this process, self-attention is applied independently to each window to extract local features. Self-attention calculates the association between each element of the input data and all other elements, including itself, using three vectors: query (Q), key (K), and value (V). To reflect relative position information for each window, relative position encoding is included in the divided windows during self-attention calculation to consider the relationships between each position within the window, thereby enabling more accurate attention calculation. While applying self-attention, local features are extracted within the given window, and through this process, local induced bias can be injected.

[0101] The shift window attention layer (SWA) generates attention between windows by applying attention between local features as shown in FIG. 6 (b), enabling the model to have a flexible representation. In this process, the feature map is rotated by the divided windows as shown in FIG. 6 (b), that is, the windows are shifted or repositioned by a certain distance from their original positions, and then self-attention is applied as shown in FIG. 6 (a). The shift window attention layer (SWA) performs self-attention within each window and also applies attention between the shifted windows to extract global information.

[0102] The Shift Window Attention Block (SWA Block) module (430) focuses on extracting local information using window-based constraints, but it is also necessary to understand global (or contextual) information to recognize groups spanning a wide area. To effectively extract this global information, the Distortion Attention Block (DA Block) module (450) is used. Similar to the Shift Window Attention Block (SWA Block) module (430), the Distortion Attention Block (DA Block) module (450) consists of a Local Attention Layer (LA), a Distortion Attention Layer (DA), and a Convolutional Block Attention Module Layer (CBAM).

[0103] The distortion attention layer (DA) samples reference points from the feature map and performs an offset through a sub-network; since this is influenced by the size of the feature map, it is applied in the later stages when the feature map size becomes relatively small. As shown in Fig. 6 (c), the distortion attention layer (DA) is a method of flexibly selecting features by using an offset from these sampled points after uniformly sampling reference points from the feature map. In other words, it allows for the extraction of features from a wider and more diverse range by flexibly selecting relevant information around the sampled reference points. The offset is calculated by a sub-offset network composed of two convolutional layers. The values ​​corresponding to the points selected through this offset are used as the key (K) and value (V) in multi-head attention. That is, information corresponding to the selected points is utilized as the input for attention. Additionally, a linear operation is applied to the global feature map and used as a query (Q); through this process, a window can be used for a specific region of interest, and contextual or global information can be effectively captured. The computational complexity of the distortion attention block (DA) module (430) is as shown in Equation 2 below.

[0104] [Mathematical Formula 2]

[0105]

[0106] Here, Ns is the number of sampled points, and k represents the kernel size of the convolution layer of the subnetwork performing the offset.

[0107]

[0108] FIG. 7 is a diagram illustrating an example of a convolution block attention module layer (CBAM) in the convolution neural network architecture of FIG. 4.

[0109] Referring to FIG. 7, the medical imaging diagnostic device (130) can achieve high performance in dense prediction tasks such as classification as well as segmentation by applying a convolution block attention module layer (CBAM) to each bottleneck point of the convolution neural network processing unit (330).

[0110] The Convolutional Block Attention Module (CBAM) layer consists of channel-level attention and spatial-level attention with skip connections, as shown in Figures 7 (a) and (b). Skip connections create a path that passes input data directly to the output, serving to preserve the original information so that it is not distorted while the attention layer performs additional calculations. The Convolutional Block Attention Module (CBAM) layer can effectively extract a refined representation by integrating two types of information—channel and spatial—each containing unique features. Input Given, each attention operation calculates the average and maximum values ​​at the spatial and channel levels. Channel-level attention, as shown in FIG. 7(a), processes Global Average Pooling (GAP) and Global Max Pooling (GMP) on the input feature map, respectively, combines the information obtained therefrom, and applies a 1×1 convolution operation. The result obtained through this process is scaled through a sigmoid operation and multiplied by the original input feature map to assign greater weights to important channels, thereby performing object presence detection. Spatial-level attention, as shown in FIG. 7(b), performs Average Pooling (AP) and Max Pooling (MP) on the input feature map, respectively, to generate a map with compressed spatial information, and combines the maps obtained from Average Pooling and Max Pooling to integrate the two different pieces of information into a single map. An attention map is generated by applying a 1×1 convolution to the combined map, and this attention map can be used to highlight important spatial locations. Finally, the attention map is multiplied with the original feature map through a sigmoid operation, and object location detection is performed by assigning higher weights to important spatial locations.

[0111]

[0112] FIGS. 8 to 10 are drawings illustrating experimental results regarding a medical imaging diagnostic method according to the present invention.

[0113] Figures 8 to 10 correspond to experimental results using publicly available HAM10000C, HAM10000S, MURA, ISIC 2018, and CVC-Clinic DB datasets, and the dataset of Table (a) in Figure 8 was used for the experiment.

[0114] Based on the HAM10000C and MURA datasets, the medical imaging diagnostic model of the present invention was trained from start to finish, and during this process, the top-1 accuracy (the ratio of the highest-ranking predictions to correct accuracy) was compared with other CNN-based and ViT-based models. As can be seen in Table (b) of Figure 8, the experimental results for the HAM10000C dataset showed that the proposed model demonstrated significant improvement in average top-1 accuracy compared to state-of-the-art (SOTA) visual recognition models. The accuracy was 98.2%, recording performance 0.9% to 5.2% higher than other models. On the MURA dataset, the proposed model showed the second-best performance following SOTA models and demonstrated superior results compared to other CNN-based models. In particular, it can be seen that the ViT-B / 32 model exhibits relatively lower performance compared to CNN-based models when trained on a limited medical imaging dataset because it lacks induced bias.

[0115] Segmentation was also performed to evaluate the high-density prediction performance of each model. To evaluate the segmentation, the mean crossing over union (mloU) metric was used, and UperNet, a Unet-based decoder, was used as the segmentation decoder. As can be seen in Table (b) of Figure 8, the results of the segmentation performance evaluation for the HAN10000S, ISIC 2018, and CVC-Clinic DB datasets are presented. The backbone model used in this evaluation was pre-trained on the HAM10000C dataset. When examining the performance on the HAM10000S and ISIC 2018 datasets, the proposed model showed superior performance compared to other models. In particular, the proposed model demonstrated better performance than the Swin-B model. The Swin-B model attempts to mitigate the induced bias of the ViT model by using the shift window attention technique. These results indicate that the proposed model's Shift Window Attention Block (SWA Block) module, Distortion Attention Block (DA Block) module, and Convolution Block Attention Module (CBAM) layer effectively capture not only the global context but also the local context of the Vision Transformer (ViT), making it effective for dense prediction. Additionally, a comparison was made with DCSAU-net, a Unet-based SOTA model. The results showed a performance improvement of 4.6% on HAM10000 and 5.7% on ISIC 2018, while showing a performance 0.7% lower only on CVC-Clinic. This demonstrates that it is more effective than the latest Unet-based SOTA model.

[0116] To evaluate the performance of the model on various datasets, colonoscopy data from the CVC-Clinic DB was utilized, and the backbone model was pre-trained on HAM10000C. As shown in Table (b) of Figure 8, the proposed model showed performance similar to other SOTA models on colonoscopy data. In particular, the performance gap with ViT-B / 32 without added induced bias was 8.4%, and the performance gap with Swin-B was 4.1%. This can be attributed to the fact that the proposed model effectively extracted feature maps more suitable for density prediction by using a Convolutional Block Attention Module (CBAM) layer.

[0117] Additional experiments were conducted using the HAM10000C, HAM10000S, and ISIC 2018 datasets with black-hat transformation preprocessing. Black-hat transformation is a preprocessing technique that removes elements unrelated to lesions, such as hair, which can affect dermatological examination diagnoses. In Table (b) of Figure 8, the parenthesized notation indicates a comparison of models to which black-hat transformation preprocessing was applied. In the case of CNN-based models, the performance of the models to which black-hat transformation preprocessing was applied improved compared to the models without it. On the other hand, in the case of ViT-based models, performance did not improve even when black-hat transformation preprocessing was applied. This suggests that removing elements unrelated to lesions allows for focus on the lesions, which is beneficial only to CNN-based models. However, the method proposed in this invention can guarantee optimal performance regardless of whether black-hat transformation preprocessing is applied.

[0118] Figure 9 shows the results of a qualitative analysis of the perceptual regions that models focus on for final discrimination using GradCAM++. GradCAM++ is a well-known visualization technique that determines which regions of an image are considered important by the network by utilizing gradients based on the model's output. It can be observed that the proposed model tends to focus more on clinically important regions related to lesions than other models.

[0119] As shown in Figure 10, visualizing the segmentation results for the HAM10000S dataset, it can be seen that the proposed model generated segmentation results most similar to the actual correct answer.

[0120] Consequently, it can be confirmed that the model proposed in this invention is effective for medical imaging diagnosis compared to other comparative models. This can contribute to improving diagnostic accuracy not only in the field of skin cancer but also in various medical computer vision fields such as CT and MRI.

[0121]

[0122] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.

[0123]

[0124] [Explanation of the symbol]

[0125] 100: Medical Imaging Diagnostic System

[0126] 110: User terminal 130: Medical imaging diagnostic device

[0127] 150: Database

[0128] 210: Processor 230: Memory

[0129] 250: User I / O Section 270: Network I / O Section

[0130] 290: Communication port section

[0131] 310: Medical image receiver 330: Convolutional neural network processor

[0132] 350: Medical Imaging Transformer Unit 370: Control Unit

[0133] 410: Stem Block Module

[0134] 430: Shift Window Attention Block (SWA Block) Module

[0135] 450: Distortion Attention Block (DA Block) Module

[0136]

[0137] [National R&D projects that supported this invention]

[0138] [Correction pursuant to Rule 91 26.11.2025][Project ID] 2710083013

[0139] [Correction pursuant to Rule 91 26.11.2025][Project No.] RS-2025-02219317

[0140] [Ministry Name] Ministry of Science and ICT

[0141] [Project Management (Specialized) Agency Name] Korea Institute of Information & Communications Technology Planning & Evaluation

[0142] [Correction pursuant to Rule 91 26.11.2025][Research Project Name] AI Star Fellowship Support (Support for Top-tier Young AI Researchers)

[0143] [Correction pursuant to Rule 91 26.11.2025][Project Title] AI Star Fellowship Support Program (Kookmin University)

[0144] [Name of Project Performing Organization] Kookmin University Industry-Academic Cooperation Foundation

[0145] [Correction pursuant to Rule 91 26.11.2025][Research Period] 2025.01.01 ~ 2025.12.31

[0146] [National R&D projects that supported this invention]

[0147] [Correction pursuant to Rule 91 26.11.2025][Project ID] 2710080395

[0148] [Correction pursuant to Rule 91 26.11.2025][Project No.] RS-2024-00417958

[0149] [Ministry Name] Ministry of Science and ICT

[0150] [Project Management (Specialized) Agency Name] Korea Institute of Information & Communications Technology Planning & Evaluation

[0151] [Correction pursuant to Rule 91 26.11.2025][Research Project Name] Global Talent Development Project in the Digital Field

[0152] [Project Title] Research and Development of Ultra-Large Multimodal Artificial Intelligence Specialized in Medical and Healthcare

[0153] [Name of Project Performing Organization] Kookmin University Industry-Academic Cooperation Foundation

[0154] [Correction pursuant to Rule 91 26.11.2025][Research Period] 2025.01.01 ~ 2025.12.31

[0155] [Correction pursuant to Rule 91 26.11.2025][National R&D project that supported this invention]

[0156] [Correction pursuant to Rule 91 26.11.2025][Project ID] 2710086724

[0157] [Correction pursuant to Rule 91 26.11.2025][Project No.] RS-2023-00212484

[0158] [Correction pursuant to Rule 91 26.11.2025][Ministry Name] Ministry of Science and ICT

[0159] [Correction pursuant to Rule 91 26.11.2025][Name of Project Management (Specialized) Agency] National Research Foundation of Korea

[0160] [Correction pursuant to Rule 91 26.11.2025][Research Project Name] Outstanding Young Researcher

[0161] [Correction pursuant to Rule 91 26.11.2025][Project Title] Prediction of Explainable Motion in Complex Real-World Driving Environments

[0162] [Correction pursuant to Rule 91 26.11.2025][Name of Project Performing Organization] Kookmin University Industry-Academic Cooperation Foundation

[0163] [Correction pursuant to Rule 91 26.11.2025][Research Period] 2025.01.01 ~ 2025.12.31

[0164] [Correction pursuant to Rule 91 26.11.2025][National R&D project that supported this invention]

[0165] [Correction pursuant to Rule 91 26.11.2025][Project ID] 2710008221

[0166] [Correction pursuant to Rule 91 26.11.2025][Project No.] RS-2024-00357879

[0167] [Correction pursuant to Rule 91 26.11.2025][Ministry Name] Ministry of Science and ICT

[0168] [Correction pursuant to Rule 91 26.11.2025][Name of Project Management (Specialized) Agency] Korea Institute of Information and Communications Technology Planning and Evaluation

[0169] [Correction pursuant to Rule 91 26.11.2025][Research Project Name] Immersive Content Core Technology Development Project

[0170] [Correction pursuant to Rule 91 26.11.2025][Project Title] AI-based Biodata Fusion and Generation Technology for Intelligent Personalized Chronic Disease Management

[0171] [Correction pursuant to Rule 91 26.11.2025][Name of Project Performing Organization] Kookmin University Industry-Academic Cooperation Foundation

[0172] [Correction pursuant to Rule 91 26.11.2025][Research Period] 2025.01.01 ~ 2025.12.31

Claims

1. A medical image receiving unit that receives the original medical image; A convolutional neural network processing unit that generates a local feature map through the extraction of local information based on convolution-based shift window attention and a global feature map through the extraction of global information based on distortion attention for the above original medical image; and A vision transformer-based medical imaging diagnostic device comprising a medical image transformer unit that performs a segmentation task and a classification task for the original medical image through at least one of the local feature map and the global feature map.

2. In paragraph 1, the convolutional neural network processing unit A vision transformer-based medical imaging diagnostic device characterized by generating a preprocessed map as a first convolutional layer through a preprocessing operation on the original medical image prior to the generation of the local feature map.

3. In paragraph 2, the convolutional neural network processing unit A vision transformer-based medical imaging diagnostic device characterized by including a stem block module that outputs a first convolution layer by performing patch-wise encoding on the original medical image using a plurality of 2D convolution layers (Conv2d) as a preprocessing operation.

4. In paragraph 1, the convolutional neural network processing unit It includes a Shift Window Attention Block (SWA Block) module, The above shift window attention block module includes a first convolution block attention module layer (CABM), and A vision transformer-based medical imaging diagnostic device characterized by performing object presence detection through channel-unit attention and object location detection through spatial-unit attention of the first convolution block attention module layer (CBAM) on a first input feature map derived based on the original medical image, and outputting the local feature map as a second convolution layer.

5. In paragraph 4, the shift window attention block module is It further includes a first local attention layer (LA) and a shift window attention layer (SWA), and A vision transformer-based medical imaging diagnostic device characterized by outputting a first input feature map by performing window unit division of the first local attention layer (LA) and division window unit shift of the shift window attention layer (SWA) on the first convolution layer of the original medical image.

6. In paragraph 5, the convolutional neural network processing unit A vision transformer-based medical imaging diagnostic device characterized by increasing feature dimensions by repeatedly connecting the above shift window attention block modules in series.

7. In paragraph 1, the convolutional neural network processing unit It includes a distortion attention block (DA Block) module, and The above distortion attention block (DA Block) module includes a second convolution block attention module layer (CBAM), and A vision transformer-based medical imaging diagnostic device characterized by performing object presence detection through channel-unit attention and object location detection through spatial-unit attention of the second convolution block attention module layer (CABM) on a second input feature map derived based on the local feature map, and outputting the global feature map as a third convolution layer.

8. In Clause 7, the distortion attention block (DA Block) module is It further includes a second local attention layer (LA) and a distortion attention layer (DA), and A vision transformer-based medical imaging diagnostic device characterized by outputting the second input feature map by performing window-unit division of the second local attention layer (LA), sampling of reference points in the division window unit of the distortion attention layer (DA), and offset of the sampling points for the second convolution layer of the local feature map.

9. In paragraph 8, the convolutional neural network processing unit A vision transformer-based medical imaging diagnostic device characterized by repeatedly connecting the above distortion attention block (DA Block) modules in series to increase the feature dimension up to the target feature dimension.

10. In paragraph 1, the medical imaging transformer part A vision transformer-based medical imaging diagnostic device characterized by fusing the local feature map and the global feature map to generate a fused feature map and performing the segmentation task on the fused feature map.

11. In Clause 10, the medical imaging transformer part A vision transformer-based medical imaging diagnostic device characterized by fusing N (where N is a natural number) local feature maps output according to the iterative connection of a shift window attention block module and M (where M is a natural number) global feature maps output according to the iterative connection of a distortion attention block (DA Block) module.

12. In paragraph 1, the medical imaging transformer part A vision transformer-based medical imaging diagnostic device characterized by performing the classification task by processing Global Average Pooling (GAP) on the global feature map.

13. A vision transformer-based medical imaging diagnostic method performed in a medical imaging diagnostic device, Medical image receiving step for receiving the original medical image; A convolutional neural network processing step for generating a local feature map through the extraction of local information based on convolution-based shift window attention and a global feature map through the extraction of global information based on distortion attention for the original medical image; and A vision transformer-based medical image diagnosis method comprising a medical image transformer step that performs a segmentation task and a classification task for the original medical image through at least one of the local feature map and the global feature map.

14. In Clause 13, the above convolutional neural network processing step is A vision transformer-based medical image diagnosis method characterized by including the step of generating a preprocessed map as a first convolutional layer through a preprocessing operation on the original medical image prior to generating the local feature map.

15. In Clause 13, the above convolutional neural network processing step is A vision transformer-based medical image diagnosis method characterized by including the step of performing object presence detection through channel-unit attention and object location detection through spatial-unit attention on a first input feature map derived based on the original medical image, and outputting the local feature map as a second convolution layer.

16. In Clause 13, the above convolutional neural network processing step is A vision transformer-based medical imaging diagnostic method characterized by including the step of performing object presence detection through channel-unit attention and object location detection through spatial-unit attention of a second convolution block attention module layer (CBAM) on a second input feature map derived based on the local feature map, and outputting the global feature map as a third convolution layer.

17. In Clause 13, the medical imaging transformer step A vision transformer-based medical imaging diagnostic method characterized by including the step of fusing the local feature map and the global feature map to generate a fused feature map and performing the segmentation task on the fused feature map.

18. In Paragraph 13, the medical imaging transformer step A vision transformer-based medical imaging diagnostic method characterized by including the step of performing the classification task by processing global average pooling (GAP) for the global feature map.

Images