Apparatus and method for analyzing disease
The disease analysis device and method address the limitations of single-modality diagnosis by using deep learning techniques to extract and fuse features from multiple time points, enhancing diagnostic accuracy and reliability for degenerative brain diseases like Alzheimer's.
Patent Information
- Application Number
- PCT/KR2024/005381
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2024-04-22
- Publication Date
- 2024-11-21
AI Technical Summary
Current disease diagnosis technologies for degenerative brain diseases like Alzheimer's rely heavily on single modalities, leading to misdiagnosis due to similarities in brain tissue, and lack precise location information of damaged tissue, resulting in low reliability for medical professionals.
A disease analysis device and method utilizing a data acquisition unit to gather volume data from multiple time points, employing a 3D convolutional neural network and bidirectional recurrent neural network to extract deep features and temporal characteristics, fusing them with additional data for accurate disease analysis and visualization processing, including 3D brain surface rendering and activation maps.
Enhances the accuracy and reliability of disease diagnosis by providing detailed information on lesion changes at the pixel level, improving the ability to detect and predict neurological conditions like Alzheimer's disease progression.
Smart Images

Figure KR2024005381_21112024_PF_FP_ABST
Abstract
Description
Disease analysis device and method
[0001] The present invention relates to a disease analysis device and method.
[0002] Degenerative brain diseases such as Alzheimer's disease (AD) and Parkinson's disease are caused by nerve damage and primarily affect people over the age of 65. Currently, there is no clear cure for these degenerative brain diseases. Therefore, most treatments focus on slowing the progression of the underlying nerve damage. Mild Cognitive Impairment (MCI) is an intermediate stage that distinguishes patients with normal cognitive function from those with degenerative brain diseases. It is generally considered a transitional stage between normal cognitive function and neurological impairment. Statistics show that approximately 10-15% of patients with MCI progress to Alzheimer's disease each year. Given that neurological damage is highly treatable in its early stages, early detection of MCI is crucial to prevent the onset of degenerative brain diseases.
[0003] Recent advancements in information and communication technology have led to the emergence of artificial intelligence (AI). These technologies are also being used to differentiate cognitive impairments, such as stable mild cognitive impairment (MCI) and progressive mild cognitive impairment (MCI). However, most technologies focus solely on a single modality utilizing already validated data. Diagnosing diseases based on a single modality is prone to misdiagnosis, and due to brain tissue similarities, it is difficult to clearly distinguish between normal cognition and MCI. Therefore, efforts are being made to diagnose neurological diseases through continuous data management and analysis, rather than relying on a single modality. However, the development of these technologies has been limited and has been plagued by low reliability. Most of these technologies fail to provide information on the precise location of tissue damage at the pixel level, making accurate assessments difficult and increasing the likelihood of misdiagnosis. Consequently, these technologies have not been sufficiently trusted by healthcare professionals.
[0004] The problem to be solved is to provide a disease analysis device and method that can detect, judge, or predict the condition of a lesion or its changes in a target object.
[0005] To solve the above-described problem, a disease analysis device and method are provided.
[0006] A disease analysis device may include a data acquisition unit that acquires a plurality of volume data from a subject at a plurality of time points, wherein each of the plurality of volume data includes at least one slice image, a processor that extracts a depth characteristic from the at least one slice image based on a first learning model, acquires a temporal characteristic for a relationship between the plurality of volume data acquired at the plurality of time points based on a second learning model using the depth characteristic, and acquires a result for disease analysis using the characteristic for the relationship between the plurality of volume data.
[0007] The above-described depth feature may include at least one of a feature inherent in the at least one slice image and a feature regarding a relationship between a plurality of slice images within the volume data.
[0008] The processor may obtain a fused feature vector by fusing the temporal characteristic with additional data or a value corresponding to the additional data, wherein the value corresponding to the additional data may include an output value of a third learning model corresponding to an input of the additional data.
[0009] Additionally, the processor may obtain a result for disease analysis by using the output value of the fourth learning model according to the input of the fused feature vector.
[0010] The above additional data may include at least one of a cognitive score, demographic characteristics, and biomarkers.
[0011] The above processor may also perform visualization processing corresponding to the results for disease analysis.
[0012] The above visualization processing may also include at least one of three-dimensional brain surface rendering and generation of an activation map.
[0013] The above processor may perform the visualization processing by inputting the output of at least one learning model corresponding to the input of the plurality of volume data into a class activation map and a grad-class activation map.
[0014] The first learning model may include a three-dimensional convolutional neural network, and the second learning model may include a bidirectional recurrent neural network.
[0015] A disease analysis method may include a step of acquiring a plurality of volume data obtained from a subject at different points in time, a step of extracting a depth characteristic from at least one slice image within the plurality of volume data based on a first learning model, a step of acquiring a temporal characteristic for a relationship between the plurality of volume data using the depth characteristic based on a second learning model, and a step of acquiring a result for disease analysis using the characteristic for the relationship between the plurality of volume data.
[0016] The above-described depth feature may include at least one of a feature inherent in the at least one slice image and a feature regarding a relationship between a plurality of slice images within the volume data.
[0017] The disease analysis method may further include a step of obtaining a fused feature vector by fusing the temporal characteristic with additional data or a value corresponding to the additional data, wherein the value corresponding to the additional data includes an output value of a third learning model corresponding to an input of the additional data, and a step of obtaining a result for the disease analysis by using an output value of a fourth learning model according to an input of the fused feature vector, wherein the additional data may include at least one of a cognitive score, a demographic characteristic, and a biomarker.
[0018] The disease analysis method may further include a step of performing visualization processing corresponding to the results for disease analysis, wherein the visualization processing may include at least one of three-dimensional brain surface rendering and generation of an activation map.
[0019] The step of performing visualization processing corresponding to the results for the disease analysis may further include the step of inputting the output of at least one learning model corresponding to the input of the plurality of volume data into a class activation map and a grad-class activation map.
[0020] The first learning model may include a three-dimensional convolutional neural network, and the second learning model may include a bidirectional recurrent neural network.
[0021] According to the disease analysis device and method described above, it is possible to more accurately and effectively detect, judge or predict the condition of a lesion or its changes in a subject.
[0022] According to the disease analysis device and method described above, it is possible to obtain detailed information on the status of lesions on a subject at the pixel level, thereby enabling medical personnel to more accurately diagnose and predict the patient's disease.
[0023] The disease analysis device and method described above enable the construction of more accurate, stable, and medically intuitive models for diseases such as Alzheimer's disease, which require continuous data management and analysis. This, in turn, enhances the reliability of neurological diagnosis using the device.
[0024] According to the disease analysis device and method described above, by applying temporally explainable artificial intelligence (XAI: eXplainable AI) technology to a classification task using 3D magnetic resonance imaging data to visually express the time-dependent aspects of magnetic resonance imaging data, the convenience of examining the progression pattern of neural damage in Alzheimer's patients can be reconsidered.
[0025] Figure 1 is a block diagram of a disease analysis system according to one embodiment.
[0026] FIG. 2 is a block diagram of a processor of a disease analysis device according to one embodiment.
[0027] FIG. 3 is a diagram for explaining a data preprocessing unit according to one embodiment.
[0028] Figure 4 is a diagram for explaining a deep feature learning unit according to one embodiment.
[0029] FIG. 5 is a diagram for explaining a temporal characteristic learning unit according to one embodiment.
[0030] FIG. 6 is a drawing for explaining an information fusion unit according to one embodiment.
[0031] Fig. 7 is a drawing for explaining a visualization processing unit according to one embodiment.
[0032] Figure 8 is a diagram of an image according to a guided Grad-CAM and an image according to a Grad-CAM.
[0033] Figures 9 to 11 are diagrams of images that can be visually presented to a user to detect the progression of Alzheimer's disease.
[0034] Figure 12 is a flowchart of a disease analysis method according to one embodiment.
[0035] The advantages and features of the invention, and the methods for achieving them, will become clearer with reference to the embodiments described below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined solely by the scope of the claims.
[0036] The terms used in this specification will be briefly explained, and the present invention will be described in detail.
[0037] The terms used in this invention have been selected from widely used, current terms, taking into account their functions. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the description of the relevant invention. Therefore, the terms used in this invention should not be defined simply as names, but rather based on their inherent meanings and the overall content of the invention.
[0038] When a part of the specification is said to "include" a component, this does not mean that other components are excluded, but rather that other components may be included, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and "unit" used in the specification mean a unit that processes at least one function or operation, and may be implemented by software, a hardware component such as an FPGA or ASIC, or a combination of software and hardware. However, terms such as "part," "module," and "unit" are not limited to software or hardware. A "part," "module," and "unit" may be configured to reside on an addressable storage medium, or may be configured to execute one or more processors. Thus, as an example, terms such as "unit," "module," and "unit" include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.
[0039] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily practice them. Furthermore, for the purpose of clearly explaining the present invention, parts of the drawings that are not relevant to the description are omitted.
[0040] Terms including ordinal numbers, such as "first," "second," etc., may be used to describe various components, but the components are not limited by the terms. The terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, the first component could be referred to as the second component, and similarly, the second component could also be referred to as the first component. The term "and / or" includes any combination of multiple related items or any one of multiple related items.
[0041]
[0042] Hereinafter, with reference to FIGS. 1 to 11, an embodiment of a disease analysis device and a disease analysis system including the same will be described.
[0043] Figure 1 is a block diagram of one embodiment of a disease analysis system.
[0044] A disease analysis system (1) according to one embodiment may include a data collection device (10) and a disease analysis device (50), as illustrated in FIG. 1. The data collection device (10) and the disease analysis device (50) may be implemented to transmit or receive data or commands, etc., unilaterally or bilaterally, as needed. For example, they may be provided to enable communication based on a circuit line, a cable, a wireless communication technology, or a combination thereof. Here, the wireless communication technology may include at least one of a short-range communication technology and a long-range communication technology. The short-range communication technology may include, for example, a communication technology such as Wi-Fi, Wi-Fi Direct, Bluetooth, Zigbee communication, CAN communication, Ultra-WideBand (UWB) communication, or Near Field Communication (NFC). Long-distance communication technologies may include, for example, mobile communication standard technologies such as 3GPP, 3GPP2, WiBro, or WiMAX. Depending on the embodiment, the data collection device (10) may be omitted. In this case, the disease analysis device (50) may include a portion corresponding to the data acquisition unit (11) of the data collection device (10).
[0045] According to one embodiment, the data collection device (10) can collect and acquire at least one data (e.g., image data) necessary for disease analysis of the subject (9) from the subject (9). Here, the subject (9) may include, for example, a human body (e.g., a patient), an animal, or a plant, or may include only a portion thereof (e.g., a tissue, an organ, or a system). The data may be acquired in the form of an electrical signal. In this case, the data collection device (10) may amplify the data in the form of an electrical signal, perform analog-to-digital conversion, and / or remove noise. In addition, the data collection device (10) may visualize the data in the form of an electrical signal in two dimensions or three dimensions and acquire corresponding image data. If the data collection device (10) is a magnetic resonance imaging device, the image data (hereinafter, magnetic resonance imaging data) acquired by the data collection device (10) may be three-dimensional volume data (3D volume data). 3D volume data may include multiple slice images. The slice images may be, but are not limited to, longitudinal cross-sectional images.
[0046] According to an embodiment, the data collection device (10) may collect multiple data from the same subject (9) at the same time or several times. For example, the data collection device (10) may acquire data on the subject (9) at a baseline time point (BL). The data collection device (10) may acquire data on the subject (9) at a time point after a certain period of time has elapsed (hereinafter, a first time point. For example, a time point (M06) after 6 months have elapsed from the baseline time point). The data collection device (10) may also acquire data on the subject (9) at a time point after a certain period of time has elapsed more than the first time point (hereinafter, a second time point. For example, a time point (M12) after 12 months have elapsed from the baseline time point). In this case, metadata corresponding to the acquisition time point may be further added to each piece of data. The data acquired multiple times may be used by the learning processing unit (120) of the processor (100) to predict disease progression or train a learning model, as described below.
[0047] The data collection device (10) may include an image sensor, an RF coil, a transducer (such as a piezoelectric element), or an X-ray detector for collecting and acquiring data. The data collection device (10) may further include a processor that is electrically connected to the image sensor, the RF coil, or the like, as needed, and processes, converts, or processes electrical signals acquired from the image sensor or the RF coil to acquire corresponding data (e.g., image data). In addition, depending on the type of the data collection device (10), it may further include other components for collecting data.
[0048] The data acquired by the data collection device (10) may be provided to the disease analysis device (50) as is or after undergoing a predetermined processing process. For example, the data acquired by the data collection device (10) may be transmitted to the disease analysis device (50) as an electrical signal as is or after being partially processed. The data acquired by the data collection device (10) may be transmitted to the disease analysis device (50) after image data corresponding to the electrical data is generated. According to an embodiment, at least one data acquired by the data collection device (10) may be transmitted to the disease analysis device (50) immediately or at a predefined time.
[0049] The above-described data collection device (10) can be implemented using at least one device capable of collecting data. For example, it can be implemented using a smart phone, a tablet PC, a visible light or infrared camera device, a sensor device (such as an optical sensor, a gas sensor, or an acoustic sensor), a vehicle, an aircraft, a machine device, or a robot. According to one embodiment, the data collection device (10) may include a medical device. The medical device may include, but is not limited to, a magnetic resonance imaging (MRI) device, an ultrasound imaging device, an X-ray imaging device, a mammography device, a computed tomography (CT) device, or a positron emission tomography (PET) device, for example.
[0050] As described above, the data acquisition device (10) may be a magnetic resonance imaging device in one embodiment. In this case, the data acquisition device (10) may include a magnetic field forming unit (10-1) that forms a static magnetic field in an internal space where the object (9) is accommodated, a gradient magnetic field forming unit (10-2) that forms a gradient magnetic field in the internal space, an RF coil (10-3) that applies an RF pulse to the object (9) and / or acquires a magnetic resonance signal from the object (9), and an image processing processor (not shown) electrically connected to the RF coil. The RF coil may be installed in the magnetic resonance imaging device or may be mounted on all or part of the object (9) (for example, the head of the object (9)). If a subject (9) is accommodated in the internal space of a magnetic resonance imaging device and a static magnetic field of a predetermined strength generated by a static magnetic field generating unit (10-1) of the magnetic resonance imaging device is applied to the subject (9), the atomic nuclei of atoms (e.g., hydrogen, phosphorus, or sodium) causing a magnetic resonance phenomenon inside the subject (9) are magnetized in response, and the spins of these atomic nuclei are aligned in the direction of the static magnetic field. At this time, if an RF pulse is applied to the subject (9), a magnetic resonance signal is induced in the RF coil due to the magnetic resonance phenomenon. If the RF coil is mounted on the head of the subject (9), a magnetic resonance signal corresponding to the head of the subject (9) is induced in the RF coil. The gradient magnetic field generating unit (10-2) can generate a gradient magnetic field with respect to the subject (9) so that the magnetic resonance signal can be spatially encoded. Accordingly, the magnetic resonance signal has spatial information about at least one slice, and acquisition of k space becomes possible. The image processing processor can obtain at least one magnetic resonance image data (raw data before processing) based on the K space according to the magnetic resonance signal.The magnetic resonance signal collected by the RF coil or the magnetic resonance image acquired by the image processing processor is transmitted to the disease analysis device (50) as is or after being processed as scheduled without any separate processing.
[0051] The disease analysis device (50) can obtain information on the disease of the subject (9) or prediction results on the future state of the disease by utilizing data collected by or directly collected by the data collection device (10). In addition, the disease analysis device (50) can also provide at least one data generated for diagnosing or predicting the disease to a user (e.g., the subject (9), a doctor, a nurse, a radiologist, or a radiologist). Here, the data for diagnosing or predicting the disease can include image data (such as a magnetic resonance image, an X-ray image, or an ultrasound image) at the time of data collection or image data estimated for a later time point.
[0052] A disease analysis device (50) according to one embodiment may include a data acquisition unit (51), an input unit (52), an output unit (59), a storage unit (55), and a processor (100). At least two of these may transmit and receive data or commands unilaterally or bilaterally based on circuit lines, cables, or wireless communication technology. Depending on the embodiment, at least one of the data acquisition unit (51), the input unit (52), the output unit (59), and the storage unit (55) may be omitted.
[0053] The data acquisition unit (51) can acquire data required for disease analysis. Here, the data required for disease analysis may include, for example, image data for the subject (9) or an electrical signal corresponding thereto. The data acquisition unit (51) may include, but is not limited to, a data input / output terminal, a communication module (for example, a LAN card, a short-range communication module, or a mobile communication module), an image capturing module, a magnetic resonance imaging module, an ultrasound imaging module, or an X-ray imaging module, depending on the embodiment.
[0054] In one embodiment, the data required for disease analysis may include data on the subject (9) acquired by the data collection device (10). For example, the data may be magnetic resonance imaging data on the subject (9) or an electrical signal corresponding thereto. The magnetic resonance imaging data may be volume data composed of a plurality of slice images, as described above. In addition, according to one embodiment, the data required for disease analysis may include at least one of at least one image data on the subject (9) at a reference time point, at least one image data on the subject (9) at a first time point, and at least one image data on the subject (9) at a second time point. In another embodiment, the data acquisition unit (51) may directly acquire such image data or electrical signals. For example, if the disease analysis device (50) is a magnetic resonance imaging device, the data acquisition unit (51) may include a static magnetic field generation unit (10-1), a gradient magnetic field generation unit (10-2), and an RF coil (10-3). In addition, the disease analysis device (50) can obtain magnetic resonance imaging data by inducing a magnetic resonance signal to the RF coil (10-3) according to the operation of the magnetic field forming unit (10-1), the gradient magnetic field forming unit (10-2), and the RF coil (10-3).
[0055] The data required for disease analysis may further include additional data about the subject (9), as needed. The additional data about the subject (9) may be data additionally used in the training and analysis process, as described below. For example, if the subject (9) is the brain, it may include at least one of a cognitive score (CS: cognitive score, which may be given in the form of a vector) of the patient, statistical data related to the subject (9) or the patient (e.g., demographic features), and a biomarker about the subject (9) itself. The biomarker may include, for example, a hippocampal feature. The hippocampal feature may include hippocampal volume, etc.
[0056] Additionally, the data required for disease analysis may further include training data for at least one learning model and / or verification data for at least one learning model. The training data or verification data may be utilized by the learning processing unit (120).
[0057] The input unit (53) may receive commands or instructions from the user regarding the disease analysis device (50), such as initiating the operation of the disease analysis device (50) or displaying analysis results. In addition, the input unit (53) may further receive at least one program (which may be referred to as an app, application, or software) written to perform the operation of the disease analysis device (50) or data for updating the same. Here, the program may include at least one learning model to be processed by the processor (100). These data, commands, or programs may be input according to operations by an administrator or user of the system. According to an embodiment, the input unit (53) may also receive data necessary for the disease analysis described above. That is, for example, data regarding the subject (9) acquired by the data collection device (10), additional data regarding the subject (9), training data for at least one learning model, and / or verification data for at least one learning model, etc. may be provided to the disease analysis device (50) through the input unit (53).
[0058] The input unit (53) can be implemented using, for example, a keyboard, a mouse, a tablet input device, a touch screen, a touch sensor, a microphone, a data input / output module, an image capture module, a motion detection sensor, a pressure sensor, or a proximity sensor.
[0059] The storage unit (55) can temporarily or non-temporarily store various data, settings, programs, or commands. For example, the storage unit (55) may store image data acquired by the data collection device (10) or the data acquisition unit (51), additional data on the object (9), data for training or verifying a learning model, a learning model before training, a previously trained learning model, and / or data or settings temporarily generated during the processing of the processor (100). The program stored in the storage unit (55) may be run upon a call from the processor (100). Depending on the embodiment, the program in the storage unit (55) may be directly written or modified by a designer such as a programmer and then stored. The program in the storage unit (55) may also be received from another physical recording medium (such as an external memory device or a compact disc (CD)) and stored or updated. Alternatively, the program in the storage unit (550) may be acquired or updated through an electronic software distribution network accessible via a wired or wireless communication network.
[0060] The storage unit (55) may include at least one of a main memory device and an auxiliary memory device, depending on the embodiment. The main memory device may be implemented using a semiconductor storage medium such as ROM and / or RAM. The auxiliary memory device may be implemented using at least one storage medium capable of permanently or semi-permanently storing data, such as a flash memory device (e.g., a Secure Digital (SD) card or a Solid State Drive (SSD)), a hard disk drive (HDD), or an optical media (e.g., a compact disc or a DVD).
[0061] The output unit (59) can output the processing result of the processor (100) to the outside. For example, the output unit (59) can visually output to the outside an image of the object (9) at the current time, an image predicted for the object (9) at some time after the present, or an image predicted for the object (9) at another time after that (169 in FIG. 7) and provide it to the user. In addition, the output unit (59) can transmit these images to another device (not shown, for example, a smartphone or a desktop computer) through a wired or wireless communication network. The output unit (59) can be implemented using, for example, a display, a printer device, a speaker device, an image output terminal, or a data input / output terminal.
[0062] The processor (100) collects at least one piece of data, analyzes the disease based on the collected data, and can predict the disease at a given point in time. The processor (100) provides the prediction result to the user in an appropriate format as needed, so that the user can easily identify the disease. According to one embodiment, the processor (100) may perform at least one operation among preprocessing of data, various processing of image data, training of at least one learning model, prediction based on the trained learning model, processing of the prediction result, and output control of the prediction result. These operations may be performed simultaneously and / or sequentially depending on the embodiment. If the processor (100) is directly electrically connected to the RF coil of the magnetic resonance imaging device and receives an electrical signal before processing or a partially processed electrical signal from the RF coil, the processor (100) may further perform a process of generating and acquiring magnetic resonance imaging data corresponding to the magnetic resonance signal of the RF coil. The generation of the magnetic resonance imaging data may be performed prior to the preprocessing of the data, the training of the learning model, or the prediction operation.
[0063] According to an embodiment, the processor (100) may be implemented using one or more of a central processing unit (CPU), a graphic processing unit (GPU), a microcontroller unit (MCU), an application processor (AP), an electronic control unit (ECU), or other electronic devices capable of performing the operations described above.
[0064] These devices may be manufactured based on one or more semiconductor chips, substrates, circuits, or related components. If necessary, the processor (100) may be implemented using these devices alone, or may be implemented using a combination of two or more similar or different devices.
[0065] FIG. 2 is a block diagram of a processor of a disease analysis device according to one embodiment.
[0066] The processor (100) may include a data preprocessing unit (110), a learning processing unit (120), an information fusion unit (150), and a visualization processing unit (160), as illustrated in FIG. 2. At least two of the data preprocessing unit (110), the learning processing unit (120), the information fusion unit (150), and the visualization processing unit (160) may be physically or logically separate. In addition, at least one of these (110, 120, 150, and 140) may be omitted, and in this case, the omitted one may be provided in another information processing device that is physically separated from the disease analysis device (50).
[0067] FIG. 3 is a diagram for explaining a data preprocessing unit according to one embodiment.
[0068] Referring to the diagram shown in FIG. 3, the data preprocessing unit (110) can perform preprocessing on acquired image data (9, for example, magnetic resonance imaging data).
[0069] For example, the data preprocessing unit (110) may perform enlargement, reduction, or direction adjustment (e.g., alignment of the direction and / or position of multiple magnetic resonance image data) for each image data (19), thereby transforming each image data (19) into a form suitable for processing. For another example, the data preprocessing unit (110) may remove unnecessary information (artifacts, etc.) during the operation process described below. If the image data is magnetic resonance image data for the head, the data preprocessing unit (110) may correct noise, artifacts, or bias fields present in the magnetic resonance image data (e.g., each slice image) (111), or perform an operation of removing unnecessary parts for analysis (e.g., skull, etc.) in the magnetic resonance image data and extracting only the brain region (113). In addition, if necessary, the data preprocessing unit (110) may perform MNI152 standard template registration, etc. (112). These operations can be performed simultaneously or at different times, and can be performed sequentially or in parallel. For example, the data preprocessing unit (110) may first perform bias field correction on the original image data (9) (111), perform MNI152 registration (112), and then perform skull removal (113). All or part of the above-described operations may be omitted. Depending on the selection of the user or designer, the data preprocessing unit (110) may also perform other necessary processing in addition to these. The preprocessed data may be transmitted to the learning processing unit (120).
[0070] According to an embodiment, the data preprocessing unit (110) may select and extract one or more images (e.g., slice images) appropriate for diagnosis from magnetic resonance imaging data (which may include at least one of data before preprocessing and preprocessed data). The selection of the slice images may be performed by calculating the overall size of half of the volume existing in both lateral directions based on the central slice in the 3D volume data. The number of extracted slice images may be defined by the designer or user, and may be, for example, 110. However, the number of extracted slice images is not limited thereto. By selecting the optimal slice images in this way, it is possible to obtain a feature vector that can better represent the disease to be diagnosed (e.g., Alzheimer's).
[0071] The learning processing unit (120) can acquire at least one of raw image data(s) and preprocessed image data(s). The learning processing unit (120) can extract characteristics within at least one image data and identify patterns (patterns between sequences) between multiple image data through at least one learning model.
[0072] To perform such an operation, a learning processing unit (120) according to one embodiment may include a deep feature learning unit (130) and a temporal feature learning unit (140).
[0073] Figure 4 is a diagram for explaining a deep feature learning unit according to one embodiment.
[0074] Referring to the diagram shown in FIG. 4, the deep feature learning unit (130) can extract deep features from image data (e.g., volume data (119) of 110Х110Х110).
[0075] Specifically, for example, the deep feature learning unit (130) can receive N pieces of image data (119: 119-1, 119-2) (N is a natural number greater than or equal to 1), and extract and acquire one or more spatial and / or temporal deep features (139) from the received N pieces of image data (119: 119-1, 119-2). The extraction of such spatial or temporal deep features (139) can be performed using at least one learning model (hereinafter, a first learning model).
[0076] If the data acquisition device (10) is a magnetic resonance imaging device, the volume data (119) may include multiple slice images corresponding to cross-sections of the object (9) at various locations (e.g., cross-sections of the brain). In this case, the deep feature learning unit (130) may acquire features inherent in each slice image within the volume data (119) corresponding to at least one point in time (intra-slice features) and / or acquire features between two or more slice images within each volume data (119) (inter-slice features). At least one of the intra-slice features and the inter-slice features may be acquired based on one or more first learning models. For example, as described below, at least one of the intra-slice features and the inter-slice features may include a convolutional neural network.
[0077] According to an embodiment, the volume data (119) may be data(s) acquired from a single point in time, or may be data(s) acquired from multiple different points in time. Here, the data acquired from multiple points in time may include, but is not limited to, volume data (119-1) acquired from a reference point in time and / or volume data (119-2) acquired from a first point in time, as illustrated in FIG. 4. These volume data (119-2) may have the same size. In this case, the deep feature learning unit (130) may extract deep features (139) from each image data (for example, longitudinal slice image(s) from the reference point in time to the first point in time and longitudinal slice image(s) from the reference point in time to the second point in time) acquired through scanning at different points in time by passing the three-dimensional volume data for the same object (9) acquired from each point in time (for example, the reference point in time, the first point in time, and the second point in time, etc.) through the first learning model. Accordingly, for each time point (e.g., reference time point, first time point, and second time point), the relationship between the spatial characteristics of each slice corresponding to each time point is obtained.
[0078] The first learning model utilized by the deep feature learning unit (130) is designed to learn functions between deep spaces and slices. The first learning model may be trained by the deep feature learning unit (130), and / or may be trained by another device (not shown) and then transmitted to the disease analysis device (50) via a memory device or a communication network.
[0079] According to one embodiment, the first learning model may be implemented by employing a 3D convolutional neural network (3D CNN) having multiple layers (131, 132, 133) to simultaneously extract spatial and temporal characteristics. Here, the 3D convolutional neural network may include, for example, AlexNet or 3D ConvNet, but is not limited thereto. According to an embodiment, the 3D convolutional neural network may be provided to include a plurality of convolutional layers (131, 132, 133) in which the number of kernels is given as 8, 32, and 64, respectively. In this case, the kernel size of the convolutional layers (131, 132, 133) may be given as 3×3×3, and the stride may be given as 1. The sizes of the output values of each convolutional layer (131, 132, 133) can be, for example, 8×110×110×110, 32×27×27×27, and 64×13×13×13, respectively. The size of the input value of the first convolutional layer (131) can be 110×110×110. The sizes of the input values of the second and third convolutional layers (132, 133) can correspond to the sizes of the output values of the first and second convolutional layers (131, 132). Meanwhile, at least one layer (131, 132, 133) among each convolutional layer (131, 132, 133) may be connected to a pooling layer and / or dropout may be applied, as described below. The activation function of each convolutional layer (131, 132, 133) may be a ReLu function, but a sigmoid function or a step function may also be used as needed. Depending on the embodiment, the last hidden layer (133) may be a fully connected layer (FCL).
[0080] The pooling layer for each convolutional layer (131, 132, 133) can also be implemented using 3D max pooling. Here, 3D max pooling is performed by calculating the maximum value of each sub-block selected from a given volume, for example. 3D max pooling is performed based on the size of the sub-block (k) and the stride (s). If the size of the volume is (h×w×d), the output of 3D max pooling can be given as (hk)(s+1)×(wk)(s+1)×(dk)(s+1). 3D max pooling allows important features that help in identifying the input volume data to be preserved. As the input data moves from lower layers to upper layers, each 3D max pooling layer compresses and optimizes the features, making it more robust against distortion and geometric variations. If necessary, another convolutional layer (not shown) may be present between the first and second convolutional layers (131, 132). Here, the number of kernels in the other convolutional layer may be 16, and the kernel size and stride may be 3×3×3 and 1, respectively. In addition, the output value of the other convolutional layer may be 8×55×55×55. In this case, the output of the first convolutional layer (131) is used as the input of the other convolutional layer. The output of the other convolutional layer is used as the input of the second convolutional layer (132).
[0081] In one embodiment, the deep feature learning unit (130) may utilize a dropout technique. Dropout is a method of performing normalization by randomly selecting neurons from different layers. Dropout can solve the overfitting problem of the learning model. Specifically, the deep feature learning unit (130) determines the output of a specific neuron of the hidden layer (131 to 133) to 0 according to a predetermined dropout ratio. Here, the dropout ratio indicates the probability that a certain neuron will be omitted (dropped) during the training process (i.e., the probability that a specific neuron will not be used in the forward propagation and / or backpropagation process), and can be defined such that if the value is 1, all neurons are dropped, and if it is 0, no neurons are dropped. When the dropout technique is utilized, the hidden layers (131, 132, 133) become simpler, thereby improving the training or prediction speed and reducing the required resources.
[0082] Information passing through each layer (131, 132, 133) of the deep feature learning unit (130) may be transferred to a flattening layer (134). The flattening layer (134) can convert the information transferred from the last convolutional layer (133) into one-dimensional data. Accordingly, one-dimensional deep features (139) can be obtained in which data on the relationship between slice images and / or inherent in the slice images themselves are arranged in a row. The obtained deep features (139) are transferred to the temporal feature learning unit (140).
[0083] According to one embodiment, the first learning model of the deep feature learning unit (130) may be implemented using 10 convolutional layers and 4 max pooling layers, differently from that illustrated in FIG. 4. Here, the max pooling layers may be connected to the 2nd, 4th, and 7th convolutional layers. Accordingly, some of the outputs of the convolutional layers may not be input to the max pooling layer, which is to preserve the spatial and temporal dimensions of the input volume data so that various features can be learned in both directions. Meanwhile, if the max pooling layers are partially not used, there is a possibility that the learning or prediction time may increase. To address this, each layer may be provided with the number and size of kernels of the corresponding layer optimized. For example, the first and second convolutional layers each use 8 3×3×3 kernels. The feature maps in the spatial and depth dimensions are reduced by a 2×2×2 max pooling layer with a stride of 2 and a 30% dropout layer after the max pooling layer. Meanwhile, as the spatial dimension of the feature map decreases and the depth dimension of the corresponding feature map increases, a larger number of filters are required. Accordingly, 16 filters are used in the third and fourth convolutional layers. The output of the fourth convolutional layer is input to an activation function (e.g., ReLU function), a max pooling layer, and a 30% dropout layer. 32 kernels are used in the fifth, sixth, and seventh convolutional layers. 64 kernels are used in the eighth, ninth, and tenth convolutional layers. The outputs of the seventh and tenth convolutional layers are input to a ReLU function, a 2×2×2 max pooling layer, and a 40% dropout layer, respectively. The output of the last convolutional layer (e.g., the tenth convolutional layer) is flattened into a 1D feature vector representing volume data at each time point, resulting in the acquisition of deep features (139). The acquired deep features (139) are transferred to the temporal feature learning unit (140).
[0084] Although the above deep feature learning unit (130) has described an embodiment in which a 3D convolutional neural network is used as a learning model, the first learning model is not limited thereto. Depending on the embodiment, the deep feature learning unit (130) may use a deep neural network (DNN), a recurrent neural network (RNN), a convolutional recurrent neural network (CRNN), a deep belief network (DBN), a long short term memory (LSTM), or a multi-layer perceptron, instead of or in addition to the 3D convolutional neural network.
[0085] FIG. 5 is a diagram for explaining a temporal characteristic learning unit according to one embodiment.
[0086] The temporal characteristic learning unit (140) can obtain information on the progression of a disease based on a series of time series data (N sequences of data) and output corresponding results.
[0087] The temporal feature learning unit (140) is provided to acquire relationships between volume data acquired at multiple points in time. In other words, the temporal feature learning unit (140) may be constructed to extract characteristics regarding relationships between multiple volume data (119-1, 119-2, etc.), rather than characteristics inherent in slice images within specific volume data (119) or characteristics regarding relationships between slice images. Since the multiple volume data (119-1, 119-2, etc.) are acquired at different points in time (reference point in time, first point in time, and / or second point in time, etc.), the temporal feature learning unit (140) acquires temporal characteristics regarding the volume data (119-1, 119-2, etc.) or relationships therebetween. In particular, if the depth characteristics (139) transmitted as described above are obtained from slice images of volume data (119-1, 119-2) obtained at different points in time, it is possible to obtain characteristics regarding the relationship between volume data (119-1, 119-2, etc.) at multiple points in time by utilizing these depth characteristics (139).
[0088] The temporal feature learning unit (140) can be implemented using at least one learning model (hereinafter referred to as the second learning model). Here, the second learning model may include, for example, a bidirectional recurrent neural network (BRNN) as illustrated in FIG. 5 . A recurrent neural network (RNN) is a neural network used to analyze sequential inputs and is primarily utilized in natural language processing tasks. While a recurrent neural network has the advantage of appropriately capturing temporal features within a sequence, it also suffers from the vanishing gradient problem. This can be solved by using a bidirectional recurrent neural network. A bidirectional recurrent neural network can be a neural network designed to obtain information about the past and future by connecting each layer of the recurrent neural network to the same output in opposite directions. For example, a bidirectional recurrent neural network can perform learning using deep features (139) extracted from two to three volume data (119: 119-1, 119-2) as its input. That is, the deep feature learning unit (130) inputs one or more image data (e.g., magnetic resonance imaging data at a reference time, a first time point, and a second time point) of the target object (9) into a bidirectional recurrent neural network and supplies them to a bidirectional (forward and backward) recurrent neural network. Accordingly, long-term dependencies between each data can be learned more efficiently.
[0089] As illustrated in FIG. 5, a bidirectional recurrent neural network may include a forward layer (141) and a backward layer (142), and each of the forward layer (141) and the backward layer (142) may have recurrent neural network instance(s) (141-0 to 141-n, 142-0 to 142-n, hereinafter referred to as RNN instances). For example, a bidirectional recurrent neural network may include 512 RNN instance(s) in both directions. The RNN instance(s) (141-0 to 141-n) of the forward layer (141) receives data (x_0 to x_n) in the original order. Here, the input data (x_0 to x_n) may include each value within the one-dimensional deep feature (139) described above. Meanwhile, the RNN instances (142-0 to 142-n) of the reverse layer (142) receive the same data (x_0 to x_n) in reverse order. The outputs (t_0 to t_n) of the RNN instances (141-0 to 141-n, 142-0 to 142-n) of the two layers (141, 142) are concatenated and passed to the next hidden layer. By utilizing the characteristics of the bidirectional recurrent neural network in this way, it is possible to appropriately secure the characteristics of disease progression by considering information from both past and future points in time.
[0090] In a bidirectional recurrent neural network, if the recurrent neural network has n input layer units, n hidden layer units, and n output layer units, the entire sequence of the embedded units can be obtained by repeatedly calling the following mathematical expressions 1 and 2 from the initial time point (t=1).
[0091] [Mathematical Formula 1]
[0092]
[0093] [Equation 2]
[0094]
[0095] In mathematical expression 1, x represents an input of length T, x^t_i represents the ith input at step i at time t, a^t_j represents the jth neural network input value at time t, and b^t_j represents a nonlinear activation function for the jth element at time t. is a hyperbolic tangent function, which is one of the nonlinear activation functions. a_h represents the forward hidden layer.
[0096] The output unit of a bidirectional recurrent neural network can be calculated using the following mathematical formula 3.
[0097] [Equation 3]
[0098]
[0099] The activation function can affect not only the output layer of a neural network but also the hidden layer of the next time step. This can be expressed in the following mathematical equation (4).
[0100] [Equation 4]
[0101]
[0102] At each stage, the input and output weights of the hidden layer units can be given the same weights. In this case, the derivatives of the weights of each layer can be obtained through summation, as shown in Equation 5 below.
[0103] [Equation 5]
[0104]
[0105] Above, the temporal characteristic learning unit (140) described an example of using a bidirectional recurrent neural network as the second learning model. However, the second learning model used by the temporal characteristic learning unit (140) is not limited to this. Depending on the embodiment, the second learning model may be implemented by employing other learning models, such as a recurrent neural network or a deep neural network.
[0106] In one embodiment, the learning processing unit (120) may further include a fully connected layer (FC: Fully Connected Layer, not shown). The fully connected layer may receive output values (for example, temporal features (t_0 to t_n)) from the temporal feature learning unit (140) and perform training and / or prediction operations based on the output values. In one embodiment, the fully connected layer may be implemented by including at least one of a hidden layer of 256 units and a hidden layer of 128 units. The output result of the fully connected layer is transmitted to the information fusion unit (150).
[0107] FIG. 6 is a drawing for explaining an information fusion unit according to one embodiment.
[0108] Referring to FIG. 6, the information fusion unit (150) receives the output result of the learning processing unit (120), for example, the output result (i.e., temporal feature) of the temporal feature learning unit (140) or the output result of the fully connected layer, and can fuse the output result with a value corresponding to the additional data (80) for the subject (9) (152). Here, the additional data (80) may include at least one of the cognitive score (81), the demographic characteristics (82), and the biomarker (83) as described above. Some of these (81 to 83) may be omitted as necessary. In addition to these (81 to 83), other data may also be used as the additional data. The value corresponding to the additional data (80) may include a value output by the artificial neural network (151) corresponding to the input additional data (80).
[0109] According to one embodiment, the information fusion unit (150) may obtain one or more additional data (80) from at least one of the data acquisition unit (51), the input unit (53), and the storage unit (55), and use the obtained additional data (80) as an input value of at least one learning model (151, hereinafter referred to as the third learning model). Here, the third learning model (151) may include, for example, an artificial neural network such as a deep neural network or a convolutional neural network. According to an embodiment, the artificial neural network employed as the third learning model (151) may include a fully connected layer having two layers, and in this case, the two layers may be composed of 32 units and 16 units, respectively. When various types of additional data (80) are prepared, the artificial neural network used may be the same for all the additional data (80), or may be different for all or part of the additional data (80), depending on the designer's choice, etc.
[0110] In one embodiment, the output of the third learning model (151) (i.e., the output of the last layer of the artificial neural network, which may include, for example, 16 feature vectors) may be concatenated with the output of the temporal feature learning unit (140) and fused (152), thereby obtaining a fused feature vector. The fused feature vector may be, for example, a 128-dimensional vector and a 16-dimensional vector. The fused feature vector(s) may be input to another learning model (153, hereinafter referred to as the fourth learning model). The fourth learning model (153) may be implemented using another artificial neural network (153). The other artificial neural network may be implemented including, for example, three dense layers. In this case, each dense layer may include 128, 64, and 2 hidden units, respectively. The output of the final layer (i.e., the output values of the last two units) can be used for disease analysis. More specifically, the output value based on the output of the final layer may indicate the probability of a patient developing a specific disease. The output value based on the output of the final layer may have a value between 0 and 1. If the subject (9) is a brain, this may correspond to the probability that the imaged brain will develop Alzheimer's disease. Accordingly, patients can be classified based on the probability of developing the disease.
[0111] Fig. 7 is a drawing for explaining a visualization processing unit according to one embodiment.
[0112] The visualization processing unit (160) can visualize the prediction results (spatial and time-series characteristics) finally obtained by the information fusion unit (150) in a form that users can easily understand. The visualization processing unit (160) can provide users and others with visual data that aids decision-making. Specifically, the visualization processing unit (160) highlights and localizes important areas that aid in explaining the results, enabling a visual explanation of the processing results.
[0113] The visualization processing unit (160) may be performed using, for example, 3D brain surface rendering, and / or may provide visual data according to the prediction result to the user by generating an activation map of a voxel unit of a 2D magnetic resonance image slice obtained from cross-sectional data at each time point.
[0114] 3D brain surface rendering may involve performing surface rendering to depict surface structures or organs based on slice images of the brain, thereby creating a 3D visualization of the brain. The 3D visualization may also be performed manually using a standard template to identify voxel coordinates in specific brain regions on the 3D surface. Specifically, 3D surface rendering may be performed by first acquiring brain surface data and then performing volume mapping.
[0115] The visualization processing unit (160) may, according to an embodiment, generate an attention map for each slice of the volume data. The attention map may show attention at a specific location that contributes to the context of the final prediction result. The generation of the attention map may be performed for a number of slice images corresponding to the number of channels of the volume data (for example, a total of 110). According to an embodiment, the MedCam Python library may be used to generate the attention map. Accordingly, it becomes possible to provide a two-dimensional attention map and a three-dimensional rendered brain volume that express a situation according to the progression of a specific disease (for example, atrophy of brain tissue during the progression of Alzheimer's disease).
[0116] As illustrated in FIG. 7, the visualization processing unit (160) may include at least one learning model (161, 162), a class activation map (163), and a grad-class activation map (164).
[0117] At least one learning model (161, 162) is configured to receive image data (159, for example, volume data) and perform learning or prediction based thereon. Here, the image data (159) may include all or part of the volume data (119) described above. At least one learning model (161, 162) may include, for example, two interconnected convolutional neural networks. The output results of the learning models (161, 162) may be input to a class activation map (163, CAM: Class Activation Map) and a Grad-class activation map (164, Grad-CAM), respectively.
[0118] The class activation map (163) is proposed for a specific neural network structure, and is typically used to interpret the prediction decision of a convolutional neural network under a structure in which a global average pooling layer (GAP) is applied to each feature map in the last convolutional layer. If the global average pooling layer of a convolutional neural network having a total of k feature maps is F^k (F^k ∈ R^ux v, where u is the width of the last layer, v is the height of the last layer, and x is the depth of the last layer), the weight matrix of the kth feature map can be expressed as w^k_c of the c class. Then, the prediction score (S_c) of the output layer can be given as the sum of the weight matrices of the global average pooling layers, as described in the following mathematical expression 6.
[0119] [Equation 6]
[0120]
[0121] Here, f_x(x,y) means the activation from the spatial location (x,y) within the kth feature MIP. The class activation map (163) can calculate the activation of all feature maps (M_c ∈ R^uxv) for class c as in the following mathematical expression 7.
[0122] [Equation 7]
[0123]
[0124] Since the prediction score (S_c) can be defined as the sum of the activations of all feature maps (M_c ∈ R^uxv), the class activation map (163) reflects the relevance of each spatial location (x, y) within the last class activation map. Therefore, by simply up-sampling the class activation map (163) identical to the input image data, it is possible to emphasize the region of interest (ROI) within the image so that the most important region for a specific class is more clearly revealed.
[0125] The Grad-Class Activation Map (164) is a technique for generating visual representations by utilizing gradient flow in the opposite direction to the input layer. The Grad-Class Activation Map (164) can be implemented using a single fully connected layer or dense layer instead of a global average pooling layer. Therefore, a visualization method based on the Grad-Class Activation Map (164) does not require modifying a conventional convolutional neural network to generate a visualization map.
[0126] The gradient-class activation map (164) for a specific class can be calculated as a weighted sum of k feature maps. Furthermore, a Relu function can be applied to the feature map to remove the influence of negative gradients for the class. This can be expressed as shown in Mathematical Expression 8 below.
[0127] [Equation 8]
[0128]
[0129] In mathematical expression 8, the weight a^c_k is calculated by the gradient of the prediction score (S_c) as described in mathematical expression 9 below and means the weight for the kth feature map.
[0130] [Equation 9]
[0131]
[0132] According to Equations 8 and 9, the weight a^c_k is substantially the same as the weight w^c_k in the class activation map (163), so the difference between the class activation map (163) and the grad-class activation map (164) is that spatial pixels with negative weights are excluded by using the Relu function as an activation function.
[0133] Figure 8 is a diagram illustrating images according to a guided GRAD activation map and images according to a GRAD activation map. The eight images on the left side of Figure 8 are magnetic resonance images of a cognitively normal brain, wherein the four images on the left side are images of the guided GRAD activation map, and the four images on the right side are images of the corresponding GRAD activation map. The eight images in the center of Figure 8 are magnetic resonance images of a brain in a case of progressing Alzheimer's disease, and as described above, the four images on the left side are images of the guided GRAD activation map, and the four images on the right side are images of the corresponding GRAD activation map. The eight images on the right side of Figure 8 are magnetic resonance images of a brain in a case of developing Alzheimer's disease, and likewise, the four images on the left side are images of the guided GRAD activation map, and the four images on the right side are images of the corresponding GRAD activation map.
[0134] For example, a Grad activation map may be designed to indicate important brain regions that influence Alzheimer's detection results, as shown in the images in the second, fourth, and sixth columns from the left in FIG. 8.
[0135] At least one visualized data (169) to be ultimately presented to the user is obtained based on the class activation map (163) and the grade-class activation map (164). The visualized data (169) may be stored in the storage unit (57) and / or provided to the user through the output unit (59).
[0136] Meanwhile, a guided grad activation map may be used instead of or in addition to at least one of the class activation map (163) and the grad-class activation map (164). The guided grad activation map may be given as described in the first, third, and fifth columns from the left in Fig. 8. The guided grad activation map has a disadvantage in that it does not provide clear location information for critical pixels. Therefore, in order to utilize the guided grad activation map, it is necessary to adopt a guided backpropagation approach to obtain accurate voxel location information that influences the decision process. In this case, the gradient (m_c) of the prediction score (y^c) for a specific class with respect to the input image data (x) may be given as in the following mathematical expression (10).
[0137] [Equation 10]
[0138]
[0139] In the guided backpropagation method, positive values of the gradient are backpropagated, and negative values of the gradient are set to 0. If G^l_i is the gradient backpropagated through the lth layer, backpropagation can be defined by the following mathematical expression (11).
[0140] [Equation 11]
[0141]
[0142] In Equation 11, f^l_i is the result of applying f^(l-1)_i to the ReLU function. Once the highlighted region is generated for a specific backpropagation, attention mapping according to the aforementioned gradient (m_c) is used to visualize each pixel.
[0143] Figures 9 to 11 are diagrams of images that can be visually presented to a user to detect the progression of Alzheimer's disease. Figure 9 is an image corresponding to a normal state, Figure 10 is an image corresponding to a state progressing to Alzheimer's disease, and Figure 11 is an image corresponding to a situation in which Alzheimer's disease has developed. Each image of Figures 9 to 11 was acquired based on the MedCam described above, and represents voxel details and 2D slices.
[0144] As shown in Figures 9 to 11, when each important area is visualized, the user can more easily determine the possibility of disease occurrence.
[0145] Referring to Figures 9 to 11, the brain regions that can easily distinguish between normal patients and Alzheimer's patients are the rostral hippocampus, medial amygdala, globus pallidus, lateral amygdala, entorhinal cortex (EC), and parahippocampal gyrus. As shown in Figure 9, in the case of normal patients, the corresponding regions do not change over time. That is, they appear the same in the scan results at each time point. As shown in Figure 10, in the case of patients with progressing Alzheimer's, the number and volume of affected brain regions can be expressed separately using separate colors or lines, and it is visually shown that these regions increase rapidly over time. That is, it is easy to see that brain tissue atrophy occurs at each time point. Meanwhile, in the case of Alzheimer's patients, as shown in Figure 11, the visualization shows that many brain regions are affected. In particular, in patients with pre-existing Alzheimer's disease at baseline, numerous brain lesions were already evident in the first scan. Using the images acquired by the visualization processing unit (160), doctors can easily and intuitively track and predict how a patient's condition has changed over time.
[0146] The disease analysis device (50) described above may be implemented using a device specifically designed to perform the processing such as the above-described calculation or control, or may be implemented by using one or more information processing devices alone or in combination. According to one embodiment, one or more information processing devices may include a medical device. The medical device may be, for example, at least one of an X-ray imaging device, a mammography device, a computed tomography device, and a magnetic resonance imaging device. In addition, one or more information processing devices may be, depending on the embodiment, a desktop computer, a laptop computer, a hardware device for a server, an artificial intelligence computing device, a smart phone, a tablet PC, a smart watch, a smart band, a head-mounted display (HMD) device, a portable game console, a navigation device, a voice recorder, a remote control device (remote control), a digital television, a set-top box, a media streaming device, an audio reproduction device (such as an artificial intelligence speaker), a home appliance, a manned or unmanned vehicle (such as a passenger car, a bus, a two-wheeled vehicle, a mobile robot, or a wireless model vehicle), a manned or unmanned aerial vehicle (such as an aircraft or helicopter), a robot, or an industrial machine. In addition, a designer or a user may employ at least one of various devices capable of computational processing and control of image data, etc., as the disease analysis device (50) described above, depending on the situation or conditions.
[0147] Referring to FIG. 12 below, one embodiment of a disease analysis method will be described.
[0148] Referring to FIG. 12, at least one learning model prepared for disease analysis can be trained prior to disease analysis (200). Training of the learning model can be performed by the disease analysis device described above or another information processing device separately prepared for training the learning model. Here, the at least one learning model can include at least one of a first learning model (e.g., a 30-dimensional convolutional neural network) prepared for acquiring characteristics of slice images within volume data, as described below, a second learning model (e.g., a bidirectional recurrent neural network) prepared for acquiring characteristics related to volume data, a third learning model (artificial neural network) prepared for additional data, and a fourth learning model (artificial neural network) prepared for fused feature vectors. Furthermore, depending on the embodiment, the at least one learning model may further include another learning model in addition to these. Training of the learning model can be performed using separately prepared training data (e.g., magnetic resonance images or tomography images obtained from an existing patient). Here, the training data may include image data captured at the reference point, the first point, and the second point. Additionally, the learning model may be validated using validation data. In this case, a portion of the training data may be used as validation data.
[0149] Data for analyzing a disease of a subject are collected and acquired (210). Data collection can be performed through at least one device capable of collecting data. Here, the device capable of collecting data may include a smartphone or a camera device. In addition, the device capable of collecting data may include a medical device. The medical device may be, for example, at least one of a magnetic resonance imaging device, an ultrasound imaging device, an X-ray imaging device, a mammography device, a computed tomography device, and a positron emission tomography device. The disease analysis device can acquire data from these devices. Depending on the embodiment, the disease analysis device may also directly perform data collection. Data collection may be performed at multiple points in time. For example, data collection may be performed at a reference point in time, a first point in time, a second point in time, etc. Accordingly, multiple data corresponding to each point in time can be acquired.
[0150] Once data is collected, preprocessing may be performed on the collected data (220). The data to be preprocessed may include magnetic resonance imaging data, and the magnetic resonance imaging data may be volume data composed of one or more slice images. Data preprocessing may include at least one of, for example, image enlargement, image reduction, image orientation adjustment, alignment of multiple images, removal of noise or artifacts, bias field correction, removal of unnecessary parts (e.g., skulls), standard template registration, and selection of one or more images suitable for analysis from among multiple images (e.g., slice images). If necessary, data preprocessing may further include various other processing(s) necessary for the data in addition to these. Data preprocessing may also be omitted depending on the embodiment.
[0151] Deep features can be acquired for at least one data (230). In one embodiment, the at least one data may include at least one slice image of a magnetic resonance image (volume data). In this case, the acquired deep features may include at least one of a feature inherent in the at least one slice image (i.e., an intra-slice feature) and a feature regarding the relationship between multiple slice images within at least one volume data (i.e., an inter-slice feature). The deep features can be acquired based on at least one first learning model. Meanwhile, when data (e.g., volume data) are acquired from multiple viewpoints, all or part of the data from the multiple viewpoints may be input together into the first learning model to acquire the deep features. Accordingly, the deep features are extracted based on each image data acquired from different viewpoints.
[0152] The first learning model may include, according to an embodiment, a three-dimensional convolutional neural network. The three-dimensional convolutional neural network may include a plurality of convolutional layers (the number of kernels corresponding to each layer may be 8, 32, and 64, respectively, the size of the kernels may be 3×3×3, and the stride may be given as 1). The size of the output value of each convolutional layer may be, for example, 8×110×110×110, 32×27×27×27, and 64×13×13×13. The activation function of each convolutional layer may be, for example, a Relu function. According to an embodiment, at least one of all the convolutional layers may be connected to a pooling layer. Here, the pooling layer may be implemented based on three-dimensional max pooling. In addition, a dropout technique may be applied to at least one convolutional layer. The output of the final convolutional layer is passed to a smoothing layer, which acquires one-dimensional deep features. These can be converted into one-dimensional data. These one-dimensional deep features are inherent in the slice images themselves or contain information about the relationships between slice images.
[0153] Next, characteristics regarding the relationship between a plurality of image data (volume data) acquired at different points in time can be acquired (240). That is, temporal characteristics regarding the plurality of volume data or their relationships are acquired. According to an embodiment, some of the plurality of volume data may be acquired at the same point in time. Acquisition of temporal characteristics may be performed based on a second learning model. The second learning model may include, for example, a bidirectional recurrent neural network. The bidirectional recurrent neural network is designed to perform learning by inputting the same input values to the forward layer and the backward layer, respectively. The one-dimensional deep characteristics acquired as described above are provided to the bidirectional recurrent neural network, and temporal characteristics between the plurality of volume data can be acquired correspondingly. The output of the bidirectional recurrent neural network may, according to one embodiment, be transmitted to a fully connected layer.
[0154] The output of a bidirectional recurrent neural network or the output of a fully connected layer is fused with additional data about a subject or a corresponding value (250). Accordingly, a fused feature vector can be obtained. Here, the value corresponding to the additional data may include an output value obtained by inputting the additional data into at least one third learning model. Here, the third learning model may include an artificial neural network, and the artificial neural network may be a fully connected layer having two layers. The fused feature vector can be passed through a fourth learning model that can be implemented using an artificial neural network, etc. The output of the fourth learning model may indicate the probability of a patient developing a specific disease. For example, the output of the fourth learning model may have a value between 0 and 1.
[0155] Finally, a cubization process corresponding to the acquired disease analysis results is performed, and the visualization processing results can be provided to the user (260). Here, the visualization process may be implemented by performing 3D brain surface rendering or generating an activation map. In one embodiment, the visualization process may input image data into at least one learning model. The visualization process may be performed by inputting the output of the learning model into at least one of a class activation map and a grade-class activation map.
[0156] The disease analysis method according to the above-described embodiment can also be implemented in the form of a program that can be run by a computer device. Here, the program can include commands, libraries, data files, and / or data structures, either singly or in combination, and can be designed and produced using machine language code or high-level language code. The program can be specifically designed to implement the above-described method, or can be implemented using various functions or definitions that are known and available to those skilled in the art in computer software. The computer device can be implemented by including a processor or memory that enable the functions of the program to be realized, and can further include a communication device as needed. The program for implementing the above-described disease analysis method can be recorded on a recording medium readable by a device such as a computer. A computer-readable recording medium may include at least one type of physical storage medium capable of temporarily or non-transitorily storing one or more programs executed upon calling of a device such as a computer, such as a semiconductor storage medium such as a ROM, a RAM, an SD card, or a flash memory (e.g., a solid state drive (SSD)), a magnetic disk storage medium such as a hard disk or a floppy disk, an optical storage medium such as a compact disk or a DVD, or a magneto-optical storage medium such as a floptical disk.
[0157] Although various embodiments of the disease analysis device and the disease analysis method have been described, the disease analysis device or the disease analysis method is not limited to the embodiments described above. Other various devices or methods that can be implemented by a person having ordinary skill in the art by modifying and changing the above-described embodiments may also be embodiments of the disease analysis device or the disease analysis method described above. For example, even if the described method(s) are performed in a different order than described, and / or the described system, structure, device, circuit, etc. component(s) are combined, connected, or combined in a different form than described, or are replaced or substituted by other components or equivalents, the disease analysis device and / or the disease analysis method described above may still be embodiments.
[0158] Those skilled in the art will appreciate that the embodiments of the present invention can be implemented in modified forms without departing from the essential characteristics of the above description. Therefore, the disclosed methods should be considered illustrative rather than restrictive. The scope of the present invention is determined by the claims, not the detailed description, and all differences within the scope equivalent thereto should be construed as being included within the scope of the present invention.
[0159]
[0160] The present invention has high industrial applicability because it can more accurately and effectively detect, judge, or predict the condition of a lesion or changes thereof on a subject using a disease analysis device and method.
[0161]
[0162] [Explanation of symbols]
[0163] 10: Data collection device 50: Disease analysis device
[0164] 100: Processor 110: Data preprocessing unit
[0165] 120: Learning processing unit 130: Deep feature learning unit
[0166] 140: Temporal Characteristics Learning Department 150: Information Convergence Department
[0167] 160: Visualization Processing Unit
Claims
1. A data acquisition unit that acquires multiple volume data from a target object at multiple points in time, each of the multiple volume data including at least one slice image; and A disease analysis device comprising a processor for extracting a depth feature from at least one slice image based on a first learning model, obtaining a temporal feature for a relationship between a plurality of volume data acquired at a plurality of time points based on a second learning model using the depth feature, and obtaining a result for disease analysis using the feature for the relationship between the plurality of volume data.
2. In paragraph 1, A disease analysis device wherein the above-mentioned deep characteristic includes at least one of a characteristic inherent in the at least one slice image and a characteristic regarding a relationship between a plurality of slice images within the volume data.
3. In paragraph 1, The processor obtains a fused feature vector by fusing the temporal characteristic with additional data or a value corresponding to the additional data, wherein the value corresponding to the additional data includes an output value of a third learning model corresponding to the input of the additional data, The above processor is a disease analysis device that obtains results for disease analysis by using the output value of the fourth learning model according to the input of the fused feature vector.
4. In paragraph 3, The above additional data is a disease analysis device including at least one of a cognitive score, demographic characteristics, and biomarkers.
5. In paragraph 1, The above processor is a disease analysis device that performs visualization processing corresponding to the results for disease analysis.
6. In paragraph 5, A disease analysis device wherein the visualization processing comprises at least one of three-dimensional brain surface rendering and generation of an activation map.
7. In paragraph 5, The above processor is a disease analysis device that performs the visualization processing by inputting the output of at least one learning model corresponding to the input of the plurality of volume data into a class activation map and a grad-class activation map.
8. In paragraph 1, A disease analysis device wherein the first learning model includes a three-dimensional convolutional neural network, and the second learning model includes a bidirectional recurrent neural network.
9. A step of acquiring multiple volume data obtained from a target object at different times; A step of extracting deep features from at least one slice image within the plurality of volume data based on a first learning model; A step of obtaining temporal characteristics of the relationship between the plurality of volume data using the deep characteristics based on the second learning model; and A disease analysis method comprising: a step of obtaining results for disease analysis by using characteristics of relationships between the plurality of volume data; 10. In paragraph 9, A disease analysis method wherein the above-mentioned deep feature includes at least one of a feature inherent in the at least one slice image and a feature regarding a relationship between a plurality of slice images within the volume data.
11. In paragraph 9, A step of obtaining a fused feature vector by fusing the temporal characteristic with additional data or a value corresponding to the additional data, wherein the value corresponding to the additional data includes an output value of a third learning model corresponding to the input of the additional data; and A disease analysis method further comprising a step of obtaining a result for the disease analysis by using the output value of the fourth learning model according to the input of the fused feature vector.
12. In paragraph 11, A disease analysis method wherein the above additional data comprises at least one of a cognitive score, demographic characteristics, and biomarkers.
13. In paragraph 9, A disease analysis method further comprising a step of performing visualization processing corresponding to the results for disease analysis.
14. In paragraph 13, A disease analysis method wherein the visualization processing comprises at least one of three-dimensional brain surface rendering and generation of an activation map.
15. In paragraph 13, The step of performing visualization processing corresponding to the results for the above disease analysis is: A disease analysis method further comprising: a step of inputting the output of at least one learning model into a class activation map and a grad-class activation map in response to the input of the plurality of volume data; 16. In paragraph 9, A disease analysis method wherein the first learning model includes a three-dimensional convolutional neural network, and the second learning model includes a bidirectional recurrent neural network.
Citation Information
Patent Citations
A separator for an electrochemical device and a method for manufacturing the same
KR1020200049649A
System for providing location based and bigdata used total platform service customized for cabin crew and applicant
KR1020240003766A
Solar photovoltaic system for orchard
KR102272495B1
KR20200125131A
