Model training method, classification method and related apparatus
By using global cue learning vectors to initialize and fine-tune the mild cognitive impairment model, the problem of high classification difficulty in mild cognitive impairment is solved, and the model transfer efficiency is improved and knowledge is effectively transferred.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN RES INST OF BIG DATA
- Filing Date
- 2023-08-25
- Publication Date
- 2026-06-09
AI Technical Summary
Existing models perform poorly on mild cognitive impairment tasks, mainly due to limited data and the lack of significant differences between MRI images and clinical attribute data in mild cognitive impairment, which increases the difficulty of classification. Existing transfer methods are inefficient and prone to forgetting learned knowledge.
By acquiring the first training dataset, including MRI images and clinical data, an Alzheimer's classification model was trained, and a progressive mild cognitive impairment classification model was initialized and fine-tuned using global cueing learning vectors. The model parameters were frozen, and only the global cueing learning vectors were adjusted to complete the parameter transfer.
This improves the efficiency of model transfer from Alzheimer's disease to mild cognitive impairment, ensuring the effectiveness of knowledge transfer and the transferability of the model.
Smart Images
Figure CN117079046B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image recognition technology, specifically relating to a model training method, a classification method, and related devices. Background Technology
[0002] Currently, many deep learning methods have been used to extract features from brain MRI images to build Alzheimer's diagnostic models. Other works use MRI images and clinical tabular data, employing convolutional neural networks and multilayer perceptrons to extract image and tabular features, respectively.
[0003] However, existing models do not perform well on mild cognitive impairment (MCI) tasks. This is partly due to limited data, and partly because there are no significant differences between the two types of MCI in MRI images and clinical attributes, increasing the difficulty of classification. Based on this, some studies have transferred weights learned from Alzheimer's disease diagnostic models to MCI models for initialization, and then fine-tuned them on training data from MCI patients. Compared to random initialization, this transfer method can apply knowledge learned from Alzheimer's disease classification tasks to the classification of MCI, resulting in better classification performance. However, this full fine-tuning method suffers from low transfer efficiency, requiring fine-tuning of all model parameters, which can easily lead to forgetting learned knowledge and reducing the model's transferability. Summary of the Invention
[0004] This application provides a model training method, a classification method, and related apparatus to improve the efficiency of model transfer.
[0005] Firstly, this application provides a model training method, including:
[0006] Obtain a first training dataset, the first training dataset including a first image information subset and a first attribute data subset, the first image information subset including a first MRI image to be trained, and the first attribute data subset including first clinical data corresponding to the first MRI image;
[0007] The first training dataset is input into the first model for training to obtain the Alzheimer's classification model. The first model refers to the Alzheimer's classification model to be trained.
[0008] The second model is initialized based on the first model parameters of the Alzheimer's classification model, where the second model refers to the progressive mild cognitive impairment classification model to be trained.
[0009] Freeze the parameters of the first model in the second model after initialization;
[0010] Determine the global cue learning vector;
[0011] The second training set is input into the second model to fine-tune the global cue learning vector, thereby obtaining a progressive mild cognitive impairment classification model. The cue fine-tuning refers to adjusting only the parameters corresponding to the global cue learning vector in the second model during the training process.
[0012] Secondly, this application provides a medical image classification method based on cue learning, including:
[0013] Acquire target image information and target attribute data, wherein the target image information refers to the MRI image to be classified, and the target attribute data is the clinical data corresponding to the MRI image to be classified;
[0014] Image registration is performed on the target image information to obtain the image information to be processed;
[0015] The target clinical data were normalized to obtain the clinical data to be processed.
[0016] The image information to be processed and the clinical data to be processed are input into the progressive mild cognitive impairment classification model;
[0017] The progressive mild cognitive impairment classification model acquires target attribute data of the target user and processes the target attribute data to obtain a corresponding first global feature sequence; acquires target image information of the target user and processes the target image information according to the global cue learning vector to obtain a second global feature sequence; performs multimodal feature fusion on the first global feature sequence and the second global feature sequence to obtain multimodal features; and predicts the disease type indicated by the target attribute data and the target image information based on the multimodal features.
[0018] Thirdly, this application provides a model training apparatus, comprising:
[0019] The acquisition unit is used to acquire a first training dataset, the first training dataset including a first image information subset and a first attribute data subset, the first image information subset including a first MRI image to be trained, and the first attribute data subset including first clinical data corresponding to the first MRI image;
[0020] The first training unit is used to input the first training dataset into the first model for training to obtain an Alzheimer's classification model, wherein the first model refers to the Alzheimer's classification model to be trained.
[0021] An initialization unit is used to initialize a second model based on the first model parameters of the Alzheimer's classification model, wherein the second model refers to the progressive mild cognitive impairment classification model to be trained.
[0022] Freeze unit, used to freeze the parameters of the first model in the initialized second model;
[0023] The defined unit is used to determine the global cue learning vector;
[0024] The second training unit is used to input the second training set into the second model to fine-tune the global cue learning vector to obtain a progressive mild cognitive impairment classification model. The cue fine-tuning refers to adjusting only the parameters corresponding to the global cue learning vector in the second model during the training process.
[0025] Fourthly, this application provides a classification model for progressive mild cognitive impairment, including:
[0026] The clinical data processing module is used to acquire target attribute data of the target user and process the target attribute data to obtain the corresponding first global feature sequence.
[0027] The image information processing module is used to acquire the target image information of the target user, process the target image information according to the global prompting learning vector, and obtain the second global feature sequence.
[0028] The modality fusion module is used to perform multimodal feature fusion on the first global feature sequence and the second global feature sequence to obtain multimodal features;
[0029] A classification module is used to predict the disease type indicated by the target attribute data and the target image information based on the multimodal features.
[0030] Fifthly, this application provides an electronic device including a processor, a memory, a communication interface, and one or more programs, said one or more programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of any one of the first to fourth aspects of this application.
[0031] In a sixth aspect, this application provides a computer storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in any one of the first to fourth aspects of this application.
[0032] In a seventh aspect, this application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in any one of the first to fourth aspects of this application. The computer program product may be a software installation package.
[0033] As can be seen, in this application, a first training dataset, including a first subset of image information and a first subset of attribute data, is first obtained; the first training dataset is input into a first model for training to obtain an Alzheimer's classification model; a second model is initialized based on the first model parameters of the Alzheimer's classification model, where the second model refers to the progressive mild cognitive impairment classification model to be trained; the first model parameters in the initialized second model are frozen; a global cue learning vector is determined; and the second training set is input into the second model to fine-tune the global cue learning vector, resulting in the progressive mild cognitive impairment classification model. In this way, parameter transfer from the Alzheimer's classification module to the progressive mild cognitive impairment model can be completed simply by fine-tuning the global cue learning vector, improving the efficiency of model transfer. Attached Figure Description
[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0035] Figure 1a This is a schematic diagram of the structure of an Alzheimer's classification module provided in an embodiment of this application;
[0036] Figure 1b This is a schematic diagram of the structure of a progressive mild cognitive impairment classification module provided in an embodiment of this application;
[0037] Figure 1c This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;
[0038] Figure 2a This is a schematic flowchart of a model training method provided in an embodiment of this application;
[0039] Figure 2b This is a flowchart illustrating a medical image classification method based on cue learning provided in an embodiment of this application;
[0040] Figure 3 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application. Detailed Implementation
[0041] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0042] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, systems, products, or apparatuses.
[0043] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0044] Currently, due to limited data and the lack of significant differences between the two types of mild cognitive impairment in MRI images and clinical attributes, classification becomes more difficult, and existing models do not perform well on tasks related to mild cognitive impairment.
[0045] To address the aforementioned issues, this application provides a system-on-a-chip (SoC) comprising a microprocessor and a baseband chip. This SoC can be applied to scenarios where a probe reference signal is generated based on scheduling information from a network device. The microprocessor can acquire scheduling information from the network device for a first probe reference signal; respond to the scheduling information to acquire attribute information of the first probe reference signal; determine a first acquisition method based on the attribute information; acquire a first parameter based on the determined first acquisition method; generate a configuration instruction based on the first parameter; and send the configuration instruction to the baseband chip. The baseband chip then receives the configuration instruction, generates the first probe reference signal based on the configuration instruction, and sends the first probe reference signal to the network device. This solution is applicable to various scenarios, including but not limited to the application scenarios mentioned above.
[0046] The system architecture involved in the embodiments of this application is described below.
[0047] like Figure 1b As shown, this application provides a classification model 100 for progressive mild cognitive impairment. The classification model includes: a first clinical data processing module 110, used to acquire target attribute data of a target user and process the target attribute data to obtain a corresponding first global feature sequence; a first image information processing module 120, used to acquire target image information of the target user and process the target image information to obtain a second global feature sequence; a first modality fusion module 130, used to perform multimodal feature fusion on the first global feature sequence and the second global feature sequence to obtain multimodal features; and a first classification module 140, used to predict the disease type indicated by the target attribute data and the target image information based on the multimodal features. Specifically, the first clinical data processing module 110 includes a first embedding layer 111 and a first attribute encoder 112; the first embedding layer 111 is used to acquire the target attribute data and extract a first feature sequence from the target attribute data; the first attribute encoder 112 is used to process the first feature sequence and a global cue learning vector to obtain the first global feature sequence. Specifically, the first image information processing module 120 includes a second embedding layer 121 and a first image encoder 122; the second embedding layer 121 is used to acquire the target image information and extract a second feature sequence from the target image information; the first image encoder 122 is used to process the second feature sequence and the global cue learning vector to obtain a second global feature sequence. Further, the first image encoder 122 includes a spatial dimension attention unit and a channel dimension attention unit; the spatial dimension attention unit is used to perform weighted averaging on the second feature sequence and the global cue learning vector to obtain a third feature sequence; the channel dimension attention unit is used to perform weighted averaging on the second feature sequence to obtain a fourth feature sequence; the third feature sequence and the fourth feature sequence are added together to obtain a fifth feature sequence; the fifth feature sequence is multiplied by the global cue learning vector to obtain a sixth feature sequence; the sixth feature sequence is added to the target image information to obtain the second global feature sequence.
[0048] This application also provides an electronic device 150, such as... Figure 1cAs shown, it includes at least one processor 151, a display screen 152, and a memory 153, and may also include a communications interface 155 and a bus 154. The processor 151, display screen 152, memory 153, and communications interface 155 can communicate with each other via the bus 154. The display screen 152 is configured to display a preset user guide interface in the initial setup mode. The communications interface 155 can transmit information. The processor 151 can call logical instructions in the memory 153 to execute the methods described in the above embodiments.
[0049] Optionally, the electronic device 150 may be a mobile electronic device, an electronic device, or other devices, and is not limited to any particular type.
[0050] Furthermore, the logic instructions in the aforementioned memory 153 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.
[0051] The memory 153, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 151 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 153, thereby implementing the methods in the above embodiments.
[0052] The memory 153 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the electronic device 150. Furthermore, the memory 153 may include high-speed random access memory (RAM) and may also include non-volatile memory. For example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, may be used, or they may be transient storage media.
[0053] The specific methods will be described in detail below.
[0054] Please see Figure 2a This application also provides a model training method, including:
[0055] Step 201: Obtain the first training dataset.
[0056] The first training data includes a first image information subset and a first attribute data subset. The first image information subset includes a first MRI image to be trained, and the first attribute data subset includes first clinical data corresponding to the first MRI image.
[0057] For example, the first image information subset includes multiple first MRI images, and the first attribute data subset includes attribute data corresponding to the multiple first MRI images. The attribute data includes, but is not limited to, ApoE4 (ApoE4 gene protein), P-tau181 (phosphorylated tau 181 protein), and T-tau (total tau protein), including numerical data and categorical data. MRI images include, but are not limited to, T1, T2, T1CE, and FLAIR sequences for brain imaging; wherein, T1 refers to T1-weighted images, which mainly reflect the differences in T1 signal intensity between tissues (good for displaying anatomical structures); T2 refers to T2-weighted images, which mainly reflect the differences in T2 signal intensity between tissues (sensitive for displaying lesions); T1CE refers to the T1CE sequence, used to observe the internal condition of tumors and differentiate between tumors and non-tumor lesions; and the FLAIR sequence is used to observe the surrounding conditions of the lesion site.
[0058] Step 202: Input the first training dataset into the first model for training to obtain the Alzheimer's classification model.
[0059] The first model refers to the Alzheimer's classification model to be trained.
[0060] In specific implementation, such as Figure 1a As shown, a first model 10 is pre-created, which includes a second clinical data processing module 11, a second image information processing module 12, a second modality fusion module 17, and a second classification module 18. The first model 10 is trained based on a first training dataset to obtain an Alzheimer's classification model. Specifically, feature sequences corresponding to Alzheimer's clinical data in the first training dataset (such as...) are extracted through a third embedding layer 13. Figure 1a The data is then processed by the second attribute encoder 14 to obtain the modality sequence corresponding to the Alzheimer's clinical data (e.g., "A" in the original text). Figure 1a The “B” in the first training dataset is input into the second modality fusion module 17; simultaneously, the fourth embedding layer 15 extracts the feature sequences corresponding to the Alzheimer’s nuclear magnetic resonance images in the first training dataset (such as…). Figure 1a The second image encoder 16 processes the "C" in the image to obtain the modal sequence corresponding to the Alzheimer's MRI image (e.g., the "C" in the image). Figure 1aIn the second modality fusion module 17, the “D” is input into the category training dataset E to perform image type and image matching training, and obtains multimodal feature vectors (such as…). Figure 1a The “F” in the model is used by the second classification module to predict the disease type indicated by Alzheimer’s clinical data and Alzheimer’s MRI images based on multimodal feature vectors. If the prediction result is accurate, the training is completed and an Alzheimer’s classification model is obtained. If the prediction result is inaccurate, iterative training is performed until the prediction result is accurate.
[0061] Step 203: Initialize the second model according to the first model parameters of the Alzheimer's classification model.
[0062] The second model refers to the progressive mild cognitive impairment classification model to be trained.
[0063] In the specific implementation, a second model is pre-created. The second model includes a clinical data processing module, an image information processing module, a modality fusion module, and a classification module. The difference between the second model and the first model is that the clinical data processing module and the image information processing module are equipped with global prompts, and the image information processing module is equipped with a high-efficiency matching attention module based on global prompts. The high-efficiency matching attention module based on global prompts includes spatial dimension attention units and channel dimension attention units.
[0064] Specifically, the model parameters corresponding to the trained Alzheimer's classification model are transferred to the second model, and then fine-tuned based on the global cue learning vector to adapt to the identification of progressive mild cognitive impairment. After processing by spatial dimension attention units and channel dimension attention units, a progressive mild cognitive impairment classification model is obtained.
[0065] Step 204: Freeze the parameters of the first model in the initialized second model.
[0066] In practice, after configuring the second model using the first model parameters, the configured first model parameters in the second model are frozen, and these parameters are not changed during subsequent training and use.
[0067] Step 205: Determine the global cue learning vector.
[0068] In the specific implementation, a global cue learning vector is randomly set. This global cue learning vector is a series of learnable parameters related to progressive mild cognitive impairment and can be regarded as part of the second model. During subsequent training of the second model, only the global cue learning vector is updated.
[0069] Step 206: Input the second training set into the second model to fine-tune the global cue learning vector to obtain a progressive mild cognitive impairment classification model.
[0070] The aforementioned "cue fine-tuning" refers to adjusting only the parameters corresponding to the global cue learning vector in the second model during the training process.
[0071] In one possible embodiment, inputting a second training set into the second model to fine-tune the global cue learning vector to obtain a classification model for progressive mild cognitive impairment includes: acquiring a second training dataset, the second training dataset comprising a second image information subset and a second attribute data subset, the second image information subset comprising a second MRI image to be trained, and the second attribute data subset comprising second clinical data corresponding to the second MRI image; inputting the second training dataset into the second model; fine-tuning the global cue learning vector in the second model according to the second training dataset to obtain a classification result; determining whether the classification result of the second model is accurate; if the classification result is inaccurate... The classification loss function is calculated, and a new global cue learning vector is obtained by adjusting it using the stochastic gradient descent algorithm. The third training dataset is input into the second model to fine-tune the new global cue learning vector, resulting in a new classification result. The third training dataset includes a third image information subset and a third attribute data subset. The third image information subset includes the third MRI image to be trained, and the third attribute data subset includes the third clinical data corresponding to the third MRI image. If the new classification result is inaccurate, the above training operation is repeated until an accurate classification result is obtained. The model parameters corresponding to the accurate classification result on the second model are frozen to obtain the progressive mild cognitive impairment classification model.
[0072] Specifically, training the second model based on the global cue learning vector and the second training dataset to obtain a classification result includes: inputting the second attribute data subset into the clinical data processing module of the second model; the clinical data processing module extracting a first global feature sequence from the second attribute data subset; inputting the global cue learning vector and the second image information subset into the image information processing module; the image information processing module extracting a second global feature sequence from the global cue learning vector and the second image information subset; inputting the first global feature sequence and the second global feature sequence into the modality fusion module; the modality fusion module performing modality fusion on the first global feature sequence and the second global feature sequence to obtain a multimodal feature sequence; inputting the multimodal feature sequence into the classification module; and the classification module predicting the classification result based on the multimodal features.
[0073] Furthermore, the image information processing module includes a spatial dimension attention unit and a channel dimension attention unit; the image information processing module extracts a second global feature sequence from the second image information subset, including: obtaining the target image information by the second embedding layer and extracting a second feature sequence from the target image information; processing the second feature sequence and the global cue learning vector by the image encoder to obtain a second global feature sequence; performing weighted averaging on the second feature sequence and the global cue learning vector by the spatial dimension attention unit to obtain a third feature sequence; performing weighted averaging on the second feature sequence by the channel dimension attention unit to obtain a fourth feature sequence; adding the third feature sequence and the fourth feature sequence to obtain a fifth feature sequence; performing a dot product operation on the fifth feature sequence and the global cue learning vector to obtain a sixth feature sequence; and adding the sixth feature sequence to the target image information to obtain the second global feature sequence.
[0074] In one possible embodiment, the modality fusion module performs modality fusion on the first global feature sequence and the second global feature sequence to obtain a multimodal feature sequence, including: setting a category training dataset, the category training dataset including multiple image category data; extracting a seventh feature sequence from the category training dataset; and performing modality fusion on the seventh feature sequence, the first global feature sequence, and the second global feature sequence to obtain the multimodal feature sequence.
[0075] In specific implementation, such as Figure 1b As shown, in this embodiment, the clinical data processing module includes a first embedding layer and an attribute encoder; the attribute encoder includes a self-attention transformer module, which includes at least one transformer layer, and a set of global cue learning vectors is inserted into the input of each transformer layer in the attribute encoder. For the i-th layer of the self-attention transformer module... In this embodiment, The set of global cue learning vectors is defined, where k represents the number of global cue learning vectors in the set. The training process based on global cue learning vectors can be expressed by the formula:
[0076]
[0077]
[0078] in, The characteristics representing table attributes are in The output, Hints indicating table attributes are in input space and splicing.
[0079] Specifically, a second training dataset is first obtained. This second training dataset consists of pre-collected historical data, such as second MRI images of patients with progressive mild cognitive impairment treated in hospitals, and the corresponding second clinical data. For new hospitals, there is not enough clinical data (including image and tabular data). The weights obtained on the Alzheimer's disease classification task can be obtained from other hospitals without infringing on the patients' privacy.
[0080] Furthermore, after obtaining the second training dataset, the second training dataset is input into the second model. The first embedding in the clinical data processing module extracts the corresponding table attribute features from the second clinical data. Then, the global cue learning vector and the table attribute features are input into the attribute encoder. The attribute encoder concatenates the global cue learning vector and the table attribute features to obtain the first global feature sequence.
[0081] For the image cue learning part, in this embodiment, the image information processing module includes a second embedding layer and an image encoder. The image encoder includes an efficient pairing attention module based on cue fine-tuning, which includes a spatial dimension attention unit and a channel dimension attention unit. Specifically, a small number of global cue learning vectors with visual embedding are concatenated. Simultaneously, in the efficient pairing attention block with cue fine-tuning, shared global cue learning vectors are inserted into the spatial dimension attention module (SWA) and the channel dimension attention module (CWA), respectively. Under the shared key-query scheme, the query vector, key vector, and value vector are denoted as... , , and .set up For connection operations, SWA and CWA can be described as follows:
[0082]
[0083]
[0084] here and These are image features and cues in the spatial dimension. and These represent the image features and cues at the channel dimension, respectively, where P is the number of visual cues. Then, skip connections are used to sum the feature maps. This process can be described as follows:
[0085]
[0086] To enable cue symbols to influence global features and interact more efficiently with visual features, a global cue symbol is set up to achieve these two goals. Specifically, a linear transformation T is applied to the input cue symbol P to obtain the global cue symbol g (i.e., a scalar), which is then multiplied by the global feature map. Since the scalar is directly multiplied by the global features, the global feature responses of each layer of the visual network can be found more effectively, allowing the model to better focus on some global features important for the diagnosis of progressive mild cognitive impairment, such as cortical atrophy. Due to this linear transformation, the T operation is learnable during the cue training phase, and the original cue symbol can interact better with other features through the global symbol. To embed the global cue symbol into our framework, we rewrite the formula for calculating the visual feature map as follows: , where T denotes the linearly changing layer.
[0087] In specific implementation, the second embedding layer extracts a second feature sequence from the second image information subset, and then simultaneously inputs the second feature sequence and the global cue learning vector into the spatial dimension attention unit and the channel dimension attention unit. The spatial dimension attention unit performs weighted averaging on the second feature sequence and the global cue learning vector to obtain a third feature sequence; the channel dimension attention unit performs weighted averaging on the second feature sequence to obtain a fourth feature sequence; the third feature sequence and the fourth feature sequence are added together to obtain a fifth feature sequence; the fifth feature sequence is dot-productd with the global cue learning vector to obtain a sixth feature sequence; and the sixth feature sequence is added with the target image information to obtain the second global feature sequence.
[0088] Specifically, after obtaining the first global feature vector (denoted as the i-th first global feature vector) and the second global feature vector (denoted as the i-th second global feature vector), the modality fusion module performs modal fusion on the first global feature vector and the second global feature vector to obtain a multimodal feature sequence; the multimodal feature sequence is input into the classification module; the classification module predicts the classification result based on the multimodal features, denoted as the i-th classification result.
[0089] If the i-th classification result is accurate, then the loss function of the i-th classification result is calculated, and the global cue learning vector is adjusted by the stochastic gradient descent algorithm to obtain a new global cue learning vector.
[0090] The new global cue learning vector and the third training dataset are then input into the (i+1)th training unit. The clinical data processing module of the (i+1)th training unit processes these vectors to obtain the (i+1)th first global feature vector. The image information processing module of the (i+1)th training unit then processes these vectors to obtain the (i+1)th second global feature vector. The (i+1)th modality fusion module of the (i+1)th training unit then fuses the (i+1)th first global feature vector and the (i+1)th second global feature vector to obtain the (i+1)th multimodal feature sequence. This (i+1)th multimodal feature sequence is input into the (i+1)th classification module. The (i+1)th classification module predicts the (i+1)th classification result based on the multimodal features. If the (i+1)th classification result is inaccurate, the above operation continues, with the (i+2)th training unit continuing training, and so on, until an accurate classification result is obtained.
[0091] After obtaining an accurate classification result, the model parameters corresponding to the accurate classification result in the second model are frozen, that is, the global cue learning vector corresponding to the accurate classification result is frozen, thereby obtaining the progressive mild cognitive impairment classification model.
[0092] like Figure 2b As shown, this application also provides a medical image classification method based on cue learning, including:
[0093] Step 301: Obtain target image information and target attribute data.
[0094] Wherein, the target image information refers to the MRI image to be classified, and the target attribute data is the clinical data corresponding to the MRI image to be classified;
[0095] Step 302: Perform image registration on the target image information to obtain the image information to be processed;
[0096] Step 303: Normalize the target clinical data to obtain the clinical data to be processed;
[0097] Step 304: Input the image information to be processed and the clinical data to be processed into the progressive mild cognitive impairment classification model;
[0098] Step 305: The progressive mild cognitive impairment classification model acquires target attribute data of the target user and processes the target attribute data to obtain a corresponding first global feature sequence; and acquires target image information of the target user, processes the target image information according to the global cue learning vector to obtain a second global feature sequence; and performs multimodal feature fusion on the first global feature sequence and the second global feature sequence to obtain multimodal features; and predicts the disease type indicated by the target attribute data and the target image information based on the multimodal features.
[0099] In practice, the patient (i.e., the user) first goes to the hospital to have a brain MRI image taken and undergoes clinical testing to obtain the target attribute data required by this application embodiment. This application uses a trained progressive mild cognitive impairment classification model to determine the stage of Alzheimer's disease in the patient based on the data from the MRI image and the target attribute data, and then takes corresponding intervention and treatment measures.
[0100] For new hospitals lacking sufficient clinical data (including image and tabular data), weights obtained from Alzheimer's disease classification tasks can be acquired from other hospitals without infringing on patient privacy. Fine-tuning using the proposed method on limited data of progressive mild cognitive impairment yields a well-performing diagnostic model.
[0101] The above primarily describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, mobile electronic devices include corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0102] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.
[0103] Please see Figure 3 This application also provides a model training device 30, comprising:
[0104] The acquisition unit 31 is used to acquire a first training dataset, the first training dataset including a first image information subset and a first attribute data subset, the first image information subset including a first MRI image to be trained, and the first attribute data subset including first clinical data corresponding to the first MRI image.
[0105] The first training unit 32 is used to input the first training dataset into the first model for training to obtain an Alzheimer's classification model, wherein the first model refers to the Alzheimer's classification model to be trained.
[0106] Initialization unit 33 is used to initialize the second model according to the first model parameters of the Alzheimer's classification model, wherein the second model refers to the progressive mild cognitive impairment classification model to be trained.
[0107] Freeze unit 34 is used to freeze the first model parameters in the initialized second model;
[0108] Unit 35 is used to determine the global cue learning vector;
[0109] The second training unit 36 is used to input the second training set into the second model to fine-tune the global cue learning vector to obtain a progressive mild cognitive impairment classification model. The cue fine-tuning refers to adjusting only the parameters corresponding to the global cue learning vector in the second model during the training process.
[0110] As can be seen, in this embodiment, a first training dataset including a first subset of image information and a first subset of attribute data is obtained; the first training dataset is input into a first model for training to obtain an Alzheimer's classification model; a second model is initialized based on the first model parameters of the Alzheimer's classification model, where the second model refers to the progressive mild cognitive impairment classification model to be trained; the first model parameters in the initialized second model are frozen; a global cue learning vector is determined; and the second training set is input into the second model to fine-tune the global cue learning vector to obtain the progressive mild cognitive impairment classification model. In this way, the parameter transfer from the Alzheimer's classification module to the progressive mild cognitive impairment model can be completed simply by fine-tuning the global cue learning vector, improving the efficiency of model transfer.
[0111] In one possible embodiment, inputting a second training set into the second model to fine-tune the global cue learning vector to obtain a classification model for progressive mild cognitive impairment includes: acquiring a second training dataset, the second training dataset comprising a second image information subset and a second attribute data subset, the second image information subset comprising a second MRI image to be trained, and the second attribute data subset comprising second clinical data corresponding to the second MRI image; inputting the second training dataset into the second model; fine-tuning the global cue learning vector in the second model according to the second training dataset to obtain a classification result; determining whether the classification result of the second model is accurate; if the classification result is inaccurate... The classification loss function is calculated, and a new global cue learning vector is obtained by adjusting it using the stochastic gradient descent algorithm. The third training dataset is input into the second model to fine-tune the new global cue learning vector, resulting in a new classification result. The third training dataset includes a third image information subset and a third attribute data subset. The third image information subset includes the third MRI image to be trained, and the third attribute data subset includes the third clinical data corresponding to the third MRI image. If the new classification result is inaccurate, the above training operation is repeated until an accurate classification result is obtained. The model parameters corresponding to the accurate classification result on the second model are frozen to obtain the progressive mild cognitive impairment classification model.
[0112] In one possible embodiment, fine-tuning the global cue learning vector in the second model based on the second training dataset to obtain a classification result includes: inputting a subset of the second attribute data into a clinical data processing module in the second model; the clinical data processing module extracting a first global feature sequence from the subset of the second attribute data; inputting the global cue learning vector and the second image information subset into an image information processing module; the image information processing module extracting a second global feature sequence from the global cue learning vector and the second image information subset; inputting the first global feature sequence and the second global feature sequence into a modality fusion module; the modality fusion module performing modality fusion on the first global feature sequence and the second global feature sequence to obtain a multimodal feature sequence; inputting the multimodal feature sequence into a classification module; and the classification module predicting the classification result based on the multimodal features.
[0113] In one possible embodiment, the image information processing module includes a spatial dimension attention unit and a channel dimension attention unit; the image information processing module extracts a second global feature sequence from the second image information subset, including: obtaining the target image information by the second embedding layer and extracting a second feature sequence from the target image information; performing weighted averaging on the second feature sequence and the global cue learning vector by the spatial dimension attention unit to obtain a third feature sequence; performing weighted averaging on the second feature sequence by the channel dimension attention unit to obtain a fourth feature sequence; adding the third feature sequence and the fourth feature sequence to obtain a fifth feature sequence; performing a dot product operation on the fifth feature sequence and the global cue learning vector to obtain a sixth feature sequence; and adding the sixth feature sequence to the target image information to obtain the second global feature sequence.
[0114] In one possible embodiment, the modality fusion module performs modality fusion on the first global feature sequence and the second global feature sequence to obtain a multimodal feature sequence, including: setting a category training dataset, the category training dataset including multiple image category data; extracting a seventh feature sequence from the category training dataset; and performing modality fusion on the seventh feature sequence, the first global feature sequence, and the second global feature sequence to obtain the multimodal feature sequence.
[0115] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.
[0116] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.
[0117] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may include an electronic device.
[0118] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0119] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0121] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.
[0122] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), etc., various media capable of storing program code.
[0123] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.
Claims
1. A model training method, characterized in that, include: Obtain a first training dataset, the first training dataset including a first image information subset and a first attribute data subset, the first image information subset including a first MRI image to be trained, and the first attribute data subset including first clinical data corresponding to the first MRI image; The first training dataset is input into the first model for training to obtain the Alzheimer's classification model. The first model refers to the Alzheimer's classification model to be trained. The second model is initialized based on the first model parameters of the Alzheimer's classification model, where the second model refers to the progressive mild cognitive impairment classification model to be trained. Freeze the parameters of the first model in the second model after initialization; A global cue learning vector is determined. The global cue learning vector is a series of learnable parameters related to progressive mild cognitive impairment and is part of the second model. In subsequent training of the second model, only the global cue learning vector will be updated. The second training set is input into the second model to fine-tune the global cue learning vector. The global cue learning vector corresponding to the accurate classification result on the second model is frozen to obtain the progressive mild cognitive impairment classification model. The cue fine-tuning refers to adjusting only the parameters corresponding to the global cue learning vector in the second model during the training process. The progressive mild cognitive impairment classification model includes: a first clinical data processing module, used to acquire target attribute data of the target user and process the target attribute data to obtain a corresponding first global feature sequence; a first image information processing module, used to acquire target image information of the target user and process the target image information to obtain a second global feature sequence; a first modality fusion module, used to perform multimodal feature fusion on the first global feature sequence and the second global feature sequence to obtain multimodal features; and a first classification module, used to predict the disease type indicated by the target attribute data and the target image information based on the multimodal features.
2. The method according to claim 1, characterized in that, The second training set is input into the second model to fine-tune the global cue learning vector, resulting in a progressive mild cognitive impairment classification model, including: Obtain a second training dataset, which includes a second image information subset and a second attribute data subset. The second image information subset includes a second magnetic resonance image to be trained, and the second attribute data subset includes second clinical data corresponding to the second magnetic resonance image. Input the second training dataset into the second model; Based on the second training dataset, the global cue learning vector in the second model is fine-tuned to obtain the classification result; Determine whether the classification results of the second model are accurate; If the classification result is inaccurate, the classification loss function is calculated and adjusted using the stochastic gradient descent algorithm to obtain a new global cue learning vector; The third training dataset is input into the second model to fine-tune the new global cue learning vector to obtain a new classification result. The third training dataset includes a third image information subset and a third attribute data subset. The third image information subset includes the third MRI image to be trained, and the third attribute data subset includes the third clinical data corresponding to the third MRI image. If the new classification result is inaccurate, repeat the above training operation until an accurate classification result is obtained; Freeze the model parameters corresponding to the accurate classification results of the second model to obtain the progressive mild cognitive impairment classification model.
3. The method according to claim 2, characterized in that, Based on the second training dataset, the global cue learning vector in the second model is fine-tuned to obtain classification results, including: Input the second attribute data subset into the clinical data processing module of the second model; The clinical data processing module extracts a first global feature sequence from the second attribute data subset; The global cue learning vector and the second image information subset are input into the image information processing module; The image information processing module extracts a second global feature sequence from the global cue learning vector and the second image information subset; The first global feature sequence and the second global feature sequence are input into the modality fusion module; The modality fusion module performs modality fusion on the first global feature sequence and the second global feature sequence to obtain a multimodal feature sequence; The multimodal feature sequence is input into the classification module; The classification module predicts the classification result based on the multimodal features.
4. The method according to claim 3, characterized in that, The image information processing module includes a second embedding layer, a spatial dimension attention unit, and a channel dimension attention unit; The image information processing module extracts a second global feature sequence from the second image information subset, including: The target image information is obtained from the second embedding layer, and a second feature sequence is extracted from the target image information; The second feature sequence and the global cue learning vector are weighted and averaged by the spatial dimension attention unit to obtain the third feature sequence; The second feature sequence is weighted and averaged by the channel dimension attention unit to obtain the fourth feature sequence; The third feature sequence and the fourth feature sequence are added together to obtain the fifth feature sequence; The fifth feature sequence is multiplied by the global cue learning vector to obtain the sixth feature sequence; The sixth feature sequence is added to the target image information to obtain the second global feature sequence.
5. The method according to claim 3, characterized in that, The modality fusion module performs modality fusion on the first global feature sequence and the second global feature sequence to obtain a multimodal feature sequence, including: Set up a category training dataset, which includes image category data for multiple categories; Extract the seventh feature sequence from the training dataset of the aforementioned category; The seventh feature sequence, the first global feature sequence, and the second global feature sequence are modally fused to obtain the multimodal feature sequence.
6. A medical image classification method based on cue learning, characterized in that, include: Acquire target image information and target attribute data, wherein the target image information refers to the MRI image to be classified, and the target attribute data is the clinical data corresponding to the MRI image to be classified; Image registration is performed on the target image information to obtain the image information to be processed; The target clinical data were normalized to obtain the clinical data to be processed. The image information to be processed and the clinical data to be processed are input into the progressive mild cognitive impairment classification model as described in claim 1; The progressive mild cognitive impairment classification model acquires target attribute data of the target user and processes the target attribute data to obtain the corresponding first global feature sequence. In addition, the target image information of the target user is obtained, and the target image information is processed according to the learnable global cueing learning vector related to progressive mild cognitive impairment, which is part of the progressive mild cognitive impairment classification model, to obtain a second global feature sequence. Furthermore, multimodal feature fusion is performed on the first global feature sequence and the second global feature sequence to obtain multimodal features; and the disease type indicated by the target attribute data and the target image information is predicted based on the multimodal features.
7. A model training device, characterized in that, include: The acquisition unit is used to acquire a first training dataset, the first training dataset including a first image information subset and a first attribute data subset, the first image information subset including a first MRI image to be trained, and the first attribute data subset including first clinical data corresponding to the first MRI image; The first training unit is used to input the first training dataset into the first model for training to obtain an Alzheimer's classification model, wherein the first model refers to the Alzheimer's classification model to be trained. An initialization unit is used to initialize a second model based on the first model parameters of the Alzheimer's classification model, wherein the second model refers to the progressive mild cognitive impairment classification model to be trained. Freeze unit, used to freeze the parameters of the first model in the initialized second model; A determining unit is used to determine a global cue learning vector, which is a series of learnable parameters related to progressive mild cognitive impairment and is part of the second model. In subsequent training of the second model, only the global cue learning vector will be updated. The second training unit is used to input the second training set into the second model to fine-tune the global cue learning vector, freeze the global cue learning vector corresponding to the accurate classification result on the second model, so as to obtain a progressive mild cognitive impairment classification model. The cue fine-tuning refers to adjusting only the parameters corresponding to the global cue learning vector in the second model during the training process. The progressive mild cognitive impairment classification model includes: a first clinical data processing module, used to acquire target attribute data of the target user and process the target attribute data to obtain a corresponding first global feature sequence; a first image information processing module, used to acquire target image information of the target user and process the target image information to obtain a second global feature sequence; a first modality fusion module, used to perform multimodal feature fusion on the first global feature sequence and the second global feature sequence to obtain multimodal features; and a first classification module, used to predict the disease type indicated by the target attribute data and the target image information based on the multimodal features.
8. An electronic device, characterized in that, The method includes a processor, a memory, a communication interface, and one or more programs, said programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of the method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, A computer program for electronic data interchange is stored, wherein the computer program causes a computer to execute instructions for the steps of the method as described in any one of claims 1-6.