Multi-source domain adaptive sMRI image processing method and system based on attention mechanism
By adopting a multi-source domain adaptive method based on attention mechanism in sMRI image processing, the problems of data privacy and spatial information mining in the prior art are solved, and high-precision image classification and data privacy protection are achieved.
Patent Information
- Application Number
- CN202411876991.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-05-16
AI Technical Summary
In the prior art, the field adaptation method cannot guarantee the privacy of the training data, and the sMRI image processing technology fails to fully mine the spatial information of the image, resulting in inaccurate image classification results.
A multi-source domain adaptive method based on attention mechanism is adopted, and a federated learning training for multi-center data is carried out by constructing a federated self-attention multi-source domain adaptive model to ensure data privacy protection and fully mine the spatial information of the image.
It realizes secure model training on multi-center data, protects data privacy, and improves the classification accuracy of sMRI images.
Smart Images
Figure CN120014314A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a multi-source domain adaptive sMRI image processing method, system, terminal and computer-readable storage medium based on an attention mechanism. Background Art
[0002] Existing studies mostly reduce the dimensionality of input features by cutting or extracting features from raw magnetic resonance imaging (MRI) data, which inevitably leads to data distortion and loss of spatial information. Dimensionality reduction methods rely on prior knowledge and may introduce subjective biases, which can affect the objectivity of the results and reduce the generalization ability of the model. In addition, although sMRI image processing has achieved good results in deep learning, most existing studies rely on data from a single data center for training. Limited by the cost and resources of data collection, the amount of data from a single center is limited, the stability of the model is poor, and it is difficult to fit effectively. However, due to the problem of data heterogeneity, simple fusion training models often perform poorly on multi-center data, and when faced with new data, the generalization ability of the model is insufficient and cannot meet the needs of practical applications.
[0003] Domain adaptation technology can effectively eliminate the heterogeneity between the source domain and the target domain, thereby improving the performance of the model on different data sets. However, existing domain adaptation methods mainly target a single source domain, fail to consider the use of data from multiple source domains, and do not fully consider the problem of distribution differences within the domain. At the same time, in multi-center data joint training, it is usually necessary to collect raw data centrally, which may lead to the leakage of private data. At the same time, the existing sMRI image processing technology does not fully exploit the spatial information of the image, and the captured image features are not comprehensive enough, resulting in inaccurate classification results of sMRI images.
[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention
[0005] The main purpose of the present invention is to provide a multi-source domain adaptive sMRI image processing method, system, terminal and computer-readable storage medium based on an attention mechanism, aiming to solve the problem that the domain adaptation method in the prior art cannot guarantee the privacy of training data, and the existing sMRI image processing technology does not fully mine the spatial information of the image, and the captured image features are not comprehensive enough, resulting in the classification results of the sMRI image being not accurate enough.
[0006] To achieve the above object, the present invention provides a multi-source domain adaptive sMRI image processing method based on an attention mechanism, and the multi-source domain adaptive sMRI image processing method based on an attention mechanism comprises the following steps:
[0007] Constructing a federated self-attention multi-source domain adaptation model, wherein the federated self-attention multi-source domain adaptation model includes: an embedding layer, a multi-source domain adaptation Transformer module, a federated learning framework, and a multi-layer perception head;
[0008] Acquire sMRI data of multiple data centers, divide the multiple data centers into a target domain and multiple source domains, initialize the federated self-attention multi-source domain adaptation model on the server, and send the federated self-attention multi-source domain adaptation model as a global model to the multiple source domains through the server;
[0009] In each source domain, feature extraction is performed on the global model according to the respective sMRI data to obtain a plurality of different types of transfer information, and the plurality of transfer information and the global model are sent to a target domain for domain adaptation to obtain a plurality of local update models;
[0010] Sending the multiple local update models to the server for integration to obtain a new global model, and sending the new global model to multiple source domains through the server again to complete a new round of training, until the multiple local update models are successfully fitted, and sending the multiple successfully fitted local update models to the server for integration to obtain a target classification model;
[0011] Acquire a sMRI image to be predicted, preprocess the sMRI image to be predicted to obtain a target sMRI image, input the target sMRI image into a target classification model of the server for prediction, and output a classification result of the target sMRI image.
[0012] Optionally, the multi-source domain adaptive sMRI image processing method based on the attention mechanism, wherein the feature extraction of the global model is performed according to the respective sMRI data to obtain a plurality of different categories of transfer information, specifically includes:
[0013] The sMRI image data of each source domain is input into the embedding layer for convolution, flattening and linking operations, and the source domain data X of each source domain is output. s , source domain data X s is a vector sequence;
[0014] Each source domain data X s The multi-layer Transformer encoder layer is input into the multi-source domain adaptive Transformer module for encoding, and the source domain data features of different categories of each source domain are output. And extract the self-attention matrix generated by each layer of Transformer encoder layer, average multiple self-attention matrices to obtain the source domain self-attention matrix A of different categories in each source domainS .
[0015] Optionally, the multi-source domain adaptive sMRI image processing method based on the attention mechanism, wherein the respective sMRI image data are respectively input into the embedding layer for convolution, flattening and linking operations, and the source domain data X of each source domain is output. s , source domain data X s is a vector sequence, including:
[0016] Convolve the respective sMRI image data with a 3D convolution kernel of a specific size and a specific stride to obtain a 3D block corresponding to each source domain, and convert the 3D block corresponding to each source domain into a high-dimensional embedding vector sequence through a linear transformation process;
[0017] A trainable category tag is appended to the beginning of the high-dimensional embedding vector sequence, and the source domain data X of each source domain is output. s , the category labels are used to aggregate global information for classification.
[0018] Optionally, the multi-source domain adaptive sMRI image processing method based on the attention mechanism, wherein the multiple transfer information and the global model are sent to the target domain for domain adaptation to obtain multiple local update models, specifically includes:
[0019] Sending the plurality of transfer information and the global model corresponding to each source domain to the target domain;
[0020] The sMRI data of the target domain are respectively input into each of the global models, and convolution, flattening and linking operations are performed through the embedding layer to output the target domain data X of the target domain in each of the global models. T ;
[0021] Each target domain data X T The multi-layer Transformer encoder layer input to the multi-source domain adaptive Transformer module is encoded, and the target domain data features of the target domain in each of the global models are output. And extract the target domain self-attention matrix A in each of the global models T ;
[0022] According to each source domain self-attention matrix A S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions
[0023] According to each of the source domain data characteristics and each of the target domain data features Calculate multiple domain adaptation loss functions
[0024] According to the source domain data X of each source domain s The corresponding pseudo labels and the true label y s Calculate multiple cross entropy classification loss functions
[0025] According to a plurality of self-attention domain adaptive loss functions Multiple domain adaptation loss functions And multiple domain adaptation loss functions Calculate the objective function of each source domain separately
[0026]
[0027] Among them, α represents the first weight that controls the influence of the adaptive loss value, and β represents the second weight that controls the influence of the adaptive loss value;
[0028] According to the objective function of each source domain Back-propagation is performed on the global model corresponding to each source domain to optimize the global model, thereby obtaining multiple local updated models.
[0029] Optionally, the multi-source domain adaptive sMRI image processing method based on the attention mechanism, wherein the self-attention matrix A of each source domain is S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions Specifically include:
[0030] According to each source domain self-attention matrix A S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions
[0031]
[0032] Among them, H represents the number of multi-head self-attention modules, S represents the number of input sequences, and ‖‖ represents the L1 norm.
[0033] Optionally, the multi-source domain adaptive sMRI image processing method based on the attention mechanism, wherein the feature of each source domain data is and each of the target domain data features Calculate multiple domain adaptation loss functions Specifically include:
[0034] According to each of the source domain data characteristics and each of the target domain data features Calculate multiple domain adaptation loss functions
[0035]
[0036] Among them, C represents the category of the source domain data, c represents the ordinal number of the category, and N s Represents the number of source domain data, N T represents the number of target domain data, i represents the ordinal number of source domain data, j represents the ordinal number of target domain data, φ represents the mapping function, represents the weight of the i-th source domain data under category c, represents the weight of the j-th target domain data under category c, represents the source domain data features of the i-th source domain data, represents the target domain data features of the j-th target domain data, represents the reproducing kernel Hilbert space.
[0037] Optionally, the multi-source domain adaptive sMRI image processing method based on the attention mechanism, wherein the objective function of each source domain is Back-propagation is performed on the global model corresponding to each source domain to optimize the global model to obtain multiple local update models, specifically including:
[0038] According to the objective function of each source domain Perform back propagation on the global model corresponding to each source domain to calculate the parameter gradient of each global model;
[0039] Each of the global models is optimized and updated according to the multiple parameter gradients until a preset condition is met, thereby obtaining multiple local updated models.
[0040] In addition, to achieve the above-mentioned object, the present invention also provides a multi-source domain adaptive sMRI image processing system based on an attention mechanism, wherein the multi-source domain adaptive sMRI image processing system based on an attention mechanism comprises:
[0041] A model building module, used to build a federated self-attention multi-source domain adaptation model, wherein the federated self-attention multi-source domain adaptation model includes: an embedding layer, a multi-source domain adaptation Transformer module, a federated learning framework, and a multi-layer perception head;
[0042] an initialization module, used for acquiring sMRI data of multiple data centers, dividing the multiple data centers into a target domain and multiple source domains, initializing the federated self-attention multi-source domain adaptation model on the server, and sending the federated self-attention multi-source domain adaptation model as a global model to multiple source domains through the server;
[0043] An adaptive update module is used to extract features of the global model in each source domain according to the respective sMRI data to obtain a plurality of different types of transfer information, and send the plurality of transfer information and the global model to a target domain for domain adaptation to obtain a plurality of local update models;
[0044] An iterative integration module, used for sending the multiple local update models to the server for integration to obtain a new global model, and sending the new global model to multiple source domains through the server again to complete a new round of training until the multiple local update models are successfully fitted, and sending the multiple successfully fitted local update models to the server for integration to obtain a target classification model;
[0045] The image prediction module is used to obtain the sMRI image to be predicted, preprocess the sMRI image to be predicted to obtain a target sMRI image, input the target sMRI image into the target classification model of the server for prediction, and output the classification result of the target sMRI image.
[0046] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multi-source domain adaptive sMRI image processing program based on an attention mechanism stored in the memory and run on the processor, and when the multi-source domain adaptive sMRI image processing program based on the attention mechanism is executed by the processor, the steps of the multi-source domain adaptive sMRI image processing method based on the attention mechanism as described above are implemented.
[0047] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multi-source domain adaptive sMRI image processing program based on an attention mechanism, and when the multi-source domain adaptive sMRI image processing program based on an attention mechanism is executed by a processor, the steps of the multi-source domain adaptive sMRI image processing method based on an attention mechanism as described above are implemented.
[0048] In the present invention, the data center is divided into a target domain and a source domain, a federated adaptive model is initialized at the server, and the federated adaptive model is sent to multiple source domains as a global model through the server, and in each source domain, the global model is extracted according to the respective sMRI data to obtain the transmission information, and the transmission information and the global model are sent to the target domain for domain adaptation to obtain a local update model; the local update model is sent to the server for integration to obtain a new global model, and the new global model is sent to the source domain again through the server for a new round of training until the local update model is successfully fitted, and the successfully fitted local update model is sent to the server for integration to obtain a classification model, and the classification of sMRI images is completed according to the classification model. The present invention performs multi-center model training through federated learning, and strengthens privacy protection during data image processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a flowchart of a preferred embodiment of the multi-source domain adaptive sMRI image processing method based on the attention mechanism of the present invention;
[0050] Figure 2 It is an architecture diagram of a federated self-attention multi-source domain adaptation model in the sMRI image processing method based on multi-source domain adaptation of the attention mechanism of the present invention;
[0051] Figure 3 It is an architecture diagram of the embedding layer in the multi-source domain adaptive sMRI image processing method based on the attention mechanism of the present invention;
[0052] Figure 4 It is an architecture diagram of a multi-source domain adaptive Transformer module in the multi-source domain adaptive sMRI image processing method based on the attention mechanism of the present invention;
[0053] Figure 5 It is an architecture diagram of a multi-layer perception head in the multi-source domain adaptive sMRI image processing method based on the attention mechanism of the present invention;
[0054] Figure 6 It is a structural diagram of a preferred embodiment of the sMRI image processing system for multi-source domain adaptation based on the attention mechanism of the present invention;
[0055] Figure 7 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION
[0056] The present application provides a multi-source domain adaptive sMRI image processing method, system and terminal based on an attention mechanism. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0058] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0059] The multi-source domain adaptive sMRI image processing method based on the attention mechanism described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the multi-source domain adaptive sMRI image processing method based on the attention mechanism includes the following steps:
[0060] Step S10: construct a federated self-attention multi-source domain adaptation model, wherein the federated self-attention multi-source domain adaptation model includes: an embedding layer, a multi-source domain adaptation Transformer module, a federated learning framework, and a multi-layer perception head.
[0061] like Figure 2As shown, specifically, a federated self-attention multi-source domain adaptation model is constructed to realize the processing and classification of sMRI images, and the federated self-attention multi-source domain adaptation model includes the following parts: Embedding layer: used to cut and convert 3D images into a series of vector sequences with position information. Multi-source domain adaptation Transformer: This module is designed to achieve self-attention domain adaptation, combining an encoder module for feature extraction and self-attention map generation, and a domain adaptation module to complete feature alignment by evaluating the self-attention domain adaptation loss and the local maximum mean difference loss. Federated learning framework: used for cross-domain training between the source domain and the target domain. Multi-layer perceptron head: used for final classification.
[0062] Step S20, obtaining sMRI data from multiple data centers, dividing the multiple data centers into a target domain and multiple source domains, initializing the federated self-attention multi-source domain adaptation model on the server, and sending the federated self-attention multi-source domain adaptation model as a global model to multiple source domains through the server.
[0063] There are advantages to using multi-center data training models to classify sMRI data. However, most existing models use reduced-dimensional image data for model training, which leads to the loss of image spatial information. At the same time, data from different centers are heterogeneous, and joint training models often perform poorly when faced with data from new centers. Traditional domain adaptation methods require centralized data for training, which can lead to the leakage of private data. Therefore, in this embodiment, secure multi-center model training is performed through a federated learning framework to ensure the protection of data privacy and prevent data leakage.
[0064] It is understandable that the source domain refers to the source of data and models. The data in the source domain is usually used to train the model so that it has a certain generalization ability. The target domain is where the model is to be applied. In practical applications, it is often necessary to migrate the trained model to the target domain to achieve tasks such as prediction or classification. The source domain and target domain are commonly used concepts in domain migration, referring to the source of data and models and the place where the model is to be applied, respectively.
[0065] Specifically, first, sMRI data from multiple data centers are acquired, and the sMRI data from each data center are preprocessed to obtain a processed gray matter image; then, the multiple data centers are divided into a target domain (target domain client) and multiple source domains (source domain clients), that is, each center is regarded as a client for a multi-center unsupervised federated learning training model; finally, the federated self-attention multi-source domain adaptation model is initialized on the server side, and the federated self-attention multi-source domain adaptation model is sent to multiple source domains as a global model through the server side to complete the classification task and alignment task.
[0066] Step S30: In each source domain, feature extraction is performed on the global model according to the respective sMRI data to obtain a plurality of different categories of transfer information, and the plurality of transfer information and the global model are sent to the target domain for domain adaptation to obtain a plurality of local update models.
[0067] The feature extraction of the global model according to the respective sMRI data to obtain a plurality of different types of transfer information specifically includes:
[0068] The sMRI image data of each source domain is input into the embedding layer for convolution, flattening and linking operations, and the source domain data X of each source domain is output. s , source domain data X s is a vector sequence;
[0069] Each source domain data X s The multi-layer Transformer encoder layer is input into the multi-source domain adaptive Transformer module for encoding, and the source domain data features of different categories of each source domain are output. And extract the self-attention matrix generated by each layer of Transformer encoder layer, average multiple self-attention matrices to obtain the source domain self-attention matrix A of different categories in each source domain S .
[0070] In this embodiment, if Figure 3 As shown in the figure, in the embedding layer, the image is convolved with a 3D convolution kernel of a specific size and stride, effectively dividing it into multiple 3D blocks. These 3D blocks are then converted into high-dimensional embedding vectors through a linear transformation process. A trainable category tag is appended to the beginning of the sequence to aggregate global information for classification. In addition, position information is injected into each embedding vector to ensure that the model can accurately identify the original position of each block in the image. The final output is a vector sequence that cleverly combines the features of the image with its position information, ready for subsequent processing.
[0071] like Figure 4As shown in the figure, after the image is converted into a sequence, it is input into the Transformer encoder layer. The encoder layer consists of a multi-head self-attention module and a multi-layer perceptron. The self-attention mechanism and the feedforward network are used to learn the contextual relationship of each element in the sequence and extract high-level feature representations. In addition, residual connections are used to improve the training ability of the multi-layer encoder stack, and layer normalization is used to stabilize the training process. In order to improve the generalization ability of the model and prevent overfitting, a dropout layer (Dropout) is also added. The Transformer encoder layer will output a series of high-dimensional vectors. Each vector deeply encodes the information of the corresponding input element and its relationship in the context of the entire sequence, providing a strong feature basis for downstream tasks. At the same time, a self-attention matrix is also output for subsequent domain adaptation functions.
[0072] Furthermore, the sending of the plurality of transfer information and the global model to the target domain for domain adaptation to obtain a plurality of local update models specifically includes:
[0073] S31, sending the plurality of transfer information and the global model corresponding to each source domain to the target domain.
[0074] It can be understood that the target domain uses the model transmitted from the corresponding source domain. When the communication frequency is 1, the model is a global model, that is, multiple transmission information and the global models and data corresponding to each source domain are sent to the target domain, and then the target domain can extract features through its own sMRI data.
[0075] S32, inputting the sMRI data of the target domain into each of the global models respectively, performing convolution, flattening and linking operations through the embedding layer, and outputting the target domain data X of the target domain in each of the global models T .
[0076] S33, each target domain data X T The multi-layer Transformer encoder layer input to the multi-source domain adaptive Transformer module is encoded, and the target domain data features of the target domain in each of the global models are output. And extract the target domain self-attention matrix A in each of the global models T .
[0077] In this embodiment, after the target domain obtains the global model, the sMRI image data of the target domain is output to the embedding layer of the global model for convolution, flattening and linking operations in the same way as the sMRI data of the source domain, thereby obtaining the target domain feature target domain data X T , and the target domain data X TThe multi-layer Transformer encoder layer input to the multi-source domain adaptive Transformer module is encoded, and the target domain data features of the target domain in each of the global models are output. And extract the target domain self-attention matrix A in each of the global models T .
[0078] S34, according to each of the source domain self-attention matrix A S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions
[0079] In order to encourage the consistent self-attention matrix obtained by the source domain and the target domain data, the present invention designs a self-attention domain adaptive loss function. Specifically, according to each of the source domain self-attention matrices A S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions
[0080]
[0081] Among them, H represents the number of multi-head self-attention modules, S represents the number of input sequences, and ‖‖ represents the L1 norm.
[0082] S35, based on each of the source domain data features and each of the target domain data features Calculate multiple domain adaptation loss functions
[0083] It is understandable that the maximum mean discrepancy (MMD) is a method to measure the difference between two distributions, which was first proposed in the two-sample detection problem. The basic idea of MMD is to map the random variables of the two distributions to a higher order, and then calculate the expected value after mapping. The maximum value of the difference between the expected values is the MMD value.
[0084] MMD is added to the objective function as a domain adaptation loss function to bring the two data distributions closer. However, there is a problem with MMD, which is that it only considers the distribution of the overall data, but not the differences between data of different categories. In the same domain, the distribution of data of different categories may also be different, which this study calls subdomains. If it is possible to bring only subdomains of the same category closer without affecting subdomains of different categories, then the subtle information of category differences can be retained to the greatest extent. Under this assumption, the local maximum mean discrepancy (LMMD) loss function It is necessary to calculate the MMD value of each subdomain of the same category.
[0085] Specifically, according to each of the source domain data characteristics and each of the target domain data features Calculate multiple domain adaptation loss functions
[0086]
[0087] Among them, C represents the category of the source domain data, c represents the ordinal number of the category, and N s Represents the number of source domain data, N T represents the number of target domain data, i represents the ordinal number of source domain data, j represents the ordinal number of target domain data, φ represents the mapping function, represents the weight of the i-th source domain data under category c, represents the weight of the j-th target domain data under category c, represents the source domain data features of the i-th source domain data, represents the target domain data features of the j-th target domain data, represents the reproducing kernel Hilbert space, and are the i-th source domain data and the j-th source domain data respectively, and are the i-th target domain data and the j-th target domain data respectively, and k represents the kernel function used to calculate the inner product of two high-dimensional vectors.
[0088] Furthermore, both the source domain data and the target domain data are encoded by the same neural network, Z ′ It is the feature after Transformer encoding, and the weight here is The calculation formula is:
[0089]
[0090] Among them, y ic represents the label of category c, y i represents the label of the sample data, and y represents the dataset.
[0091] It can be understood that in this embodiment, only the data in the source domain can provide labels, while the data in the target domain has no labels. In this case, it is necessary to use the pseudo labels generated by the classification layer. Instead, as the model learns, the accuracy of the model will continue to improve.
[0092] S36, based on the source domain data X of each source domain s The corresponding pseudo labels and the true label ys Calculate multiple cross entropy classification loss functions
[0093] S37, according to the self-attention domain adaptive loss function Multiple domain adaptation loss functions And multiple domain adaptation loss functions Calculate the objective function of each source domain separately
[0094]
[0095] Among them, α represents the first weight that controls the influence of the adaptive loss value, and β represents the second weight that controls the influence of the adaptive loss value.
[0096] S38, according to the objective function of each source domain Back-propagation is performed on the global model corresponding to each source domain to optimize the global model, thereby obtaining multiple local updated models.
[0097] Specifically, according to the objective function of each source domain Perform back propagation on the global model corresponding to each source domain to calculate the parameter gradient of each global model;
[0098] Furthermore, each of the global models is optimized and updated respectively according to the multiple parameter gradients until a preset condition is met, thereby obtaining multiple local updated models.
[0099] Step S40: Send the multiple local update models to the server for integration to obtain a new global model, and send the new global model to multiple source domains again through the server to complete a new round of training until the multiple local update models are successfully fitted, and send the multiple successfully fitted local update models to the server for integration to obtain a target classification model.
[0100] It can be understood that the framework of privately training models in source domains and target domains proposed in the present invention has the following specific processes: each source domain client receives a global model and uses source domain data for training; each source domain client communicates with a target domain client to transmit information such as model parameters, gradients, feature outputs, etc., and the target domain client uses this information for training; each target domain client performs weighted aggregation according to the proportion of different categories in the corresponding source domain, and uses the FedAvg method to update the global model to obtain multiple local updated models, and sends the multiple local updated models to the server for integration to obtain a new global model, and the new global model is sent to multiple source domains again through the server; the above steps are repeated until the model is successfully fitted.
[0101] Step S50, obtaining a to-be-predicted sMRI image, preprocessing the to-be-predicted sMRI image to obtain a target sMRI image, inputting the target sMRI image into the target classification model of the server for prediction, and outputting a classification result of the target sMRI image.
[0102] Specifically, an sMRI image to be predicted is obtained, and the sMRI image to be predicted is preprocessed. The specific processing flow is as follows: first, the SPM12 software package is used to perform anterior commissure-posterior commissure (AC-PC) calibration of the brain, and then the CAT12 toolkit is used to complete the following operations: noise filtering, internal resampling, bias field correction, skull removal, linear alignment and tissue segmentation, and only the gray matter part is retained, and spatial normalization and cropping boundary processing are performed to obtain the target sMRI image.
[0103] Furthermore, the target sMRI image is input into the target classification model of the server for prediction, and the classification result of the target sMRI image is output through the multi-layer perceptron in the target classification model. Figure 5 As shown in the figure, after the global model extracts the feature distribution of the test data, it extracts the previously added category tags and inputs them into the MLP head layer for classification. The head layer consists of a normalization layer and a fully connected layer, and finally the Argmax function is used for classification.
[0104] It can be seen that the present invention has the following advantages:
[0105] (1) The present invention utilizes the 3D ViT model to fully mine the spatial information of the image and improves the accuracy of sMRI image classification by capturing comprehensive image features.
[0106] (2) The present invention realizes unsupervised multi-source domain adaptation on multi-center data by aligning self-attention maps and integrating local maximum mean differences.
[0107] (3) The present invention performs secure multi-center model training through a federated learning framework to ensure the protection of data privacy and prevent data leakage.
[0108] Furthermore, if Figure 6 As shown, based on the above-mentioned multi-source domain adaptive sMRI image processing method based on the attention mechanism, the present invention also provides a multi-source domain adaptive sMRI image processing system based on the attention mechanism, wherein the multi-source domain adaptive sMRI image processing system based on the attention mechanism includes:
[0109] A model building module 51 is used to build a federated self-attention multi-source domain adaptation model, wherein the federated self-attention multi-source domain adaptation model includes: an embedding layer, a multi-source domain adaptation Transformer module, a federated learning framework, and a multi-layer perception head;
[0110] An initialization module 52 is used to obtain sMRI data of multiple data centers, divide the multiple data centers into a target domain and multiple source domains, initialize the federated self-attention multi-source domain adaptation model on the server, and send the federated self-attention multi-source domain adaptation model as a global model to multiple source domains through the server;
[0111] An adaptive updating module 53 is used to extract features of the global model in each source domain according to the respective sMRI data to obtain a plurality of different types of transfer information, and send the plurality of transfer information and the global model to a target domain for domain adaptation to obtain a plurality of local update models;
[0112] An iterative integration module 54 is used to send the multiple local update models to the server for integration to obtain a new global model, and send the new global model to multiple source domains through the server again to complete a new round of training until the multiple local update models are successfully fitted, and send the multiple successfully fitted local update models to the server for integration to obtain a target classification model;
[0113] The image prediction module 55 is used to obtain the sMRI image to be predicted, preprocess the sMRI image to be predicted to obtain a target sMRI image, input the target sMRI image into the target classification model of the server for prediction, and output the classification result of the target sMRI image.
[0114] Furthermore, if Figure 7 As shown, based on the above-mentioned multi-source domain adaptive sMRI image processing method and system based on the attention mechanism, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 7 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0115] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, a multi-source domain adaptive sMRI image processing program 40 based on an attention mechanism is stored on the memory 20, and the multi-source domain adaptive sMRI image processing program 40 based on an attention mechanism can be executed by the processor 10, thereby realizing the multi-source domain adaptive sMRI image processing method based on the attention mechanism in the present application.
[0116] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the multi-source domain adaptive sMRI image processing method based on the attention mechanism.
[0117] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.
[0118] In one embodiment, when the processor 10 executes the sMRI image processing program 40 based on the multi-source domain adaptation of the attention mechanism in the memory 20, the following steps are implemented:
[0119] Constructing a federated self-attention multi-source domain adaptation model, wherein the federated self-attention multi-source domain adaptation model includes: an embedding layer, a multi-source domain adaptation Transformer module, a federated learning framework, and a multi-layer perception head;
[0120] Acquire sMRI data of multiple data centers, divide the multiple data centers into a target domain and multiple source domains, initialize the federated self-attention multi-source domain adaptation model on the server, and send the federated self-attention multi-source domain adaptation model as a global model to the multiple source domains through the server;
[0121] In each source domain, feature extraction is performed on the global model according to the respective sMRI data to obtain a plurality of different types of transfer information, and the plurality of transfer information and the global model are sent to a target domain for domain adaptation to obtain a plurality of local update models;
[0122] Sending the multiple local update models to the server for integration to obtain a new global model, and sending the new global model to multiple source domains through the server again to complete a new round of training, until the multiple local update models are successfully fitted, and sending the multiple successfully fitted local update models to the server for integration to obtain a target classification model;
[0123] Acquire a sMRI image to be predicted, preprocess the sMRI image to be predicted to obtain a target sMRI image, input the target sMRI image into a target classification model of the server for prediction, and output a classification result of the target sMRI image.
[0124] The feature extraction of the global model according to the respective sMRI data to obtain multiple different types of transfer information specifically includes:
[0125] The sMRI image data of each source domain is input into the embedding layer for convolution, flattening and linking operations, and the source domain data X of each source domain is output. s , source domain data X s is a vector sequence;
[0126] Each source domain data X s The multi-layer Transformer encoder layer is input into the multi-source domain adaptive Transformer module for encoding, and the source domain data features of different categories of each source domain are output. And extract the self-attention matrix generated by each layer of Transformer encoder layer, average multiple self-attention matrices to obtain the source domain self-attention matrix A of different categories in each source domain S .
[0127] The sMRI image data of each source domain are respectively input into the embedding layer for convolution, flattening and linking operations, and the source domain data X of each source domain is output. s , source domain data X s is a vector sequence, including:
[0128] Convolve the respective sMRI image data with a 3D convolution kernel of a specific size and a specific stride to obtain a 3D block corresponding to each source domain, and convert the 3D block corresponding to each source domain into a high-dimensional embedding vector sequence through a linear transformation process;
[0129] A trainable category tag is appended to the beginning of the high-dimensional embedding vector sequence, and the source domain data X of each source domain is output. s , the category labels are used to aggregate global information for classification.
[0130] The sending of the plurality of transfer information and the global model to the target domain for domain adaptation to obtain a plurality of local update models specifically includes:
[0131] Sending the plurality of transfer information and the global model corresponding to each source domain to the target domain;
[0132] The sMRI data of the target domain are respectively input into each of the global models, and convolution, flattening and linking operations are performed through the embedding layer to output the target domain data X of the target domain in each of the global models. T ;
[0133] Each target domain data X T The multi-layer Transformer encoder layer input to the multi-source domain adaptive Transformer module is encoded, and the target domain data features of the target domain in each of the global models are output. And extract the target domain self-attention matrix A in each of the global models T ;
[0134] According to each source domain self-attention matrix A S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions
[0135] According to each of the source domain data characteristics and each of the target domain data features Calculate multiple domain adaptation loss functions
[0136] According to the source domain data X of each source domain s The corresponding pseudo labels and the true label y s Calculate multiple cross entropy classification loss functions
[0137] According to a plurality of self-attention domain adaptive loss functions Multiple domain adaptation loss functions And multiple domain adaptation loss functions Calculate the objective function of each source domain separately
[0138]
[0139] Among them, α represents the first weight that controls the influence of the adaptive loss value, and β represents the second weight that controls the influence of the adaptive loss value;
[0140] According to the objective function of each source domain Back-propagation is performed on the global model corresponding to each source domain to optimize the global model, thereby obtaining multiple local updated models.
[0141] Among them, the self-attention matrix A of each source domain S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions Specifically include:
[0142] According to each source domain self-attention matrix A S And each target domain self-attention matrix A s , calculate multiple self-attention domain adaptive loss functions
[0143]
[0144] Among them, H represents the number of multi-head self-attention modules, S represents the number of input sequences, and ‖‖ represents the L1 norm.
[0145] Wherein, according to each of the source domain data features and each of the target domain data features Calculate multiple domain adaptation loss functions Specifically include:
[0146] According to each of the source domain data characteristics and each of the target domain data features Calculate multiple domain adaptation loss functions
[0147]
[0148] Among them, C represents the category of the source domain data, c represents the ordinal number of the category, and N s Represents the number of source domain data, N T represents the number of target domain data, i represents the ordinal number of source domain data, j represents the ordinal number of target domain data, φ represents the mapping function, represents the weight of the i-th source domain data under category c, represents the weight of the j-th target domain data under category c, represents the source domain data features of the i-th source domain data, represents the target domain data features of the j-th target domain data, represents the reproducing kernel Hilbert space.
[0149] Wherein, the objective function according to each source domain Back-propagation is performed on the global model corresponding to each source domain to optimize the global model to obtain multiple local update models, specifically including:
[0150] According to the objective function of each source domain Perform back propagation on the global model corresponding to each source domain to calculate the parameter gradient of each global model;
[0151] Each of the global models is optimized and updated according to the multiple parameter gradients until a preset condition is met, thereby obtaining multiple local updated models.
[0152] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multi-source domain adaptive sMRI image processing program based on an attention mechanism, and when the multi-source domain adaptive sMRI image processing program based on an attention mechanism is executed by a processor, the steps of the multi-source domain adaptive sMRI image processing method based on an attention mechanism as described above are implemented.
[0153] In summary, the present invention proposes a multi-source domain adaptive sMRI image processing method and system based on an attention mechanism, the method comprising: dividing a data center into a target domain and a source domain, initializing a federated adaptive model on a server, and sending the federated adaptive model as a global model to multiple source domains through the server, extracting features from the global model according to respective sMRI data in each source domain, obtaining transfer information, sending the transfer information and the global model to the target domain for domain adaptation, and obtaining a local update model; sending the local update model to the server for integration, obtaining a new global model, and sending the new global model to the source domain again through the server for a new round of training, until the local update model is successfully fitted, sending the successfully fitted local update model to the server for integration, obtaining a classification model, and completing the classification of sMRI images according to the classification model. The present invention performs multi-center model training through federated learning, thereby strengthening privacy protection during data image processing.
[0154] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.
[0155] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0156] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A multi-source domain adaptive sMRI image processing method based on attention mechanism, characterized in that: The multi-source domain adaptive sMRI image processing method based on the attention mechanism includes: Constructing a federated self-attention multi-source domain adaptation model, wherein the federated self-attention multi-source domain adaptation model includes: an embedding layer, a multi-source domain adaptation Transformer module, a federated learning framework, and a multi-layer perception head; Acquire sMRI data of multiple data centers, divide the multiple data centers into a target domain and multiple source domains, initialize the federated self-attention multi-source domain adaptation model on the server, and send the federated self-attention multi-source domain adaptation model as a global model to the multiple source domains through the server; In each source domain, feature extraction is performed on the global model according to the respective sMRI data to obtain a plurality of different types of transfer information, and the plurality of transfer information and the global model are sent to a target domain for domain adaptation to obtain a plurality of local update models; Sending the multiple local update models to the server for integration to obtain a new global model, and sending the new global model to multiple source domains through the server again to complete a new round of training, until the multiple local update models are successfully fitted, and sending the multiple successfully fitted local update models to the server for integration to obtain a target classification model; Acquire a sMRI image to be predicted, preprocess the sMRI image to be predicted to obtain a target sMRI image, input the target sMRI image into a target classification model of the server for prediction, and output a classification result of the target sMRI image.
2. The multi-source domain adaptive sMRI image processing method based on the attention mechanism according to claim 1 is characterized in that: The feature extraction of the global model according to the respective sMRI data to obtain a plurality of different types of transfer information specifically includes: The sMRI image data of each source domain is input into the embedding layer for convolution, flattening and linking operations, and the source domain data X of each source domain is output. s , source domain data X s is a vector sequence; Each source domain data X s The multi-layer Transformer encoder layer is input into the multi-source domain adaptive Transformer module for encoding, and the source domain data features of different categories of each source domain are output. And extract the self-attention matrix generated by each Transformer encoder layer, average multiple self-attention matrices to obtain the source domain self-attention matrix A of different categories in each source domain S .
3. The multi-source domain adaptive sMRI image processing method based on the attention mechanism according to claim 2 is characterized in that: The respective sMRI image data are respectively input into the embedding layer for convolution, flattening and linking operations, and the source domain data X of each source domain is output. s , source domain data X s is a vector sequence, including: Convolve the respective sMRI image data with a 3D convolution kernel of a specific size and a specific stride to obtain a 3D block corresponding to each source domain, and convert the 3D block corresponding to each source domain into a high-dimensional embedding vector sequence through a linear transformation process; A trainable category tag is appended to the beginning of the high-dimensional embedding vector sequence, and the source domain data X of each source domain is output. s , the category labels are used to aggregate global information for classification.
4. The multi-source domain adaptive sMRI image processing method based on the attention mechanism according to claim 3 is characterized in that: The sending the plurality of transfer information and the global model to the target domain for domain adaptation to obtain a plurality of local update models specifically includes: Sending the plurality of transfer information and the global model corresponding to each source domain to the target domain; The sMRI data of the target domain are respectively input into each of the global models, and convolution, flattening and linking operations are performed through the embedding layer to output the target domain data X of the target domain in each of the global models. T ; Each target domain data X T The multi-layer Transformer encoder layer input to the multi-source domain adaptive Transformer module is encoded, and the target domain data features of the target domain in each of the global models are output. And extract the target domain self-attention matrix A in each of the global models T ; According to each source domain self-attention matrix A S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions According to each of the source domain data characteristics and each of the target domain data features Calculate multiple domain adaptation loss functions According to the source domain data X of each source domain s The corresponding pseudo labels and the true label y s Calculate multiple cross entropy classification loss functions According to a plurality of self-attention domain adaptive loss functions Multiple domain adaptive loss functions And multiple domain adaptive loss functions Calculate the objective function of each source domain separately Among them, α represents the first weight that controls the influence of the adaptive loss value, and β represents the second weight that controls the influence of the adaptive loss value; According to the objective function of each source domain Back-propagation is performed on the global model corresponding to each source domain to optimize the global model, thereby obtaining multiple local updated models.
5. The multi-source domain adaptive sMRI image processing method based on the attention mechanism according to claim 4 is characterized in that: According to each of the source domain self-attention matrix A S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions Specifically include: According to each source domain self-attention matrix A S And each target domain self-attention matrix A T , calculate multiple self-attention domain adaptive loss functions Among them, H represents the number of multi-head self-attention modules, S represents the number of input sequences, and ‖‖ represents the L1 norm.
6. The multi-source domain adaptive sMRI image processing method based on the attention mechanism according to claim 5, characterized in that: According to each of the source domain data features and each of the target domain data features Calculate multiple domain adaptation loss functions Specifically include: According to each of the source domain data characteristics and each of the target domain data features Calculate multiple domain adaptation loss functions Among them, C represents the category of the source domain data, c represents the ordinal number of the category, and N s Represents the number of source domain data, N T represents the number of target domain data, i represents the ordinal number of source domain data, j represents the ordinal number of target domain data, φ represents the mapping function, represents the weight of the i-th source domain data under category c, represents the weight of the j-th target domain data under category c, represents the source domain data features of the i-th source domain data, represents the target domain data features of the j-th target domain data, represents the reproducing kernel Hilbert space.
7. The multi-source domain adaptive sMRI image processing method based on the attention mechanism according to claim 4, characterized in that: The objective function according to each of the source domains Back-propagation is performed on the global model corresponding to each source domain to optimize the global model to obtain multiple local update models, specifically including: According to the objective function of each source domain Perform back propagation on the global model corresponding to each source domain to calculate the parameter gradient of each global model; Each of the global models is optimized and updated according to the multiple parameter gradients until a preset condition is met, thereby obtaining multiple local updated models.
8. A multi-source domain adaptive sMRI image processing system based on attention mechanism, characterized in that: The multi-source domain adaptive sMRI image processing system based on the attention mechanism includes: A model building module, used to build a federated self-attention multi-source domain adaptation model, wherein the federated self-attention multi-source domain adaptation model includes: an embedding layer, a multi-source domain adaptation Transformer module, a federated learning framework, and a multi-layer perception head; an initialization module, used for acquiring sMRI data of multiple data centers, dividing the multiple data centers into a target domain and multiple source domains, initializing the federated self-attention multi-source domain adaptation model on the server, and sending the federated self-attention multi-source domain adaptation model as a global model to multiple source domains through the server; An adaptive update module is used to extract features of the global model in each source domain according to the respective sMRI data to obtain a plurality of different types of transfer information, and send the plurality of transfer information and the global model to a target domain for domain adaptation to obtain a plurality of local update models; An iterative integration module, used for sending the multiple local update models to the server for integration to obtain a new global model, and sending the new global model to multiple source domains through the server again to complete a new round of training until the multiple local update models are successfully fitted, and sending the multiple successfully fitted local update models to the server for integration to obtain a target classification model; The image prediction module is used to obtain the sMRI image to be predicted, preprocess the sMRI image to be predicted to obtain a target sMRI image, input the target sMRI image into the target classification model of the server for prediction, and output the classification result of the target sMRI image.
9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a multi-source domain adaptive sMRI image processing program based on an attention mechanism, which is stored in the memory and can be run on the processor. When the multi-source domain adaptive sMRI image processing program based on the attention mechanism is executed by the processor, the steps of the multi-source domain adaptive sMRI image processing method based on the attention mechanism as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multi-source domain adaptive sMRI image processing program based on an attention mechanism. When the multi-source domain adaptive sMRI image processing program based on an attention mechanism is executed by a processor, the steps of the multi-source domain adaptive sMRI image processing method based on an attention mechanism as described in any one of claims 1 to 7 are implemented.