Longitudinal neural image data modeling and feature clustering method based on transformer model

Abstract

The application discloses a longitudinal neural image data modeling and feature clustering method based on a Transformer model, which comprises the following steps: acquiring longitudinal neural image data of a subject at three time points to generate a second-order difference brain network; dividing the second-order difference brain network into multiple interaction sub-matrices; encoding the interaction sub-matrices to obtain an interaction embedding vector of the subject; matching the interaction embedding vector of the subject with a memory vector to obtain a new representation vector of the subject and a corresponding score list; performing graph reconstruction on the new representation vector; and determining a clustering center at the same time. The application combines deep learning with longitudinal neural image analysis, and constructs an efficient and automatic time sequence feature modeling and clustering analysis method, which can be used as an intelligent auxiliary analysis tool in mental disease research, and has important scientific research application value and broad popularization prospect in supporting individual difference research and connection mode exploration.

Description

TECHNICAL FIELD

[0001] The present application belongs to the cross field of psychiatry, medical image analysis and machine learning, and particularly relates to a longitudinal neuroimaging data modeling and feature clustering method based on a Transformer model, which is suitable for structured representation and pattern mining of multi-time point brain imaging data. BACKGROUND

[0002] Psychiatric diseases have high clinical manifestation and biological heterogeneity, which makes their diagnosis and treatment face many challenges. Existing researches mainly focus on the grouping of symptom characteristics, but this grouping method based on subjective information lacks dynamic description of the underlying neurobiological processes and fails to deeply explore the internal mechanism and development trajectory of the disease. Therefore, although the division based on symptoms is helpful for the analysis and treatment of diseases, it often becomes a major bottleneck in basic research and mechanism exploration of psychiatric diseases, and also limits the implementation of precision medicine goals.

[0003] On the other hand, previous cross-sectional studies have shown that different psychiatric patients may have unique underlying biological mechanisms, which are not static but dynamically change during the treatment of the disease. More importantly, the development trajectory of psychiatric diseases may also differ among different individuals. For example, the changes in brain structure and function that occur during the disease process are closely related to the changes in clinical symptoms, which indicates that a deep understanding of these changes is crucial for improving treatment effectiveness and predicting the course of disease treatment. Therefore, from the perspective of neuroimaging, extracting feature patterns reflecting the evolution of the disease can not only help to understand individual differences, but also provide very important information for precision diagnosis and treatment.

[0004] In recent years, with the development of neuroimaging technology, especially the wide application of longitudinal neuroimaging data, researchers have been able to track the evolution of individual brain function networks over time by analyzing imaging data at different time points. Compared with traditional cross-sectional studies, longitudinal imaging research provides a more dynamic perspective and can reveal the evolution trajectory of brain structure and function in patients with different psychiatric diseases during the course of the disease. However, although longitudinal neuroimaging data provide more abundant information, traditional statistical analysis or shallow machine learning methods have certain limitations in dealing with high-dimensional and time-series complex neuroimaging data, and it is difficult to effectively extract deep temporal dependence structures and potential feature representations. In view of this problem, deep learning technology, especially time series modeling based on Transformer model, has shown superior performance in medical image analysis. The Transformer model has been widely used in neuroimaging research due to its strong sequence modeling ability and self-attention mechanism. Compared with traditional methods, it can effectively handle information interaction across time points and capture the dynamic evolution of hidden patterns.

[0005] Therefore, how to develop a method with time series modeling and feature induction ability by using the powerful modeling ability of the Transformer, which can identify the potential evolution pattern of individuals in the disease course from the dynamic dimension, and then provide strong technical support for individual difference research, scientific classification and precision medicine of mental illness is a difficult problem that needs to be solved at present. SUMMARY

[0006] The present application aims to overcome the deficiencies of the prior art in multi-time point neural image data modeling and individualized feature extraction, and provides a longitudinal neural image data modeling and feature clustering method based on a Transformer model, which can model and cluster analysis the cross-time point change pattern in neural image data, so as to extract discriminative connected feature evolution trajectories, for supporting subsequent individual difference analysis and data-driven grouping research.

[0007] In order to achieve the above purpose, the present application provides a longitudinal neural image data modeling and feature clustering method based on a Transformer model, comprising the following steps:

[0008] S1, obtaining longitudinal neural image data of a subject at three time points, and converting into a brain functional connection matrix; calculating the second-order difference relationship between the brain functional connection matrices of adjacent time points to generate a second-order difference brain network;

[0009] S2, dividing the brain functional connection matrix into a plurality of resting state sub-networks, introducing a multi-head attention mechanism based on the Transformer, capturing the interaction relationship between the sub-networks, and dividing the second-order difference brain network of the subject into a plurality of interaction sub-matrices between the sub-networks;

[0010] S3, using an encoder containing graph convolution operation and interaction attention operation to encode the interaction sub-matrix to obtain an interaction embedding vector of the subject;

[0011] S4, constructing a memory bank containing a plurality of memory vectors, matching the interaction embedding vector of the subject with the memory vectors to obtain a new representation vector of the subject, and a score list of the matching of the interaction embedding vector of the subject with each memory vector;

[0012] S5, introducing a graph decoder to perform graph reconstruction with the new representation vector; at the same time, based on the obtained score list, determining a clustering center by a K-means method.

[0013] The further preferred technical solution of the present application is that in step S1, longitudinal neural image data of a subject at three time points is obtained, and is converted into a brain functional connection matrix, and the second-order difference relationship between the brain functional connection matrices of adjacent time points is calculated to generate a second-order difference brain network; specifically:

[0014] Longitudinal neuroimaging data of the subjects at three time points is acquired and converted into brain functional connectivity matrices, which are then standardized and preprocessed;

[0015] Based on adjacent time points, the second-order difference relationship of the brain functional connectivity matrix is calculated, and the calculation formula is:

[0016] ;

[0017] Wherein, represents the brain functional connectivity matrix at T time point;

[0018] The generated second-order difference brain network data is represented as , n represents the sample size, represents the integrated data of the second-order difference relationship of the s-th subject, is the second-order difference relationship data of the first time point and the second time point, is the second-order difference relationship data of the second time point and the third time point.

[0019] As preferred, the Yeo-7 network atlas is used in step S2 to divide the brain functional connectivity matrix into 7 resting state sub-networks, namely visual network VIS, somatic motor network SMN, dorsal attention network DAN, ventral attention network VAN, edge network LIM, frontal-parietal network FPN and default mode network DMN;

[0020] The introduced multi-head attention mechanism based on Transformer captures the interaction relationship between each sub-network, specifically:

[0021] The fMRI time series of each sub-network is cut into fixed-length segments, each segment is mapped to a fixed-dimensional feature vector through linear transformation, and a learnable position encoding is added to retain the spatio-temporal information;

[0022] The attention head is calculated independently for each sub-network, the correlation weight between the sub-networks is calculated, the features are weighted and fused after softmax normalization, then the output of the attention head is spliced and integrated through a linear layer;

[0023] Spatial self-attention is applied to the attention-weighted features, and the dimension is compressed through a fully connected layer to output a representation containing the interaction features between the sub-networks.

[0024] As preferred, the encoder in step S3 includes a graph convolution operation , an interaction attention operation , and an aggregation function ; the second-order difference brain network of the p-th sub-network of the s-th subject is encoded by the encoder mapping to high-dimensional space , denoted as ; specifically including:

[0025] the graph convolution operation is defined as , where respectively represent the learnable weights of the horizontal filter and the vertical filter, is the weight of the last layer information aggregation operation of the graph convolution, and the network interaction embedding of each second-order difference brain network of the s-th subject is calculated, denoted as:

[0026] ;

[0027] The multi-head attention mechanism is used to focus on the relationship between different network nodes, and the attention embedding is obtained through the QKV attention mechanism, denoted as ;

[0028] The aggregation function is used to aggregate the network interaction embedding of the embedding attention mechanism of each second-order difference brain network to obtain the interaction embedding vector of the subject , denoted as:

[0029] .

[0030] As preferred, the memory bank containing multiple memory vectors is constructed in step S4, the subject interaction embedding vector is matched with the memory vectors to obtain the new representation vector of the subject and the score list of the matching of the subject interaction embedding vector with each memory vector; Specifically:

[0031] Initialize memory vectors as the brain biomarker bank;

[0032] Match the subject interaction embedding vector with the memory vector,

[0033] Define the interaction embedding vector of each subject to be represented by the weighted combination of the memory vectors in the biomarker bank, and introduce the following matching mechanism to interact the subject interaction embedding vector with the memory vector:

[0034] ;

[0035] ;

[0036] ;

[0037] where, is a learnable weight; is the s-th The memory vector is the weight assigned to the i-th subject, representing the i-th pattern score of the i-th subject The memory vector is the weight assigned to the i-th subject, representing the i-th pattern score of the i-th subject The memory vector is the weight assigned to the i-th subject, representing the i-th pattern score of the i-th subject The memory vector is the weight assigned to the i-th subject, representing the i-th pattern score of the i-th subject The memory vector is the weight assigned to the i-th subject, representing the i-th pattern score of the i-th subject The memory vector is the weight assigned to the i-th subject, representing the i-th pattern score of the i-th subject

[0038] As a preferred, a graphical decoder is introduced in step S5, by The memory vector is the weight assigned to the i-th subject, representing the i-th pattern score of the i-th subject

[0039] As a preferred, after the graphical reconstruction, the encoder, the memory vector and the reconstructed image are optimized with a comprehensive loss function, represented as:

[0040]

[0041] Wherein, is the contrast loss, used to promote the learning of the interactive embedding vector;

[0042] is the triple loss function, used to update the memory vector, represented as:

[0043]

[0044] Wherein, is the first two memory vectors of the s-th subject according to the weight , taking as the anchor, ensuring to store more information from the anchor than ;

[0045] For the reconstructed graph , a reconstruction loss is used to make it close to the input graph, and the reconstruction loss is represented as:

[0046]

[0047] is the orthogonal regularization loss of the memory vector, represented as:

[0048] .

[0049] Another aspect of the present application provides a non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to execute the above-mentioned longitudinal neural image data modeling and feature clustering method based on the Transformer model.

[0050] ​​​In still another aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, and the processor invokes a logical instruction in the memory to execute the longitudinal neural image data modeling and feature clustering method based on the Transformer model.

[0051] In still another aspect, the present application provides a computer program product, comprising a computer program stored on a non-transitory computer readable storage medium, wherein the computer program is executed by a processor to cause a computer to execute the longitudinal neural image data modeling and feature clustering method based on the Transformer model.

[0052] Compared with the prior art, the longitudinal neural image data modeling and feature clustering method based on the Transformer model of the present application has the following beneficial effects:

[0053] (1) Deep modeling of longitudinal dynamic features is achieved: by introducing the Transformer model, the present application can effectively capture the time evolution characteristics of brain functional connectivity by deeply modeling the neural image data of individuals with mental illness at multiple time points. Compared with traditional static analysis methods, the present application breaks through the limitation of ignoring time dimension information and provides more comprehensive and detailed brain function change trend and evolution track. This dynamic modeling method helps to reveal the potential neurobiological mechanisms in the disease process and promotes the deep understanding of the relationship between disease development and clinical symptoms.

[0054] (2) Sensitivity and expression ability of connection pattern grouping are enhanced: by using the time sequence feature embedding extracted by the Transformer and combining with the clustering algorithm for individual difference analysis, the present application can accurately identify the significant structural differences in dynamic connection features of different patients. Compared with traditional methods, the present application shows higher sensitivity and stronger expression ability in connection pattern recognition, can effectively distinguish subpopulations showing different evolution trajectories in the disease course, and improves the recognition accuracy and interpretability of potential connection evolution patterns.

[0055] (3) Good adaptability and expansibility: the present method is compatible with multiple longitudinal neural image modalities (such as DTI), can be widely used in different mental illness related research scenarios, has strong cross-task transfer ability and practical promotion potential. In addition to mental illness research, the present application is also applicable to other neural disease individual feature analysis tasks that require long-term tracking and dynamic modeling.

[0056] (4) Advanced technology, high degree of automation: the application adopts the Transformer deep learning framework, which can automatically extract key connection features from high-dimensional, complex, and multi-time point neuroimaging data. This method significantly reduces the dependence on artificial feature design, improves the efficiency of data processing and the intelligent level of the system, and avoids the manual intervention and subjective bias in traditional methods. In addition, the high degree of automation of this method makes it have good system implementation and engineering transformation prospects.

[0057] In summary, the application combines deep learning with longitudinal neuroimaging analysis and constructs an efficient and automated time series feature modeling and clustering analysis method, which can be used as an intelligent auxiliary analysis tool in mental illness research and has important scientific research application value and broad popularization prospects in supporting individual difference research and connection mode exploration. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The flowchart of the longitudinal neuroimaging data modeling and feature clustering method based on the Transformer model of the application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the application clearer, the technical scheme in the application will be described clearly and completely below in combination with the drawings in the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments, and they should not be understood as limiting the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the application. In the description of the application, it should be understood that the terms used are only for the purpose of description, and should not be understood as indicating or implying relative importance.

[0060] The following will be described in combination with Figure 1 The longitudinal neuroimaging data modeling and feature clustering method based on the Transformer model provided by the application is described.

[0061] Embodiment 1: The embodiment provides a longitudinal neuroimaging data modeling and feature clustering method based on the Transformer model. The method includes two stages, each stage consists of multiple ordered steps, and each step has a clear data flow and logical dependence relationship.

[0062] As shown in Figure 1 , the first stage is an encoding stage based on subnetwork connection, and the second stage is to build a subnetwork interaction mode library learning network.

[0063] The first stage aims to construct sub-network connection representation of the dynamic characteristics of brain region interaction from longitudinal neuroimaging data, to provide structured input for subsequent clustering analysis. Specifically, the following steps are included:

[0064] S1, construction of second-order difference brain network.

[0065] Longitudinal neuroimaging data of subjects at three time points are obtained and converted into brain functional connectivity matrix. Functional magnetic resonance imaging (fMRI) technology has been able to map the functional connectivity (FC) atlas between brain regions, and this method can convert longitudinal neuroimaging data into brain functional connectivity matrix.

[0066] Then the second-order difference relationship between the brain functional connectivity matrices of adjacent time points is calculated to generate a second-order difference brain network;

[0067] Based on adjacent time points, the second-order difference relationship of the brain functional connectivity matrix is calculated, and the calculation formula is:

[0068]

[0069] wherein, represents the brain functional connectivity matrix at T time point;

[0070] The generated second-order difference brain network data is represented as , n represents the sample size, represents the integrated data of the second-order difference relationship of the s-th subject, is the second-order difference relationship data of the first time point and the second time point, is the second-order difference relationship data of the second time point and the third time point, and this network aims to emphasize the regions in the whole brain network that have undergone significant changes between two consecutive time points.

[0071] S2, brain sub-network interaction learning.

[0072] According to the brain region division template, the brain functional connectivity matrix is divided into 7 resting state sub-networks using Yeo-7 network atlas, namely visual network VIS, somatic motor network SMN, dorsal attention network DAN, ventral attention network VAN, limbic network LIM, frontal-parietal network FPN and default mode network DMN.

[0073] ​The embodiment studies the interaction characteristics between sub-networks instead of relying on artificially designed features, which can more effectively capture significant features between subjects and achieve more accurate clustering. Compared with traditional methods, this method can automatically learn the potential interaction relationship from the original data, enhancing the description of the dynamic evolution of complex functional brain networks. When modeling this interaction process through graph theory, the learning of the association between sub-networks is crucial, and the Transformer-based method has been proven to be a robust method for modeling associations. Therefore, the brain sub-network interaction module based on multi-head attention of Transformer is introduced in the embodiment to capture interaction features that are helpful for subsequent analysis.

[0074] The specific method for capturing the interaction relationship between each sub-network in the embodiment introducing the multi-head attention mechanism based on Transformer is:

[0075] The fMRI time series of each sub-network is cut into fixed-length segments, each segment is mapped to a fixed-dimensional feature vector through linear transformation, and learnable position encoding is added to retain spatial and temporal information;

[0076] The attention head is calculated independently for each sub-network, the correlation weight between sub-networks is calculated, the features are weighted and fused after softmax normalization, then the outputs of the attention heads are spliced and integrated through a linear layer;

[0077] Spatial self-attention is applied to the attention-weighted features, and the dimension is compressed through a fully connected layer, outputting a representation containing the interaction features between sub-networks.

[0078] S3, interaction embedding vector generation.

[0079] The embodiment constructs an encoder, which includes a graph convolution operation aiming to learn the representation of sub-network connection information, an interaction attention operation to effectively simulate the interaction of brain sub-networks, and an aggregation function to further extract effective information. The second-order difference brain network of the pth sub-network of the s th subject is mapped to a high-dimensional space using the encoder, represented as .

[0080] Specifically, the graph convolution operation is defined as , where represent the learnable weights of the horizontal filter and the vertical filter, respectively, The weights of the last layer of information aggregation operation in graph convolution are used to calculate the network interaction embedding of each second-order difference brain network of the s-th subject, expressed as:

[0081] ;

[0082] Then, the encoder utilizes a multi-head attention mechanism to focus on the relationships between different network nodes, effectively simulating their interactions. The attention embedding is obtained through the QKV attention mechanism, represented as... ;

[0083] Next, using aggregate functions The network interaction of the embedded attention mechanism of each second-order differential brain network is embedded. By combining these data, we obtain the participants' interaction embedding vectors. , represented as:

[0084] .

[0085] The second stage, based on the aforementioned embedding representation, introduces representative brain connectivity prototypes and clustering mechanisms to achieve dynamic modeling of individual differences and discovery of potential patterns.

[0086] S4, Brain biological model library learning.

[0087] Inspired by memory networks, this embodiment proposes the construction of a brain biomarker library. This library can store typical evolutionary patterns, facilitating personalized matching with individual subjects.

[0088] initialization memory vectors As a library of brain biomarkers, these can be updated during the optimization process. These memory vectors are orthogonal to each other and can remember different information. Therefore, orthogonal regularization is introduced. .

[0089] The interaction between the biomarker library and different participants is achieved through a search and matching process. It is assumed that the high-dimensional embedding of each participant can be represented by a weighted combination of memory vectors from the biomarker library, and the specificity among participants is reflected by assigning different weights to these memory vectors. Specifically, the following matching mechanism is introduced:

[0090] ;

[0091] ;

[0092] ;

[0093] in, These are learnable weights; It is the first The memory vector is the first weight assigned to the subject, more specifically, also represents the first pattern score of the subject, so the pattern score of the subject represented by the memory vector can be expressed as a combination of value list These individual scores sensitively reflect individualized characteristics; is a new representation vector generated by the weighted combination of memory vectors.

[0094] S5, structure reconstruction.

[0095] In order to enable the model to focus on more useful meta-path-based interaction information, the input graph is reconstructed using the bank memory vector. Specifically, a graph decoder is introduced to Output the reconstructed graph as the reconstructed neural image.

[0096] S6, optimization target design.

[0097] After graph reconstruction, the encoder, memory vector and reconstructed image are optimized with a comprehensive loss function, represented as:

[0098] ;

[0099] Where, is the contrastive loss, which is used to promote the learning of interaction embedding vectors;

[0100] is a triple loss function used to update the memory vector, represented as:

[0101] ;

[0102] Where, is the first two memory vectors of the s-th subject sorted according to the weight , which is regarded as an anchor to ensure stores more information from the anchor than ;

[0103] For the reconstructed graph , a reconstruction loss is used to make it close to the input graph, and the reconstruction loss is represented as:

[0104] ;

[0105] is the orthogonal regularization loss of the memory vector, represented as:

[0106] . ​​

[0107] S7, score-based interaction mode clustering.

[0108] Each subject is assigned a list of scores by matching the interaction embedding with the memory vector After analyzing these score lists, it is found that the scores of the subjects are similar, but there are different groups. By K-means method, the clustering center can be determined.

[0109] Embodiment 2: The embodiment provides a non-transitory computer readable storage medium, which stores computer instructions, the computer instructions causing a computer to execute a longitudinal neural image data modeling and feature clustering method based on a Transformer model, the method comprising the following steps:

[0110] S1, obtaining longitudinal neural image data of a subject at three time points, and converting into a brain functional connection matrix; calculating the second-order difference relationship between the brain functional connection matrices of adjacent time points to generate a second-order difference brain network;

[0111] S2, dividing the brain functional connection matrix into a plurality of resting state sub-networks, introducing a multi-head attention mechanism based on Transformer, capturing the interaction relationship between the sub-networks, and dividing the second-order difference brain network of the subject into a plurality of interaction sub-matrices between the sub-networks;

[0112] S3, using an encoder containing graph convolution operation and interaction attention operation to encode the interaction sub-matrix to obtain an interaction embedding vector of the subject;

[0113] S4, constructing a memory bank containing a plurality of memory vectors, matching the interaction embedding vector of the subject with the memory vectors to obtain a new representation vector of the subject and a score list of the matching of the interaction embedding vector of the subject with each memory vector;

[0114] S5, introducing a graph decoder to perform graph reconstruction with the new representation vector; at the same time, based on the obtained score list, a clustering center is determined by a K-means method.

[0115] Embodiment 3: The embodiment provides an electronic device, which can include a processor, a communications interface, a memory and a communications bus, wherein the processor, the communications interface and the memory complete mutual communication through the communications bus. The processor can call logical instructions in the memory to execute a longitudinal neural image data modeling and feature clustering method based on a Transformer model, the method comprising the following steps:

[0116] S1, obtain longitudinal neural image data of a subject at three time points, and convert the data into a brain functional connection matrix; calculate the second-order difference relationship between the brain functional connection matrices at adjacent time points to generate a second-order difference brain network;

[0117] S2, divide the brain functional connection matrix into multiple resting state sub-networks, introduce a multi-head attention mechanism based on a Transformer, capture the interaction relationship between the sub-networks, and divide the second-order difference brain network of the subject into an interaction sub-matrix between multiple sub-networks;

[0118] S3, use an encoder containing graph convolution operation and interaction attention operation to encode the interaction sub-matrix to obtain an interaction embedding vector of the subject;

[0119] S4, construct a memory bank containing multiple memory vectors, match the interaction embedding vector of the subject with the memory vectors to obtain a new representation vector of the subject, and a score list of matching the interaction embedding vector of the subject with each memory vector;

[0120] S5, introduce a graph decoder to perform graph reconstruction with the new representation vector; at the same time, based on the obtained score list, determine a cluster center by a K-means method.

[0121] In addition, the logical instructions in the memory described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0122] Embodiment 4: The present embodiment provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, and the computer program being executable by a processor to enable a computer to execute a longitudinal neural image data modeling and feature clustering method based on a Transformer model, the method comprising the following steps:

[0123] S1, obtain longitudinal neural image data of a subject at three time points, and convert the data into a brain functional connection matrix; calculate the second-order difference relationship between brain functional connection matrices at adjacent time points to generate a second-order difference brain network;

[0124] S2, divide the brain functional connection matrix into a plurality of resting state sub-networks, introduce a multi-head attention mechanism based on a Transformer, capture the interaction relationship between the sub-networks, and divide the second-order difference brain network of the subject into an interaction sub-matrix between the plurality of sub-networks;

[0125] S3, encode the interaction sub-matrix by using an encoder containing graph convolution operation and interaction attention operation to obtain an interaction embedding vector of the subject;

[0126] S4, construct a memory bank containing a plurality of memory vectors, match the interaction embedding vector of the subject with the memory vectors to obtain a new representation vector of the subject, and a score list of matching the interaction embedding vector of the subject with each memory vector;

[0127] S5, introduce a graph decoder to perform graph reconstruction with the new representation vector; at the same time, based on the obtained score list, determine a clustering center by a K-means method.

[0128] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0129] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0130] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for modeling and feature clustering longitudinal neural imaging data based on the Transformer model, characterized in that, Includes the following steps: S1. Acquire longitudinal neuroimaging data of the subjects at three time points and convert them into brain functional connectivity matrices; calculate the second-order difference relationship between brain functional connectivity matrices at adjacent time points to generate a second-order difference brain network. S2. Divide the brain functional connectivity matrix into multiple resting state sub-networks, introduce a Transformer-based multi-head attention mechanism, capture the interaction relationship between each sub-network, and divide the subject's second-order difference brain network into multiple interaction sub-matrices between sub-networks. S3. Encode the interaction submatrix using an encoder that includes graph convolution operations and interaction attention operations to obtain the subject's interaction embedding vector. S4. Construct a memory bank containing multiple memory vectors, match the subject's interaction embedding vector with the memory vectors to obtain the subject's new representation vector, and a score list of the subject's interaction embedding vector matching each memory vector; S5. Introduce a graph decoder to reconstruct the graph using a new representation vector; simultaneously, determine the cluster centers using the K-means method based on the obtained score list.

2. The method for longitudinal neural imaging data modeling and feature clustering based on the Transformer model according to claim 1, characterized in that, In step S1, longitudinal neuroimaging data of the subject at three time points are acquired and converted into brain functional connectivity matrices. The second-order difference relationship between brain functional connectivity matrices at adjacent time points is calculated to generate a second-order difference brain network; specifically: Longitudinal neuroimaging data of subjects at three time points were acquired and converted into brain functional connectivity matrices. Then, the brain functional connectivity matrices were standardized and preprocessed. Based on adjacent time points, the second-order difference relationship of the brain functional connectivity matrix is ​​calculated using the following formula: ； in, This represents the brain functional connectivity matrix at time point T. The generated second-order difference brain network data is represented as follows , where n represents the sample size. The integrated data representing the two second-order difference relationships of the s-th subject. It is the second-order difference relationship data between the first and second time points. It is the second-order difference relationship data between the second and third time points.

3. The method for longitudinal neural imaging data modeling and feature clustering based on the Transformer model according to claim 1, characterized in that, Step S2 uses the Yeo-7 network map to divide the brain functional connectivity matrix into 7 resting state sub-networks, namely the visual network (VIS), somatic motor network (SMN), dorsal attention network (DAN), ventral attention network (VAN), limbic network (LIM), frontoparietal network (FPN), and default mode network (DMN). The introduction of a Transformer-based multi-head attention mechanism to capture the interaction relationships between sub-networks specifically involves: The fMRI time series of each sub-network is divided into segments of fixed length. Each segment is mapped to a feature vector of fixed dimension through linear transformation, and learnable positional encoding is added to preserve spatiotemporal information. The attention head is computed independently for each sub-network, the correlation weights between sub-networks are calculated, the features are weighted and fused after normalization by softmax, and then the outputs of the attention heads are concatenated and integrated through a linear layer. Spatial self-attention is applied to the attention-weighted features, and the dimensionality is compressed through a fully connected layer to output a representation that includes the interaction features between subnetworks.

4. The method for longitudinal neural imaging data modeling and feature clustering based on the Transformer model according to claim 1, characterized in that, The encoder in step S3 includes a graph convolution operation. An interactive attention operation and aggregate functions The second-order difference brain network of the p-th subnetwork of the s-th subject was generated using an encoder. Mapping to higher-dimensional space , represented as ; Specifically, it includes: Convolution operation on graphs Defined as ,in These represent the learnable weights of the horizontal and vertical filters, respectively. The weights of the last layer of information aggregation operation in graph convolution are used to calculate the network interaction embedding of each second-order difference brain network of the s-th subject, expressed as: ； Multi-head attention is used to focus on the relationships between different network nodes. The attention embedding is obtained through the QKV attention mechanism and is represented as follows: ; Using aggregate functions The network interaction of the embedded attention mechanism of each second-order differential brain network is embedded. By combining these data, we obtain the participants' interaction embedding vectors. , is represented as: 。 5. The method for longitudinal neural imaging data modeling and feature clustering based on the Transformer model according to claim 4, characterized in that, Step S4 involves constructing a memory bank containing multiple memory vectors, matching the subject's interaction embedding vector with the memory vectors to obtain the subject's new representation vector, and a score list of the matching between the subject's interaction embedding vector and each memory vector; specifically: initialization memory vectors As a brain biomarker library; The subject's interaction embedding vector is matched with the memory vector. Each participant's interaction embedding vector is defined as a weighted combination of memory vectors from a biomarker library. A matching mechanism is introduced to interact the participant's interaction embedding vector with the memory vectors: ； ； ； in, These are learnable weights; It is the first The memory vector is the first one. The weight assigned to the nth subject represents the weight of the nth subject. The first subject Each participant was assigned a score list based on the pattern. ; It is a new representation vector generated by weighted combination of memory vectors.

6. The method for longitudinal neural imaging data modeling and feature clustering based on the Transformer model according to claim 5, characterized in that, Step S5 introduces a graphics decoder, through Output the reconstructed graphics.

7. The method for longitudinal neural imaging data modeling and feature clustering based on the Transformer model according to claim 6, characterized in that, After image reconstruction, the encoder, memory vector, and reconstructed image are optimized using a comprehensive loss function, as follows: ； in, To compare the loss, it is used to facilitate the learning of interactive embedding vectors; The triple loss function, used to update the memory vector, is expressed as: ； in, It is based on weight The first two memory vectors of the s-th subject in the sorting will Treat it as an anchor to ensure Compare Store more information from the anchor; For the reconstructed image A reconstruction loss was used to approximate the input map. The reconstruction loss is expressed as: ； The orthogonal regularization loss of the memory vector is expressed as: 。 8. A non-transitory computer-readable storage medium, characterized in that, It stores computer instructions that cause the computer to execute the longitudinal neuroimaging data modeling and feature clustering method based on the Transformer model as described in any one of claims 1-7.

9. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor calls logical instructions from the memory to execute the longitudinal neural imaging data modeling and feature clustering method based on the Transformer model as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer performs the longitudinal neural imaging data modeling and feature clustering method based on the Transformer model as described in any one of claims 1-7.

Images