A machine learning-based multimodal medical image data analysis method

Abstract

This invention discloses a machine learning-based method for multimodal medical image data analysis, relating to the field of medical image processing technology. The method includes the following steps: S1, outputting a set of keypoint spatial coordinates; S2, constructing a cross-modal heterogeneous graph; S3, outputting the adjacency matrix of the cross-modal heterogeneous graph; S4, generating a modal contribution weight matrix; S5, performing weighted heterogeneous aggregation based on an improved MPNN model to generate distortion features, and introducing a shape memory alloy phase transformation feature recovery mechanism based on materials science to adaptively drive distortion feature shrinkage compensation and output a high-order graph node feature matrix; S6, outputting a multimodal fused graph-level vector; S7, outputting the medical image data analysis results. This invention overcomes the limitations of traditional methods, such as static graph topological distortion, imbalance of modal aggregation contributions, and semantic shift of high-order features, providing a precise and adaptive solution for multimodal medical image data analysis.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a multimodal medical image data analysis method based on machine learning. Background Technology

[0002] With the explosive growth of multimodal medical image data, classical machine learning algorithms face severe challenges in feature fusion for deep interaction and accurate decision-making across modal features. Existing multimodal medical image analysis methods, such as fusion models based on graph neural networks, while improving the efficiency of cross-modal feature interaction by utilizing the message passing mechanism of graph structures, mainly rely on preset spatial distance thresholds or static topological connections for heterogeneous graph construction and feature aggregation. This method, which is based solely on static spatial graph construction, ignores the implicit dynamic semantic relationships in multimodal image data (such as the heterogeneity of cross-modal lesion responses and semantic shifts between modalities) and deep topological evolution characteristics, leading to semantic topological breaks and structural redundancy when constructing cross-modal heterogeneous graphs, thus limiting the accuracy of cross-modal feature interaction. Furthermore, classical graph aggregation methods often fail to adequately constrain semantic distortion during multimodal feature aggregation. The use of fixed-weight aggregation mechanisms leads to distortion of the contributions of nodes from different modalities, and high-order feature shifts that deviate from the original semantics are easily generated after aggregation. Meanwhile, in the graph-level feature compression and discrimination decision stages, existing methods mostly rely on static truncation and feature mapping, lacking adaptive baseline metrics based on data distribution characteristics. This results in inaccurate allocation of key feature weights and scale shifts in discrimination mapping, increasing the risk of misjudgment in multimodal medical image data analysis.

[0003] Therefore, how to provide a machine learning-based multimodal medical image data analysis method is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention proposes a machine learning-based multimodal medical image data analysis method. Through a dynamic filtering and reconstruction mechanism based on semantic similarity statistical distribution, it performs low-semantic topological edge removal and cross-modal semantic bridging reconstruction on cross-modal heterogeneous graphs, outputting a cross-modal heterogeneous graph adjacency matrix. The semantic correlation degree of the node pairs at both ends of the cross-modal edge is calculated and mapped to generate a modal contribution weight matrix, using the initial vector of the topologically connected interactive nodes defined by the cross-modal heterogeneous graph adjacency matrix. Based on an improved MPNN model, weighted heterogeneous aggregation is performed on the source-end features using the cross-modal heterogeneous graph adjacency matrix and the modal contribution weight matrix to generate a high-order hidden state vector of distortion. A memory alloy phase transition feature recovery mechanism based on materials science is introduced, setting the node initial vector as the original semantic anchor point, calculating the feature offset as a semantic temperature parameter, and adaptively calculating a dynamic phase transition benchmark based on the distribution mean and distribution dispersion to generate a shrinkage coefficient, driving the distorted features to perform dynamic shrinkage compensation towards the original semantic anchor point. This mechanism effectively eliminates the accumulation of semantic distortion during heterogeneous aggregation by establishing an adaptive rectification path from "semantic offset measurement" to "feature elastic shrinkage," ensuring that the high-order graph node feature matrix output by the improved MPNN model can dynamically maintain the consistency of the original multimodal semantic anchors. Furthermore, based on the cross-modal heterogeneous graph adjacency matrix and feature deviation metric, it performs adaptive offset filtering and weighted pooling compression, and applies adaptive scaling normalization and learnable discriminative mapping based on a dynamic scale baseline to the resulting graph-level vectors. This invention overcomes the limitations of traditional methods, such as static graph topological distortion, imbalanced modal aggregation contributions, and semantic offset of high-order features, providing a precise and adaptive solution for multimodal medical image data analysis.

[0005] A multimodal medical image data analysis method based on machine learning according to an embodiment of the present invention specifically includes: S1. Acquire multimodal medical image data, perform spatial registration to align the multimodal spatial topology, sequentially encode and extract the grid feature map of each modality along the feature channel, and measure the spatial saliency response in parallel to extract the output key point spatial coordinate set; S2. Based on the spatial metric of the key point spatial coordinate set, construct the same-modal and cross-modal spatial edges, anchor the node initial vector at the corresponding position of the key point coordinate in the mesh feature map, and construct the cross-modal heterogeneous graph by combining the corresponding spatial edges. S3. Based on the cross-modal heterogeneous graph, calculate the semantic similarity between the nodes at both ends of the topological edge according to the node initial vector, filter out the low semantic topological edges based on the statistical distribution features of semantic similarity, perform cross-modal semantic reconstruction to bridge the topological gaps, and output the cross-modal heterogeneous graph adjacency matrix. S4. Calculate the semantic correlation degree of the node pairs at both ends of the cross-modal heterogeneous graph connected by the initial vector of the topological connection interaction nodes defined by the adjacency matrix of the cross-modal heterogeneous graph, and map the correlation degree to generate the modal contribution weight matrix. S5. Based on the improved MPNN model, the source features are weighted heterogeneous aggregation by combining the cross-modal heterogeneous graph adjacency matrix and modal contribution weight matrix to generate distorted high-order hidden state vectors; a memory alloy phase transformation feature recovery mechanism based on materials science is introduced, and the distorted features are dynamically shrunk back to the initial semantic anchor point based on feature offset adaptive drive, and the high-order graph node feature matrix is ​​output. S6. Measure the criticality of the feature matrix of the higher-order graph node, compress and map the feature matrix of the higher-order graph node according to the criticality, and output the multimodal fused graph-level vector. S7. Perform learnable discriminant mapping on the multimodal fusion graph-level vectors to obtain the category response probability, and output the medical image data analysis results based on probability decision.

[0006] Optionally, S1 specifically includes: S11. Perform a spatial affine transformation on the acquired multimodal medical image data to align the multimodal spatial topology and output the registered image data; S12. Based on the registered image data, perform convolutional coding and downsampling operations sequentially along the feature channels to output the grid feature maps of each modality; S13. Based on the mesh feature maps of each modality, calculate the local feature response amplitude along the spatial dimension, and extract the coordinates of the peak points of the response amplitude to generate an initial spatial coordinate set; S14. Statistically analyze the global response distribution characteristics of the initial spatial coordinate set, calculate the dynamic response threshold, filter out coordinate points below the dynamic response threshold, and output the key point spatial coordinate set.

[0007] Optionally, S2 specifically includes: S21. Based on the key point spatial coordinate set, calculate the first Euclidean distance of key point pairs within the same modality and the second Euclidean distance of key point pairs across modalities, statistically analyze the global spatial distribution characteristics of the first Euclidean distance and the second Euclidean distance, and calculate the dynamic same-modal spatial threshold and the dynamic cross-modal spatial threshold. S22. Filter out keypoint pairs corresponding to the first Euclidean distance that is greater than the dynamic same-modal space threshold, and connect the retained same-modal keypoint pairs to output same-modal space edges; filter out keypoint pairs corresponding to the second Euclidean distance that is greater than the dynamic cross-modal space threshold, and connect the retained cross-modal keypoint pairs to output cross-modal space edges. S23. Based on the key point spatial coordinate set, perform feature slicing extraction operation at the corresponding spatial position of the grid feature map and output the node initial vector; S24. Based on the node initial vectors and edges in the same modal space and across modal spaces, perform graph structure assembly operations to output a cross-modal heterogeneous graph.

[0008] Optionally, S3 specifically includes: S31. Based on the cross-modal heterogeneous graph and node initial vectors, calculate the first cosine similarity of the initial vectors of the nodes at both ends of the same-modal topological edge as the same-modal semantic similarity, calculate the heterogeneous response probability of the initial vectors of the nodes at both ends of the cross-modal topological edge as the cross-modal semantic similarity, statistically analyze the global distribution characteristics of same-modal semantic similarity and cross-modal semantic similarity, and calculate the dynamic semantic filtering threshold. S32. Remove low semantic homomodal topological edges and low semantic crossmodal topological edges that are below the dynamic semantic filtering threshold, and output the initial cleaned topological edge set; S33. Based on the initial cleaned topology edge set, extract cross-modal topology void regions, calculate the heterogeneous reconstruction response probability between the initial vectors of the boundary nodes of the void regions, connect the boundary node pairs whose heterogeneous reconstruction response probabilities satisfy the dynamic semantic filtering threshold to generate reconstructed cross-modal edges, and merge the initial cleaned topology edge set and the reconstructed cross-modal edges to output the reconstructed topology edge set. S34. Based on the reconstructed topological edge set, perform graph adjacency mapping and output the cross-modal heterogeneous graph adjacency matrix.

[0009] Optionally, S4 specifically includes: S41. Based on the adjacency matrix and node initial vectors of the cross-modal heterogeneous graph, perform message passing interaction operations along the cross-modal topology connections defined by the adjacency matrix of the cross-modal heterogeneous graph, and output the same-modal interaction node vectors and cross-modal interaction node vectors. S42. Calculate the same-modal semantic preservation degree based on the same-modal interaction node vector and node initial vector, calculate the cross-modal semantic transfer degree based on the cross-modal interaction node vector and node initial vector, and output the semantic association degree of the node pairs at both ends of the cross-modal edge by fusing the same-modal semantic preservation degree and the cross-modal semantic transfer degree. S43. Perform grouping statistical operation on the semantic relevance based on the modality attribution of cross-modal edges, calculate the dynamic intra-group relevance baseline of each group of modal pairs, perform normalization offset operation on the semantic relevance based on the dynamic intra-group relevance baseline, and output the cross-modal edge weights. S44. Perform global mean pooling on the cross-modal edge weights based on the modal attribution of the cross-modal edges. Use the mean of the modal pairs obtained by pooling as the evolution source features. Perform matrix mapping operation on the evolution source features and output the modal contribution weight matrix.

[0010] Optionally, the improved MPNN model includes a source message construction layer, a topology path propagation layer, a semantic aggregation and update layer, a shape memory alloy phase change rectification layer, and a feature matrix output layer. The source-end message construction layer is used to define the message passing topology path with the cross-modal heterogeneous graph adjacency matrix, extract the initial vector of the source node as the sender, and perform source-end weighting on the initial vector of the node according to the corresponding weight coefficient in the modal contribution weight matrix, and output the weighted source-end message vector. The topology path transmission layer is used to route and transmit the weighted source message vector along the corresponding cross-modal and same-modal space edges according to the topological connection relationship defined by the cross-modal heterogeneous graph adjacency matrix, distribute it to each target node as the receiving end, and output the target node received message set. The semantic aggregation update layer is used to perform aggregation operations on the weighted source message vectors in the message set received by each target node, concatenate the aggregation result with the node initial vector of the target node itself, and output the distorted higher-order hidden state vector of the target node through nonlinear transformation mapping and normalization processing. The shape memory alloy phase change rectifier layer is used to introduce a shape memory alloy phase change characteristic recovery mechanism based on materials science. The specific execution process includes: The node initial vector is set as the original semantic anchor point, and the feature offset of the distorted higher-order hidden state vector relative to the original semantic anchor point is calculated as the semantic temperature parameter. The distribution characteristics of semantic temperature parameters are statistically analyzed along the node dimension. The distribution mean and distribution dispersion are extracted based on the distribution characteristics. The distribution mean and distribution dispersion are linearly combined to adaptively calculate the dynamic phase transition benchmark. The difference between the semantic temperature parameter and the dynamic phase transition reference is calculated, and a nonlinear mapping is performed to generate a continuously differentiable phase transition coefficient, which is then used as a shrinkage coefficient. The distortion of the higher-order hidden state vector is elastically shrunk back to the original semantic anchor point using the shrinkage coefficient, and the phase change rectification feature vector is output. The feature matrix output layer is used to stack and map the phase change rectification feature vectors of all target nodes in the order of nodes, and output a high-order graph node feature matrix.

[0011] Optionally, S6 specifically includes: S61. Based on the feature matrix of a high-order graph node and the adjacency matrix of a cross-modal heterogeneous graph, aggregate the features of neighboring nodes along the topological connection defined by the adjacency matrix of the cross-modal heterogeneous graph to output local mean features, calculate the feature deviation between the node's own features and the local mean features, calculate the global centrality by combining the adjacency matrix of the cross-modal heterogeneous graph, and output the initial key index of a one-dimensional high-order graph node by fusing the feature deviation and the global centrality. S62. Statistically analyze the global distribution characteristics of the initial critical index of the nodes in the one-dimensional high-order graph, calculate the dynamic critical baseline, perform normalization offset operation on the initial critical index of the nodes in the one-dimensional high-order graph based on the dynamic critical baseline, filter out negative offset nodes with values ​​below zero, and output the criticality weight of the nodes in the one-dimensional high-order graph. S63. Perform weighted summation pooling operation on the feature matrix of high-order graph nodes based on the keyness weight of one-dimensional high-order graph nodes, and output a multimodal fused graph-level vector.

[0012] Optionally, S7 specifically includes: S71. Based on the multimodal fusion graph-level vector, statistically analyze the feature amplitude distribution of the multimodal fusion graph-level vector and calculate the dynamic scale baseline. Based on the dynamic scale baseline, perform adaptive scaling and normalization operation on the multimodal fusion graph-level vector and output the scale-normalized graph-level vector. S72. Perform parameterized linear mapping operation on scale-normalized graph-level vectors to output the initial class response logits, and perform normalized exponential mapping on the initial class response logits to output the class response probability. S73. Perform extreme value retrieval operation based on category response probability to output the maximum response probability and its corresponding target category label, and combine the target category label and the maximum response probability to generate medical image data analysis results.

[0013] The beneficial effects of this invention are: (1) This invention achieves precise alignment and interaction between the spatial topology and deep semantics of multimodal images by constructing a cross-modal dynamic topology reconstruction and weighted heterogeneous aggregation mechanism. Based on keypoint spatial metrics, intramodal and cross-modal spatial edges are constructed. Dynamic semantic filtering thresholds are calculated based on the statistical distribution characteristics of semantic similarity. Low-semantic topology edge filtering and cross-modal semantic reconstruction are performed to bridge topological gaps and generate a cross-modal heterogeneous graph adjacency matrix. Semantic correlation is calculated along the initial vectors of interactive nodes connected to the topology. Group statistics and normalized offsets are performed based on modality affiliation, and the mapping evolution generates a modality contribution weight matrix. Based on the improved MPNN model, the cross-modal heterogeneous graph adjacency matrix and the modality contribution weight matrix are combined to perform source-end weighted heterogeneous aggregation. This mechanism constrains the interaction of multimodal features to dynamically purified topological paths and adaptive modality weights, effectively eliminating semantic redundancy and heterogeneous interference in the cross-modal spatial structure, and ensuring that the generated distorted high-order hidden state vectors are accurately integrated with the core semantics of multimodal images.

[0014] (2) This invention establishes a feature semantic anti-offset verification and criticality accurate measurement paradigm by introducing a material science-based memory alloy phase transformation feature recovery mechanism and a full-link dynamic baseline adaptive decision system. The initial node vector is set as the original semantic anchor point. The feature offset of the distorted higher-order hidden state vector relative to the original semantic anchor point is calculated as a semantic temperature parameter. Based on the distribution mean and distribution dispersion, a dynamic phase transformation benchmark is adaptively calculated to generate a shrinkage coefficient, driving the distorted features to perform elastic shrinkage compensation towards the initial semantic anchor point. Based on the cross-modal heterogeneous graph adjacency matrix, the feature deviation degree and global centrality are fused to output the initial criticality index. The dynamic critical baseline is calculated based on the global distribution characteristics, performing normalized offset and negative filtering to extract the criticality weights of higher-order graph nodes. The dynamic scale baseline is calculated for graph-level vectors based on the feature amplitude distribution, performing adaptive scaling normalization. This system achieves adaptive constraints on data distribution throughout the entire process from feature aggregation, graph-level compression to discriminative decision-making through semantic temperature-driven phase transition shrinkage compensation, topology-aware keyness truncation pooling, and scale normalization mapping. This ensures that the output features strictly maintain the consistency of the original semantic anchor points and have extremely high discriminative sensitivity to key lesions. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a multimodal medical image data analysis method based on machine learning proposed in this invention; Figure 2 This is a flowchart illustrating the working principle of the improved MPNN model, a multimodal medical image data analysis method based on machine learning proposed in this invention. Detailed Implementation

[0016] The invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0017] refer to Figure 1 and Figure 2 A multimodal medical image data analysis method based on machine learning, specifically including: S1. Acquire multimodal medical image data, perform spatial registration to align the multimodal spatial topology, sequentially encode and extract the grid feature map of each modality along the feature channel, and measure the spatial saliency response in parallel to extract the output key point spatial coordinate set; S2. Based on the spatial metric of the key point spatial coordinate set, construct the same-modal and cross-modal spatial edges, anchor the node initial vector at the corresponding position of the key point coordinate in the mesh feature map, and construct the cross-modal heterogeneous graph by combining the corresponding spatial edges. S3. Based on the cross-modal heterogeneous graph, calculate the semantic similarity between the nodes at both ends of the topological edge according to the node initial vector, filter out the low semantic topological edges based on the statistical distribution features of semantic similarity, perform cross-modal semantic reconstruction to bridge the topological gaps, and output the cross-modal heterogeneous graph adjacency matrix. S4. Calculate the semantic correlation degree of the node pairs at both ends of the cross-modal heterogeneous graph connected by the initial vector of the topological connection interaction nodes defined by the adjacency matrix of the cross-modal heterogeneous graph, and map the correlation degree to generate the modal contribution weight matrix. S5. Based on the improved MPNN model, the source features are weighted heterogeneous aggregation by combining the cross-modal heterogeneous graph adjacency matrix and modal contribution weight matrix to generate distorted high-order hidden state vectors; a memory alloy phase transformation feature recovery mechanism based on materials science is introduced, and the distorted features are dynamically shrunk back to the initial semantic anchor point based on feature offset adaptive drive, and the high-order graph node feature matrix is ​​output. S6. Measure the criticality of the feature matrix of the higher-order graph node, compress and map the feature matrix of the higher-order graph node according to the criticality, and output the multimodal fused graph-level vector. S7. Perform learnable discriminant mapping on the multimodal fusion graph-level vectors to obtain the category response probability, and output the medical image data analysis results based on probability decision.

[0018] In this embodiment, S1 specifically includes: S11. Read the voxel coordinates and intensities of each modality in the multimodal medical image data, count the joint occurrence frequency of voxel intensities between different modalities and divide by the total number of primes to calculate the joint probability, and calculate the difference between the sum of the entropies of the two edge distributions and the joint distribution entropy as the mutual information based on the joint probability; before optimization, perform initial spatial translation alignment based on the centroid or geometric center of each modality image, adopt a coarse-to-fine multi-resolution strategy, solve the translation, rotation and scaling parameters that maximize the mutual information and substitute them back to the affine transformation matrix, perform spatial affine transformation to align the spatial topology for non-reference modalities, and output the registered image data.

[0019] S12. Based on the registered image data, perform three-dimensional convolutional encoding with a 3x3x3 kernel, a stride of 1, padding of 1, and an activation function of ReLU in sequence, as well as three-dimensional max pooling downsampling with a 2x2x2 window and a stride of 2. Extract the maximum feature value within the pooling window and output the three-dimensional mesh feature map of each modality after downsampling.

[0020] S13. Based on the 3D mesh feature maps of each modality, select the coordinates of the original feature values ​​greater than all 26 adjacent feature points in a 3x3x3 local neighborhood as local spatial maxima. Calculate the mean of the differences between the maxima and the 26 adjacent feature values ​​as the first-order gradient. Calculate the sum of the second-order differences of the maxima along the three spatial axes as the Laplacian response. Calculate the eigenvalues ​​of the local Hessian matrix. If the ratio of the absolute value of the second smallest eigenvalue to the absolute value of the smallest eigenvalue exceeds a preset threshold, it is determined to be a non-blocky edge response and the amplitude is set to 0. Otherwise, calculate the product of the first-order gradient and the Laplacian response and take its absolute value as the local feature response amplitude. Fit a quadratic function in the 3x3x3 neighborhood of the maxima to solve for the sub-pixel offset of the extreme point. Multiply the integer coordinates of the feature map of the maxima by the sampling step size 2 to map back to the original image space, and superimpose the sub-pixel offset to generate floating-point coordinates in the original image space. Generate the initial spatial coordinate set of each modality and its corresponding response amplitude.

[0021] S14. For each modality, independently extract coordinate points with local feature response amplitudes greater than 0 from the initial spatial coordinate set as candidate keypoints. Calculate the difference between the upper and lower quartiles of the candidate keypoint response amplitude as the interquartile range. Add 1.5 times the interquartile range to the upper quartile to calculate the dynamic response threshold. Filter out coordinate points in the modality whose local feature response amplitudes are lower than the dynamic response threshold, retaining high-response coordinates to output the keypoint spatial coordinate set. Subsequently, based on the registered spatial topology, search for cross-modal candidate points within a preset 5 mm spatial neighborhood centered on the keypoints. Extract local mutual information between different modalities within the local spatial window. Find the optimal cross-modal corresponding point pairs within the spatial neighborhood through bidirectional nearest neighbor matching. Eliminate isolated points that cannot form cross-modal spatial correspondences, and output a keypoint spatial coordinate set with cross-modal consistency.

[0022] In this embodiment, S2 specifically includes: S21. Read the key point spatial coordinate set with cross-modal consistency output from the previous step. For each key point within the same modality, search for the 20 nearest neighbors with the nearest preset value in spatial distance and calculate the first Euclidean distance. For each key point across modalities, search for the 20 nearest neighbors with the nearest preset value in spatial distance and calculate the second Euclidean distance. Count all the values ​​of the first Euclidean distance and the second Euclidean distance and calculate the difference between their upper quartile and lower quartile as the interquartile range. Add 1.5 times the interquartile range to the upper quartile to calculate the dynamic same-modal spatial threshold and the dynamic cross-modal spatial threshold. Output the first Euclidean distance set, the second Euclidean distance set, the dynamic same-modal spatial threshold, and the dynamic cross-modal spatial threshold.

[0023] S22. Read the first Euclidean distance set, the second Euclidean distance set, the dynamic same-modal space threshold, and the dynamic cross-modal space threshold output from the previous steps. Filter out the key point pairs in the first Euclidean distance set whose distance values ​​are greater than the dynamic same-modal space threshold. Connect the remaining same-modal key point pairs to output same-modal space edges. Filter out the key point pairs in the second Euclidean distance set whose distance values ​​are greater than the dynamic cross-modal space threshold. Connect the remaining cross-modal key point pairs to output cross-modal space edges. Output the same-modal space edge set and the cross-modal space edge set.

[0024] S23. Read the key point spatial coordinate set with cross-modal consistency output from the previous step and the downsampled 3D mesh feature map of each modality output from the previous step. Divide the floating-point coordinates in the key point spatial coordinate set by the downsampling step size 2 to map them back to the floating-point coordinates of the feature map, and then round down to locate the integer index space of the feature map. Compare the integer index with the maximum effective index of each dimension of the feature map. If any dimension exceeds the boundary, truncate the index of that dimension to the maximum effective index. Extract a slice vector of length 64 along the channel dimension at the valid index position after the above verification. Input the element values ​​in the slice vector into the modified linear unit ReLU function. The calculation formula is to output the element value when the element value is greater than 0 and output 0 when the element value is less than or equal to 0. Output the slice vector after activation processing as the node initial vector and generate a set of node initial vectors.

[0025] S24. Read the node initial vector set, same-modal space edge set, and cross-modal space edge set output from the previous step, initialize an empty undirected graph data structure, add each vector in the node initial vector set as a node attribute to the undirected graph in sequence, add the endpoint index pairs in the same-modal space edge set as same-modal edge type attributes to the undirected graph, add the endpoint index pairs in the cross-modal space edge set as cross-modal edge type attributes to the undirected graph, complete the graph structure assembly operation, and output the cross-modal heterogeneous graph.

[0026] In this embodiment, S3 specifically includes: S31. Read the cross-modal heterogeneous graph and the set of initial node vectors output from the previous step. For the same-modal topological edge in the graph, calculate the first cosine similarity of the node vectors at both ends as the same-modal semantic similarity. For the cross-modal topological edge in the graph, multiply the node vectors at both ends element by element, and then pass them through a 128-neuron fully connected layer and a modified linear unit ReLU function, a 1-neuron output layer and a Sigmoid function in sequence to output the heterogeneous response probability as the cross-modal semantic similarity. Select the smaller value of the quartiles under the same-modal and cross-modal semantic similarities to output the dynamic semantic filtering threshold and the semantic similarity set.

[0027] S32. Read the semantic similarity set and dynamic semantic filtering threshold output by the previous step, filter out the same-modal and cross-modal topological edges with similarity below the threshold, retain the high semantic topological edges, and output the initial cleaned topological edge set.

[0028] S33. Read the initial cleaned topology edge set, cross-modal heterogeneous graph and node initial vector set output by the previous step, compare the full cross-modal edges with the cleaned edge set to extract the cross-modal topology gap region, reuse the heterogeneous response probability calculation process for the boundary node pairs of the gap region, connect the boundary node pairs whose response probabilities meet the dynamic semantic filtering threshold to generate reconstructed cross-modal edges, merge them into the initial cleaned topology edge set, and output the reconstructed topology edge set.

[0029] S34. Read the reconstructed topology edge set and node initial vector set output by the previous step, initialize the all-zero two-dimensional matrix with the total number of nodes, traverse the reconstructed topology edge set and set the intersection position and symmetric intersection position corresponding to the two end nodes to 1, and output the cross-modal heterogeneous graph adjacency matrix.

[0030] The cross-modal topology dynamic filtering and reconstruction process proposed in this step is similar to the traditional graph structure topology construction process in that both are based on feature space mapping and adjacency relationship construction theory. That is, by projecting the features of multimodal nodes to a high-dimensional latent space for similarity measurement calculation, using a similarity threshold judgment mechanism to filter effective topological edges, and using the mathematical expression of graph adjacency matrix to map the connection relationship between nodes.

[0031] The difference lies in that this invention breaks through the limitations of traditional static graph construction, which relies on fixed thresholds or spatial nearest neighbors, leading to semantic redundancy and topological breaks. It adds a statistical distribution adaptive threshold calculation step to filter out low-semantic topological edges, and replaces the traditional one-way retention operation with a cross-modal semantic reconstruction mechanism. It extracts the boundary nodes of the topological void region and calculates the heterogeneous reconstruction response probability. For void boundaries that meet the dynamic semantic filtering threshold, it performs cross-modal bridging to generate reconstructed cross-modal edges. Finally, it merges the initial cleaned edge set and the reconstructed edge set to output the adjacency matrix, rather than simply hard-truncating and retaining the original spatial edges.

[0032] The beneficial effects of the improvements are that, by statistically driving thresholds and reconstructing missing semantics, this invention hard-embeds global distribution feature constraints and cross-modal topological integrity into the forward construction of the graph structure, breaking the limitations of traditional methods that are prone to generating redundant connections and topological islands under complex anatomical structures, and achieving a precise conversion from static spatial nearest neighbors to dynamic semantic connectivity. This design significantly enhances the ability to mine potential associations between heterogeneous modalities, can accurately purify invalid topologies and complete semantic links in the global probability space, and, combined with a dynamic threshold adaptive mechanism, effectively improves the purity of cross-modal feature interactions and the high-order topological integrity of heterogeneous graph structures.

[0033] In this embodiment, S4 specifically includes: S41. Read the cross-modal heterogeneous graph adjacency matrix and node initial vector set output from the previous step. Based on the elements with a value of 1 in the cross-modal heterogeneous graph adjacency matrix, locate the same-modal topological connections and cross-modal topological connections. For same-modal topological connections, sum the node initial vectors of adjacent same-modal nodes and calculate the average. Input a fully connected layer with a weight matrix of 64 rows and 64 columns and a bias vector length of 64. Apply the Modified Linear Unit (ReLU) function to the output of the fully connected layer. The calculation formula is to output the element value when the element value is greater than 0 and output 0 when the element value is less than or equal to 0. Output the same-modal interaction node vector. For cross-modal topological connections, sum the node initial vectors of adjacent cross-modal nodes and calculate the average. Input a fully connected layer with a weight matrix of 64 rows and 64 columns and a bias vector length of 64. Apply the Sigmoid function to the output of the fully connected layer. The calculation formula is 1 divided by 1 plus the inverse power of the fully connected layer output result. Output the cross-modal interaction node vector.

[0034] S42. Read the set of same-modal interaction node vectors, cross-modal interaction node vectors, and node initial vectors output from the previous step. For the same-modal interaction node vector and its corresponding node initial vector, calculate the sum of the products of corresponding elements of the two vectors divided by the product of the square root of the sum of the squares of the elements of the two vectors as the same-modal semantic preservation degree. For the cross-modal interaction node vector and its corresponding node initial vector, calculate the sum of the products of corresponding elements of the two vectors divided by the product of the square root of the sum of the squares of the elements of the two vectors as the cross-modal semantic transfer degree. Multiply the same-modal semantic preservation degree and the cross-modal semantic transfer degree to output the semantic association degree of the node pair at both ends of the cross-modal edge.

[0035] S43. Read the semantic correlation degree output from the previous step, perform grouping and statistical operations on the semantic correlation degree according to the modality affiliation of the cross-modal edge, calculate the median of the semantic correlation degree within each modal pair as the dynamic intra-group correlation baseline, subtract the dynamic intra-group correlation baseline from the semantic correlation degree of each cross-modal edge in the same group, add the value 1.0, perform normalization offset operation, and output the cross-modal edge weight.

[0036] S44. Read the cross-modal edge weights output from the previous step. Perform global mean pooling on the cross-modal edge weights according to their modality affiliation. Sum the cross-modal edge weights within the same modality affiliation group and divide by the total number of cross-modal edges in that group. Use the mean of the modality pair associations obtained from pooling as the evolutionary source features. Perform matrix mapping on the evolutionary source features. Input the evolutionary source features into a fully connected layer with 1 row and 2 columns in the weight matrix and a bias vector length of 2. Apply the Softmax function to the output of the fully connected layer. The calculation formula is the current element value of the natural constant e divided by the sum of all element values ​​of the natural constant e. Output the modality contribution weight matrix.

[0037] The modal contribution weight matrix evolution process proposed in this step is similar to the traditional multimodal feature fusion weight allocation process in that both are based on the theory of cross-modal feature interaction and importance measurement. That is, by projecting multimodal features to a high-dimensional latent space to perform message passing and dependency calculation between nodes, the correlation law of feature response is used to capture the interaction contribution between modalities, and both use matrix mapping operations to transform the interaction features into modal dimension weight coefficients.

[0038] The difference lies in that this invention breaks through the limitations of traditional static weight allocation or global attention ignoring cross-modal heterogeneous semantic differences. It adds a decoupled calculation step between intramodal semantic preservation and cross-modal semantic transfer, accurately quantifies the semantic changes before and after the interaction, and replaces the traditional fixed normalization layer with a grouped statistical dynamic baseline mechanism. It performs intra-group statistics and normalization offset operations on the micro-semantic correlation degree according to modality to obtain cross-modal edge weights. Finally, it outputs the macro-modal contribution weight matrix through global pooling and matrix mapping evolution of the edge weights, rather than a single end-to-end attention scoring prediction.

[0039] The beneficial effects of the improvements are that, through semantic change decoupling and intra-group baseline offset normalization, the asymmetric interaction constraints of heterogeneous modalities are forcibly embedded into the weight evolution process. This breaks the limitation of traditional methods in complex medical imaging scenarios where they are easily affected by intra-modal feature interference, leading to distortion in contribution assessment. It achieves a precise conversion from black-box attention scoring to macro-weight evolution driven by micro-semantic changes. This design significantly enhances the ability to characterize the sensitivity of cross-modal heterogeneous feature interactions, accurately removes intra-modal redundant interference in the dynamic intra-group statistical space, and, combined with evolution source feature mapping, effectively improves the adaptability of modal contribution assessment and the robustness of multi-modal fusion decision-making.

[0040] In this embodiment, the improved MPNN model includes a source message construction layer, a topology path propagation layer, a semantic aggregation and update layer, a shape memory alloy phase change rectification layer, and a feature matrix output layer: The source message construction layer is used to read the cross-modal heterogeneous graph adjacency matrix, node initial vector set, and modal contribution weight matrix output by the preceding steps. Based on the elements with a value of 1 in the cross-modal heterogeneous graph adjacency matrix, it locates the source node of the sending end, extracts the node initial vector corresponding to the source node in the node initial vector set, reads the weight coefficient corresponding to the modality to which the source node belongs in the modal contribution weight matrix, multiplies each element of the node initial vector by the weight coefficient to perform source weighting, and outputs the weighted source message vector.

[0041] The topology path transmission layer is used to read the cross-modal heterogeneous graph adjacency matrix and weighted source message vectors output by the previous steps. Based on the topological connection relationship defined by the cross-modal heterogeneous graph adjacency matrix, the weighted source message vectors are routed and transmitted along the corresponding cross-modal and same-modal spatial edges and distributed to each target node that is the receiving end. All weighted source message vectors received by the same target node are collected and the target node receives the message set.

[0042] The semantic aggregation update layer reads the target node's received message set and node initial vector set output by the previous steps. It performs an average operation on the corresponding elements of all weighted source message vectors in the target node's received message set to obtain an aggregation result vector. It extracts the target node's own node initial vector from the node initial vector set and concatenates the aggregation result vector with the node's initial vector to obtain a concatenated vector. The concatenated vector is input into a fully connected layer with a weight matrix of 128 rows and 64 columns and a bias vector length of 64. The ReLU function is applied to the output of the fully connected layer. The calculation formula is to output the element value when the element value is greater than 0 and output 0 when the element value is less than or equal to 0. The result after applying the ReLU function is normalized by dividing the value of each element in the vector by the square root of the sum of the squares of all elements in the vector. The distorted higher-order hidden state vector of the target node is then output.

[0043] The shape memory alloy phase change rectifier layer is used to introduce a material science-based mechanism for recovering the phase change characteristics of shape memory alloys. The specific execution process includes: Read the set of initial node vectors and the distorted higher-order hidden state vectors output by the previous step, set the initial node vectors in the set of initial node vectors as the original semantic anchors, calculate the Euclidean distance between the distorted higher-order hidden state vectors and the original semantic anchors as the feature offset, and define the feature offset as the semantic temperature parameter. The calculation formula is the square root of the sum of squares of the differences between corresponding elements of two vectors; the mean and standard deviation of the distribution of semantic temperature parameters of all nodes are statistically analyzed along the node dimension, and the mean is added to the value of 1.0 and multiplied by the standard deviation of the distribution for linear combination to calculate the adaptive dynamic phase transition benchmark; The difference between the semantic temperature parameter and the dynamic phase transition reference is calculated. The difference is input into the Sigmoid function. The calculation formula is 1 divided by 1 plus the opposite power of the difference of the natural constant e. A continuously differentiable phase transition coefficient is generated and this coefficient is used as the shrinkage coefficient. The distortion higher-order hidden state vector is elastically shrunk back to the original semantic anchor point using a shrinkage coefficient. The calculation formula is the shrinkage coefficient multiplied by the distortion higher-order hidden state vector plus the difference of 1 minus the shrinkage coefficient multiplied by the original semantic anchor point, and the output phase change rectification feature vector is generated.

[0044] The feature matrix output layer is used to read the phase change rectification feature vectors output by the previous step. The phase change rectification feature vectors of all target nodes are stacked and concatenated in the order of nodes. The fully connected layer with 64 rows and 64 columns of input weight matrix and 64 bias vector length performs the mapping operation. The modified linear unit ReLU function is applied to the output of the fully connected layer. The calculation formula is to output the element value when the element value is greater than 0 and output 0 when the element value is less than or equal to 0. The high-order graph node feature matrix is ​​output.

[0045] The improved MPNN model feature aggregation and rectification process proposed in this step is similar to the traditional MPNN model message passing process in that it is based on graph message passing and neighborhood aggregation theory. That is, by projecting the features of multimodal nodes onto the graph topology space to construct weighted messages and pass paths, the aggregation operation of neighborhood nodes and nonlinear transformation are used to capture the dynamic evolution law of high-order topology, and a feature mapping layer is used to output the hidden state of nodes as a graph node feature matrix.

[0046] The difference lies in that this invention breaks through the limitations of traditional MPNN models, where multi-layer aggregation easily leads to feature oversmoothing and semantic shift. It adds a memory alloy phase transition feature recovery mechanism based on materials science to replace the traditional residual connection or direct normalization operation. The initial vector of the node is set as the original semantic anchor point. The feature shift of the distorted higher-order hidden state vector after aggregation is calculated as the semantic temperature parameter. Based on its distribution mean and dispersion, the dynamic phase transition benchmark is adaptively calculated to generate a continuously differentiable shrinkage coefficient. Finally, the distorted feature is driven to perform elastic shrinkage compensation to the original semantic anchor point to output the phase transition rectified feature vector, rather than a simple layer-by-layer feature accumulation or linear transformation.

[0047] The beneficial effects of the improvements are that, by introducing a memory alloy phase transition feature recovery mechanism based on materials science, this invention forcibly embeds physical biomimetic phase transition constraints into the forward propagation of the improved MPNN model. This breaks the limitation of traditional methods that easily lose the original semantic boundaries and cause feature drift under deep heterogeneous aggregation, and achieves a precise conversion from unconstrained black-box aggregation to physical mechanism-driven strong constraint to prevent offset. This design significantly enhances the defense capability against cross-modal heterogeneous feature distortion, can accurately perform semantic temperature-driven elastic shrinkage in high-dimensional latent space, and combined with dynamic phase transition benchmark adaptive adjustment, effectively improves the fidelity of graph node feature representation and the absolute reliability of multimodal diagnostic decisions.

[0048] In this embodiment, S6 specifically includes: S61. Read the high-order graph node feature matrix and the cross-modal heterogeneous graph adjacency matrix output from the previous step. Locate the topological neighbors of the node based on the elements with a value of 1 in the cross-modal heterogeneous graph adjacency matrix. Add the corresponding feature vectors of the neighbor nodes in the high-order graph node feature matrix along the topological connection and calculate the mean to output the local mean feature vector. Calculate the Euclidean distance between the node's own feature vector and the local mean feature vector in the high-order graph node feature matrix as the feature deviation degree. The calculation formula is the square root of the sum of the squares of the differences between the corresponding elements of the two vectors. Count the total number of elements with a value of 1 in each row of the cross-modal heterogeneous graph adjacency matrix as the global centrality. Multiply the feature deviation degree by the global centrality and perform a fusion operation to output the initial key index of the one-dimensional high-order graph node.

[0049] S62. Read the initial critical index of the one-dimensional higher-order graph nodes output by the previous step, calculate the median of the initial critical index of all nodes as the dynamic critical baseline, subtract the dynamic critical baseline from the initial critical index of each node and add 1.0 to perform a normalization offset operation, filter out negative offset nodes with values ​​lower than 0 after the normalization offset operation and set their values ​​to 0, keep the nodes with values ​​greater than or equal to 0, and output the criticality weight of the one-dimensional higher-order graph nodes.

[0050] S63. Read the one-dimensional high-order graph node criticality weights and high-order graph node feature matrix output from the previous step, extract the feature vector of each node in the high-order graph node feature matrix, multiply the feature vector of each node by the weight value of the corresponding node in the one-dimensional high-order graph node criticality weights, perform the summation operation of the corresponding position elements on all weighted node feature vectors, and output the multimodal fusion graph-level vector.

[0051] In this embodiment, S7 specifically includes: S71. Read the multimodal fusion graph-level vector output from the previous step, calculate the square root of the sum of squares of all elements in the multimodal fusion graph-level vector as the feature amplitude, set the feature amplitude as the dynamic scale baseline, perform adaptive scaling normalization operation on the multimodal fusion graph-level vector based on the dynamic scale baseline, the calculation formula is the value of each element of the multimodal fusion graph-level vector divided by the dynamic scale baseline, and output the scale-normalized graph-level vector.

[0052] S72. Read the scale-normalized graph vector output from the previous step, perform a parameterized linear mapping operation on the scale-normalized graph vector, input the scale-normalized graph vector into a fully connected layer with a weight matrix of 64 rows and 2 columns and a bias vector length of 2, and output the initial class response logits; perform a normalized exponential mapping on the initial class response logits, apply the Softmax function, and calculate the value of the current element of the natural constant e by dividing the sum of the values ​​of all elements of the natural constant e, and output the class response probability.

[0053] S73. Read the category response probabilities output by the previous steps, perform extreme value search to compare the numerical values ​​of all category response probabilities, extract the element with the largest value as the maximum response probability, extract the category index corresponding to the maximum response probability as the target category label, combine the target category label with the maximum response probability, and output the medical image data analysis results.

[0054] Example 1: To verify the feasibility of this invention in multimodal medical image data analysis, the method of this invention was applied to the intelligent auxiliary diagnosis and treatment system of a tertiary hospital in a certain city (hereinafter referred to as "Hospital M"). In traditional medical image analysis systems, manual interpretation based on single-modal images or simple feature stitching and fusion algorithms are usually used. These methods not only struggle to accurately capture subtle correlations between lesions across modalities in complex anatomical structures, but also fail to accurately measure the contribution weight of different modalities to the final diagnostic decision, easily leading to misjudgment or underreporting of lesion boundaries and pathological properties. To solve the above problems, Hospital M decided to adopt the multimodal medical image data analysis method based on machine learning proposed in this invention.

[0055] During implementation, Hospital M first used PET and MRI dual-modal medical imaging equipment to acquire metabolic function and anatomical structure images of patients. After spatial affine transformation to align the multimodal spatial topology, convolutional coding and downsampling operations were sequentially performed along the feature channels to output grid feature maps for each modality. Simultaneously, local feature response amplitudes were calculated along the spatial dimension, the coordinates of peak response points were extracted, and low-response coordinates were filtered out based on a dynamic response threshold, outputting a set of key point spatial coordinates. Radiology experts at Hospital M performed precise lesion type labeling and spatial coordinate registration on the acquired multi-source data, serving as the benchmark for model training and diagnostic decisions.

[0056] Hospital M calculates the Euclidean distance between keypoint pairs and cross-modal keypoint pairs using a keypoint spatial metric and semantic reconstruction mechanism. It constructs same-modal and cross-modal spatial edges based on dynamic spatial thresholds and anchors initial node vectors at corresponding positions in the mesh feature map, assembling and outputting a cross-modal heterogeneous graph. Next, it calculates the same-modal and cross-modal semantic similarity of nodes at both ends of topological edges based on the initial node vectors. Low-semantic topological edges are removed based on a dynamic semantic filtering threshold, and the heterogeneous reconstruction response probability between boundary nodes in missing regions is calculated to bridge topological gaps, outputting the cross-modal heterogeneous graph adjacency matrix. Subsequently, it connects the initial vectors of interacting nodes along the cross-modal topology, calculates semantic association, and performs grouping statistics and normalized offsets based on modality affiliation, mapping and evolving to generate a modality contribution weight matrix.

[0057] In the core feature aggregation and verification stage, this invention, based on an improved MPNN model, combines a cross-modal heterogeneous graph adjacency matrix and a modal contribution weight matrix to perform weighted heterogeneous aggregation of source features, generating a distorted high-order hidden state vector. A memory alloy phase transition feature recovery mechanism based on materials science is introduced, setting the initial node vector as the original semantic anchor point. The feature offset of the distorted feature relative to the original anchor point is calculated as a semantic temperature parameter. A dynamic phase transition benchmark is adaptively calculated based on the distribution mean and dispersion to generate a shrinkage coefficient, driving the distorted feature to perform elastic shrinkage compensation towards the original semantic anchor point, outputting a high-order graph node feature matrix. Subsequently, the feature deviation and global centrality are calculated using the cross-modal heterogeneous graph adjacency matrix to generate an initial key index. Based on the dynamic key baseline, normalized offset and negative filtering are performed to obtain key weights. Weighted sum pooling is performed on the feature matrix to output a multimodal fused graph-level vector. Finally, adaptive scaling and normalization are performed on the graph-level vector based on the dynamic scale baseline. Class response probabilities are generated through parameterized linear mapping and normalized exponential mapping, and the medical image data analysis results are output based on probability decisions.

[0058] During implementation, the technical team at Hospital M discovered that, compared to traditional manual interpretation and conventional single-modal or simple stitching fusion methods, the method of this invention significantly improves the accuracy and robustness of multimodal medical image data analysis. Traditional methods cannot quantify the semantic topological associations of cross-modal features and lack constraints on semantic distortions caused by deep feature interactions. In contrast, the method of this invention effectively achieves accurate fusion of multimodal lesion features and semantic offset prevention verification through dynamic topological reconstruction, modal contribution evolution weighting, and shape memory alloy phase transition shrinkage compensation.

[0059] To further verify the actual performance of the method of the present invention, Hospital M conducted a detailed comparative test between the method of the present invention and the traditional method. The specific performance data is shown in Table 1: Table 1. Performance Comparison of Multimodal Medical Imaging Data Analysis Methods in Hospitals

[0060] As shown in Table 1, the performance of the multimodal medical image data analysis system was comprehensively improved after applying the method of this invention. The accuracy of lesion identification increased from 83.5% with traditional methods to 97.2%, and the average cross-modal registration error decreased from 4.5 mm to 0.6 mm, significantly improving the accuracy of multimodal feature interaction and providing a reliable basis for subsequent diagnostic decisions. The misclassification rate of benign or malignant lesions decreased from 15.2% to 2.1%, effectively avoiding erroneous clinical interventions. The response time for auxiliary diagnosis was significantly shortened from 45 seconds to 6 seconds, significantly enhancing the system's timeliness. In addition, the early lesion detection rate increased from 75.0% to 95.8%, and the feature semantic offset decreased from 28.5% to 3.5%, ensuring a high degree of consistency between the fused features and the original semantic anchor points. The time for radiologists to review images decreased from 25 minutes / case to 8 minutes / case, significantly reducing the workload of physicians. The satisfaction rate with clinical treatment plans also significantly improved, from 82.0% to 96.5%.

[0061] Through the method of this invention, hospital M has successfully achieved precise analysis of multimodal medical imaging data and deep integration of lesion features, effectively reducing the risk of missed diagnosis and misdiagnosis, ensuring the scientific nature and safety of clinical diagnosis and treatment, significantly improving the intelligence and precision of medical image-assisted diagnosis, significantly reducing the workload of radiologists, enhancing the stability and generalization ability of the auxiliary diagnosis and treatment system, and providing strong technical support for the construction of smart healthcare.

[0062] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for analyzing multi-modal medical image data based on machine learning, characterized by, Includes the following steps: S1. Acquire multimodal medical image data, perform spatial registration to align the multimodal spatial topology, sequentially encode and extract the grid feature map of each modality along the feature channel, and measure the spatial saliency response in parallel to extract the output key point spatial coordinate set; S2. Based on the spatial metric of the key point spatial coordinate set, construct the same-modal and cross-modal spatial edges, anchor the node initial vector at the corresponding position of the key point coordinate in the mesh feature map, and construct the cross-modal heterogeneous graph by combining the corresponding spatial edges. S3. Based on the cross-modal heterogeneous graph, calculate the semantic similarity between the nodes at both ends of the topological edge according to the node initial vector, filter out the low semantic topological edges based on the statistical distribution features of semantic similarity, perform cross-modal semantic reconstruction to bridge the topological gaps, and output the cross-modal heterogeneous graph adjacency matrix. S4. Calculate the semantic correlation degree of the node pairs at both ends of the cross-modal heterogeneous graph connected by the initial vector of the topological connection interaction nodes defined by the adjacency matrix of the cross-modal heterogeneous graph, and map the correlation degree to generate the modal contribution weight matrix. S5. Based on the improved MPNN model, the source features are weighted heterogeneous aggregation by combining the cross-modal heterogeneous graph adjacency matrix and modal contribution weight matrix to generate distorted high-order hidden state vectors; a memory alloy phase transformation feature recovery mechanism based on materials science is introduced, and the distorted features are dynamically shrunk back to the initial semantic anchor point based on feature offset adaptive drive, and the high-order graph node feature matrix is ​​output. S6. Measure the criticality of the feature matrix of the higher-order graph node, compress and map the feature matrix of the higher-order graph node according to the criticality, and output the multimodal fused graph-level vector. S7. Perform learnable discriminant mapping on the multimodal fusion graph-level vectors to obtain the category response probability, and output the medical image data analysis results based on probability decision.

2. The multimodal medical image data analysis method based on machine learning according to claim 1, characterized in that, S1 specifically includes: S11. Perform a spatial affine transformation on the acquired multimodal medical image data to align the multimodal spatial topology and output the registered image data; S12. Based on the registered image data, perform convolutional coding and downsampling operations sequentially along the feature channels to output the grid feature maps of each modality; S13. Based on the mesh feature maps of each modality, calculate the local feature response amplitude along the spatial dimension, and extract the coordinates of the peak points of the response amplitude to generate an initial spatial coordinate set; S14. Statistically analyze the global response distribution characteristics of the initial spatial coordinate set, calculate the dynamic response threshold, filter out coordinate points below the dynamic response threshold, and output the key point spatial coordinate set.

3. The multimodal medical image data analysis method based on machine learning according to claim 1, characterized in that, S2 specifically includes: S21. Based on the key point spatial coordinate set, calculate the first Euclidean distance of key point pairs within the same modality and the second Euclidean distance of key point pairs across modalities, statistically analyze the global spatial distribution characteristics of the first Euclidean distance and the second Euclidean distance, and calculate the dynamic same-modal spatial threshold and the dynamic cross-modal spatial threshold. S22. Filter out keypoint pairs corresponding to the first Euclidean distance that is greater than the dynamic same-modal space threshold, and connect the retained same-modal keypoint pairs to output same-modal space edges; filter out keypoint pairs corresponding to the second Euclidean distance that is greater than the dynamic cross-modal space threshold, and connect the retained cross-modal keypoint pairs to output cross-modal space edges. S23. Based on the key point spatial coordinate set, perform feature slicing extraction operation at the corresponding spatial position of the grid feature map and output the node initial vector; S24. Based on the node initial vectors and edges in the same modal space and across modal spaces, perform graph structure assembly operations to output a cross-modal heterogeneous graph.

4. The multimodal medical image data analysis method based on machine learning according to claim 1, characterized in that, S3 specifically includes: S31. Based on the cross-modal heterogeneous graph and node initial vectors, calculate the first cosine similarity of the initial vectors of the nodes at both ends of the same-modal topological edge as the same-modal semantic similarity, calculate the heterogeneous response probability of the initial vectors of the nodes at both ends of the cross-modal topological edge as the cross-modal semantic similarity, statistically analyze the global distribution characteristics of same-modal semantic similarity and cross-modal semantic similarity, and calculate the dynamic semantic filtering threshold. S32. Remove low semantic homomodal topological edges and low semantic crossmodal topological edges that are below the dynamic semantic filtering threshold, and output the initial cleaned topological edge set; S33. Based on the initial cleaned topology edge set, extract cross-modal topology void regions, calculate the heterogeneous reconstruction response probability between the initial vectors of the boundary nodes of the void regions, connect the boundary node pairs whose heterogeneous reconstruction response probabilities satisfy the dynamic semantic filtering threshold to generate reconstructed cross-modal edges, and merge the initial cleaned topology edge set and the reconstructed cross-modal edges to output the reconstructed topology edge set. S34. Based on the reconstructed topological edge set, perform graph adjacency mapping and output the cross-modal heterogeneous graph adjacency matrix.

5. The multimodal medical image data analysis method based on machine learning according to claim 1, characterized in that, S4 specifically includes: S41. Based on the adjacency matrix and node initial vectors of the cross-modal heterogeneous graph, perform message passing interaction operations along the cross-modal topology connections defined by the adjacency matrix of the cross-modal heterogeneous graph, and output the same-modal interaction node vectors and cross-modal interaction node vectors. S42. Calculate the same-modal semantic preservation degree based on the same-modal interaction node vector and node initial vector, calculate the cross-modal semantic transfer degree based on the cross-modal interaction node vector and node initial vector, and output the semantic association degree of the node pairs at both ends of the cross-modal edge by fusing the same-modal semantic preservation degree and the cross-modal semantic transfer degree. S43. Perform grouping statistical operation on the semantic relevance based on the modality attribution of cross-modal edges, calculate the dynamic intra-group relevance baseline of each group of modal pairs, perform normalization offset operation on the semantic relevance based on the dynamic intra-group relevance baseline, and output the cross-modal edge weights. S44. Perform global mean pooling on the cross-modal edge weights based on the modal attribution of the cross-modal edges. Use the mean of the modal pairs obtained by pooling as the evolution source features. Perform matrix mapping operation on the evolution source features and output the modal contribution weight matrix.

6. The multimodal medical image data analysis method based on machine learning according to claim 1, characterized in that, The improved MPNN model includes a source message construction layer, a topology path propagation layer, a semantic aggregation and update layer, a shape memory alloy phase change rectification layer, and a feature matrix output layer. The source-end message construction layer is used to define the message passing topology path with the cross-modal heterogeneous graph adjacency matrix, extract the initial vector of the source node as the sender, and perform source-end weighting on the initial vector of the node according to the corresponding weight coefficient in the modal contribution weight matrix, and output the weighted source-end message vector. The topology path transmission layer is used to route and transmit the weighted source message vector along the corresponding cross-modal and same-modal space edges according to the topological connection relationship defined by the cross-modal heterogeneous graph adjacency matrix, distribute it to each target node as the receiving end, and output the target node received message set. The semantic aggregation update layer is used to perform aggregation operations on the weighted source message vectors in the message set received by each target node, concatenate the aggregation result with the node initial vector of the target node itself, and output the distorted higher-order hidden state vector of the target node through nonlinear transformation mapping and normalization processing. The shape memory alloy phase change rectifier layer is used to introduce a shape memory alloy phase change characteristic recovery mechanism based on materials science. The specific execution process includes: The node initial vector is set as the original semantic anchor point, and the feature offset of the distorted higher-order hidden state vector relative to the original semantic anchor point is calculated as the semantic temperature parameter. The distribution characteristics of semantic temperature parameters are statistically analyzed along the node dimension. The distribution mean and distribution dispersion are extracted based on the distribution characteristics. The distribution mean and distribution dispersion are linearly combined to adaptively calculate the dynamic phase transition benchmark. The difference between the semantic temperature parameter and the dynamic phase transition reference is calculated, and a nonlinear mapping is performed to generate a continuously differentiable phase transition coefficient, which is then used as a shrinkage coefficient. The distortion of the higher-order hidden state vector is elastically shrunk back to the original semantic anchor point using the shrinkage coefficient, and the phase change rectification feature vector is output. The feature matrix output layer is used to stack and map the phase change rectification feature vectors of all target nodes in the order of nodes, and output a high-order graph node feature matrix.

7. The multimodal medical image data analysis method based on machine learning according to claim 1, characterized in that, S6 specifically includes: S61. Based on the feature matrix of a high-order graph node and the adjacency matrix of a cross-modal heterogeneous graph, aggregate the features of neighboring nodes along the topological connection defined by the adjacency matrix of the cross-modal heterogeneous graph to output local mean features, calculate the feature deviation between the node's own features and the local mean features, calculate the global centrality by combining the adjacency matrix of the cross-modal heterogeneous graph, and output the initial key index of a one-dimensional high-order graph node by fusing the feature deviation and the global centrality. S62. Statistically analyze the global distribution characteristics of the initial critical index of the nodes in the one-dimensional high-order graph, calculate the dynamic critical baseline, perform normalization offset operation on the initial critical index of the nodes in the one-dimensional high-order graph based on the dynamic critical baseline, filter out negative offset nodes with values ​​below zero, and output the criticality weight of the nodes in the one-dimensional high-order graph. S63. Perform weighted summation pooling operation on the feature matrix of high-order graph nodes based on the keyness weight of one-dimensional high-order graph nodes, and output a multimodal fused graph-level vector.

8. The multimodal medical image data analysis method based on machine learning according to claim 1, characterized in that, Specifically, S7 includes: S71. Based on the multimodal fusion graph-level vector, statistically analyze the feature amplitude distribution of the multimodal fusion graph-level vector and calculate the dynamic scale baseline. Based on the dynamic scale baseline, perform adaptive scaling and normalization operation on the multimodal fusion graph-level vector and output the scale-normalized graph-level vector. S72. Perform parameterized linear mapping operation on scale-normalized graph-level vectors to output the initial class response logits, and perform normalized exponential mapping on the initial class response logits to output the class response probability. S73. Perform extreme value retrieval operation based on category response probability to output the maximum response probability and its corresponding target category label, and combine the target category label and the maximum response probability to generate medical image data analysis results.

Images