Medical aid decision-making method and system based on multi-modal deep learning

Abstract

The invention discloses a medical aid decision-making method and system based on multi-modal deep learning, and relates to the technical field of auxiliary diagnosis, and the method comprises the steps: obtaining medical image data and non-image data of a target object; based on the medical image data and the non-image data, identifying and segmenting a target focus area, and respectively extracting a depth image feature and a depth non-image feature of the target focus area; the depth image features and the depth non-image features are fused, and comprehensive feature representation of the focus is generated; and segmenting at least one key anatomical structure around the target lesion area by taking the target lesion area as a center. According to the method, the focus-key anatomical structure heterogeneous graph is constructed, and the graph neural network is used for reasoning, so that key pathological interaction relationships such as traction of a nodule to a blood vessel and infringement of the nodule to a bronchial tube can be automatically and quantitatively modeled. Accurate modeling and deep reasoning of the focus-microenvironment complex interaction relation are achieved, and the biological basis of diagnosis and the credibility of the model are improved.

Description

Technical Field

[0001] This invention relates to the field of auxiliary diagnostic technology, and specifically to a medical auxiliary decision-making method and system based on multimodal deep learning. Background Technology

[0002] With the rapid advancement of medical imaging technology and its deep integration with artificial intelligence, medical diagnosis, especially the early screening and precise diagnosis of oncological diseases, is continuously moving towards multimodality, intelligence, and quantification. This trend has brought significant advantages in improving diagnostic efficiency, reducing physician subjectivity, and achieving personalized medicine. However, along with this progress, the challenges faced in clinical diagnostic decision-making are becoming increasingly complex and severe.

[0003] Taking the differentiation between benign and malignant pulmonary nodules as an example, the diagnosis heavily relies on the physician's interpretation of the nodule's morphology, density, and other imaging characteristics on chest CT images. It also requires comprehensive evaluation based on multiple sources of information, including the patient's clinical history and pathology reports. Early and accurate differentiation is crucial for the early diagnosis and treatment of lung cancer and for improving patient prognosis. Misdiagnosis or missed diagnosis can lead to unnecessary surgical trauma or delays in optimal treatment, resulting in serious health consequences.

[0004] In intelligent medical decision support systems, the fusion analysis and deep reasoning of multi-source information occupy a central position. Currently, most existing deep learning-based diagnostic methods focus on automatically extracting visual features of lesions from medical images, or further performing simple splicing or early fusion with structured clinical data. Undeniably, these methods can improve the objectivity and consistency of diagnosis to some extent, providing valuable references for doctors. However, they have a significant limitation: they cannot model and reason about the complex spatial and structural interactions between lesions and their surrounding anatomical microenvironment. In clinical practice, secondary signs such as the traction of nodules on surrounding blood vessels and invasion of the bronchial wall are key criteria for radiologists to determine benign or malignant conditions. Existing methods view lesion features in isolation or only perform superficial data merging, neglecting to model this crucial diagnostic dimension. This makes it difficult for the model to fully simulate expert diagnostic thinking, limiting its generalization ability in complex and atypical cases, and ultimately potentially affecting the reliability of decisions and clinical adoption. Summary of the Invention

[0005] The purpose of this invention is to provide a medical decision support method and system based on multimodal deep learning to address the shortcomings of the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a medical auxiliary decision-making method based on multimodal deep learning, comprising: Acquire medical imaging and non-imaging data of the target object; Based on the medical imaging data and non-imaging data, the target lesion area is identified and segmented, and the depth imaging features and depth non-imaging features of the target lesion area are extracted respectively. By fusing the depth image features and the depth non-image features, a comprehensive feature representation of the lesion is generated; Centered on the target lesion area, at least one key anatomical structure around it is segmented; A heterogeneous graph representing the spatial and structural relationship between the target lesion region and the surrounding key anatomical structures is constructed, and a graph neural network is used to reason about the heterogeneous graph to obtain updated features; The integrated feature representation is fused with the updated feature, and auxiliary decision-making information for the target lesion is generated based on the fused feature.

[0007] In a preferred embodiment, the step of identifying and segmenting the target lesion region based on the medical imaging data and non-imaging data, and extracting the depth imaging features and depth non-imaging features of the target lesion region respectively, includes: The medical imaging data is chest CT images, and the target lesion area is a lung nodule; the non-imaging data includes at least one of electronic medical record text data, gene sequencing data, or pathology report text data. The depth image features refer to the features extracted from the image data of the segmented lung nodule region; The deep non-image features refer to the semantic or numerical features related to the lung nodules extracted from the non-image data.

[0008] In a preferred embodiment, the step of fusing the depth image features and the depth non-image features to generate a comprehensive feature representation of the lesion includes: The fusion of depth image features and depth non-image features is achieved through a cross-modal attention fusion mechanism.

[0009] In a preferred embodiment, the step of segmenting at least one key anatomical structure surrounding the target lesion region as the center includes: The key anatomical structures include at least one of blood vessels, bronchi, and pleura.

[0010] In a preferred embodiment, the step of constructing a heteromorphic map representing the spatial and structural relationships between the target lesion region and the surrounding key anatomical structures includes: The target lesion area and surrounding key anatomical structures are abstracted as nodes in the diagram, and connecting edges are constructed based on the spatial and structural relationships between the nodes. The connecting edges include edges that reflect spatial proximity, edges that reflect the continuity of the anatomical structure itself, and higher-order causal edges that reflect the deformation or invasion of the anatomical structure caused by the lesion. The higher-order causal edges include deformation / tension edges defined based on deformation field relationships, and invasion / enclosure edges defined based on boundary ambiguity or enclosing geometry relationships.

[0011] In a preferred embodiment, the step of using a graph neural network to infer the heterogeneous graph and obtain updated features includes: A heterogeneous graph attention network is adopted, which uses an attention mechanism to assign different weights to different types of connection edges and neighbor nodes in order to aggregate information.

[0012] In a preferred embodiment, the step of using a graph neural network to infer the heterogeneous graph and obtain updated features includes: A physics-inspired model is used to model and infer the interaction relationship between the nodes. The target lesion region is modeled as a gravitational core, the key anatomical structure is modeled as an orbital entity affected by it, and the interaction process under the action of the gravitational field is simulated using dynamic equations.

[0013] In a preferred embodiment, the step of fusing the integrated feature representation with the updated features and generating auxiliary decision-making information for the target lesion based on the fused features includes: The integrated feature representation is fused with the updated feature and input into the classifier to obtain the benign or malignant classification probability and risk level suggestion of the target lesion; Generate interpretable output that reflects the basis of the auxiliary decision-making information; The interpretability output includes visualizing connections in the heterogeneous graph where the attention weights assigned by the graph neural network are higher than a predetermined threshold, and outputting an index that quantitatively assesses the intensity of interaction between the lesion and the surrounding environment.

[0014] In a preferred embodiment, the method further includes the step of: The medical imaging data were subjected to whole-lung field effect analysis to extract the spatial distribution characteristics of lesions within the bilateral lungs and the radiomics characteristics of the background lung parenchyma. The extracted spatial distribution features of the lesions and the radiomics features of the background lung parenchyma are fused with the comprehensive feature representation and the updated features to generate the auxiliary decision information.

[0015] This invention also provides a medical decision support system based on multimodal deep learning, comprising: The data acquisition and processing module is used to acquire and process medical imaging data and non-imaging data, identify and segment target lesion areas, and extract depth imaging features and depth non-imaging features respectively. The feature fusion module is used to fuse the depth image features and depth non-image features to generate a comprehensive feature representation of the lesion; The microenvironment graph modeling and reasoning module is used to segment key anatomical structures around the target lesion, construct a heterogeneous graph, and use graph neural networks for reasoning to obtain updated features; The decision generation module is used to fuse the comprehensive feature representation with the updated features, and generate auxiliary decision information based on the fused features.

[0016] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention constructs a heterogeneous map of lesions and key anatomical structures, and utilizes graph neural networks for reasoning to automatically and quantitatively model key pathological interactions such as nodule traction on blood vessels and invasion of bronchi. It achieves precise modeling and deep reasoning of complex interactions between lesions and their microenvironment, greatly enhancing the biological basis of diagnosis and the reliability of the model.

[0017] This invention constructs a graph structure and defines higher-order causal edges that reflect the deformation or invasion of anatomical structures caused by lesions. By using image processing techniques, subjective radiographic descriptions are transformed into objective and measurable graph edge attributes, providing the model with strong guiding signals with clear clinical semantics.

[0018] This invention fuses imaging and non-imaging data through a cross-modal attention mechanism to generate comprehensive features of the lesion itself; then, it fuses the relationship features between the lesion and the microenvironment through graph reasoning; finally, it integrates the two for decision-making. This constructs a hierarchical, collaborative multimodal information fusion framework, achieving deep integration from data to decision. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a flowchart of the method of the present invention.

[0021] Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1, please refer to Figure 1 As shown in this embodiment, a medical decision support method based on multimodal deep learning includes: S1. Obtain medical imaging data and non-imaging data of the target object; S2. Based on the medical imaging data and non-imaging data, identify and segment the target lesion area, and extract the depth imaging features and depth non-imaging features of the target lesion area respectively. S3. Fuse the depth image features and the depth non-image features to generate a comprehensive feature representation of the lesion; S4. Taking the target lesion area as the center, divide at least one key anatomical structure around it; S5. Construct a heterogeneous graph representing the spatial and structural relationship between the target lesion region and the surrounding key anatomical structures, and use a graph neural network to reason about the heterogeneous graph to obtain updated features; S6. The integrated feature representation is fused with the updated feature, and auxiliary decision-making information for the target lesion is generated based on the fused feature.

[0024] As described in S1-S6 above, the fusion analysis and deep reasoning of multi-source information occupy a central position in intelligent medical decision support systems. Currently, most existing deep learning-based diagnostic methods focus on automatically extracting visual features of lesions from medical images, or further performing simple splicing or early fusion with structured clinical data. Undeniably, these methods can improve the objectivity and consistency of diagnosis to some extent, providing valuable references for doctors. However, they have a significant limitation: they cannot model and reason about the complex spatial and structural interactions between lesions and their surrounding anatomical microenvironment. In clinical practice, secondary signs such as the traction of nodules on surrounding blood vessels and invasion of the bronchial wall are key criteria for radiologists to determine benign or malignant conditions. Existing methods view lesion features in isolation or only perform superficial data merging, neglecting to model this crucial diagnostic dimension. This makes it difficult for the model to fully simulate expert diagnostic thinking, limiting its generalization ability in complex and atypical cases, and ultimately potentially affecting the reliability of decisions and clinical adoption.

[0025] This invention constructs a heterogeneous graph of lesions and key anatomical structures, and utilizes graph neural networks for reasoning to automatically and quantitatively model key pathological interactions such as nodule traction on blood vessels and invasion of bronchi. It achieves accurate modeling and deep reasoning of complex interactions between lesions and the microenvironment, greatly enhancing the biological basis of diagnosis and the reliability of the model. This invention constructs a graph structure, defining higher-order causal edges reflecting the deformation or invasion of anatomical structures caused by lesions. This transforms subjective imaging descriptions into objective, measurable graph edge attributes through image processing technology, providing the model with strong guidance signals with clear clinical semantics. This invention fuses imaging and non-imaging data through a cross-modal attention mechanism to generate comprehensive lesion features; then, it fuses the relationship features between lesions and the microenvironment through graph reasoning; finally, it integrates the two for decision-making. This constructs a hierarchical, collaborative multimodal information fusion framework, achieving deep integration from data to decision.

[0026] In one embodiment, step S1 of acquiring medical imaging data and non-imaging data of the target object includes: The target individuals are those undergoing clinical diagnosis or health screening. For example, an individual may undergo a chest CT scan due to an abnormal shadow found in their lungs, generating corresponding imaging data. Simultaneously, the individual's electronic medical record system contains historical medical records, laboratory test reports, etc., constituting the aforementioned non-imaging data. Based on this specific individual's multi-source data, a comprehensive analysis is performed to provide decision support. As described above, in the actual application of this method, all data acquisition and processing strictly adhere to medical data privacy and security regulations. All data used for analysis and model training is used only after obtaining informed consent from patients and approval from the hospital's ethics committee. Before inputting into the system, non-image data containing direct personal identifiers (such as name, ID number, address, and contact information) are anonymized or desensitized; for medical image data, personal information contained in its file header is removed. In the data preprocessing module, the system directly extracts deep features from the desensitized raw data. The deep image features and deep non-image features are abstract feature vectors transformed by the network model. These feature vectors themselves cannot be used to reconstruct the original personal identification information or the complete original report text, thus achieving privacy isolation at the technical level. The entire processing is usually completed on the hospital's internal secure server or trusted computing environment to ensure that the raw data does not leave the institution's control and further prevent the risk of data leakage.

[0027] In one embodiment, step S2, which involves identifying and segmenting the target lesion region based on the medical imaging data and non-imaging data, and extracting the depth imaging features and depth non-imaging features of the target lesion region respectively, includes: S21. The medical imaging data is a chest CT image, and the target lesion area is a pulmonary nodule; S22, The non-image data includes at least one of electronic medical record text data, gene sequencing data, or pathology report text data; S23. The depth image features refer to the features extracted from the image data of the segmented lung nodule region. S24. The aforementioned deep non-image features refer to semantic or numerical features related to the lung nodules extracted from the non-image data. As described in S21-24 above, in this embodiment, the medical imaging data specifically refers to the patient's thin-slice chest CT scan image. The target lesion region is a pulmonary nodule. The non-image data may be at least one of the following: electronic medical record text data, gene sequencing data, or pathology report text data related to the patient. Before being input into the system, all these data have undergone desensitization and authorization processing according to the privacy specifications of the aforementioned embodiments. Specifically, the identification and segmentation of pulmonary nodules involves automatically processing the chest CT sequence using a trained three-dimensional deep learning segmentation model (e.g., 3DU-Net or its variants). This model can identify suspicious nodule locations within the lung region and output its precise three-dimensional pixel-level segmentation mask. For example, for a CT image, the model can segment an isolated pulmonary nodule region located in the upper lobe of the right lung, measuring 8mm × 6mm; this region is the target lesion region. Depth image features are extracted from the segmented pulmonary nodule region image data. This process is not based on manually designed features, but rather implemented through a pre-trained deep convolutional neural network encoder. Specifically, a CT image patch containing a nodule (e.g., a 64×64×64 voxel region cropped centered on the nodule) is input into the encoder. The network's forward propagation ultimately outputs a high-dimensional feature vector (e.g., a 1024-dimensional vector) at a specific layer (e.g., before a fully connected layer). This vector is the depth image feature, encoding the nodule's morphology, texture, density, and deep abstract patterns in the image. For electronic medical records or pathology reports, natural language processing techniques are employed. First, a pre-trained word embedding model in the medical field is used to convert key clinical descriptions in the text (e.g., ground-glass nodules, lobulated appearance, recent history of hemoptysis) into semantic vectors. Then, a text encoder integrates these vectors, ultimately outputting a fixed-length semantic feature vector that captures key semantic information related to nodule diagnosis in the text. For numerical data such as gene sequencing, it is directly structured into feature vectors. For example, the mutation status of the EGFR gene (such as exon 19 deletion) can be encoded as a specific categorical variable, or quantitative indicators such as circulating tumor DNA concentration can be used as numerical features to jointly construct a numerical feature vector. Through the above steps, the system completes the transformation from raw multimodal data to standardized, computable deep feature vectors. These feature vectors together form the basis for subsequent fusion and inference.

[0028] In one embodiment, step S3, which involves fusing the depth image features and the depth non-image features to generate a comprehensive feature representation of the lesion, includes: S31. The fusion of the depth image features and the depth non-image features is achieved through a cross-modal attention fusion mechanism; As described above, specifically, this mechanism can be implemented using a cross-modal attention module. This module takes as input the deep image features (denoted as Fimg, e.g., a 1024-dimensional vector) and deep non-image features (denoted as Fnonimg, e.g., a 768-dimensional vector) extracted in step S2. First, Fimg and Fnonimg are mapped to a set of query, key, and value vectors, respectively, through a learnable linear transformation layer. For example, the image features generate a query vector Q, and the non-image features generate a key vector K and a value vector V. The similarity (e.g., dot product) between Q and K is calculated and normalized using the Softmax function to generate an attention weight matrix. Each element of this matrix quantifies the correlation strength between a certain aspect of the image feature (represented by Q) and various parts of the non-image features (represented by K). The value vector V of the non-image features is weighted and summed using the aforementioned attention weights to generate an enhanced representation of the image features based on the non-image information context. This process can be formally represented as: AttendedContext = Softmax(Q * K^T / sqrt(d)) * V, where Q, K, and V are obtained by linearly transforming the depth image feature Fimg and the depth non-image feature Fnonimg through three independent fully connected layers (weight matrices WQ, WK, WV), i.e., Q = Fimg * WQ, K = Fnonimg * WK, V = Fnonimg * WV. d is the dimension of the K vector, and sqrt(d) is used to scale the dot product to prevent the gradient from being too large or too small. The Softmax function ensures that the sum of all weights is 1, and AttendedContext is the weighted aggregated context vector. Finally, the original image feature Fimg is fused with its obtained context-enhanced representation (e.g., through residual connections or gating mechanisms), and then integrated and dimensionality-reduced by a feedforward neural network, ultimately outputting a unified, fixed-dimensional comprehensive feature representation of the lesion (e.g., a 512-dimensional vector). Suppose that patients with textual semantics in non-image features have a long history of smoking, while image features encode the visual pattern of rough nodule edges. A cross-modal attention mechanism can automatically learn the implicit association between smoking history and edge roughness, and strengthen the interaction and combination of these two related information when generating a comprehensive feature representation, making the final feature representation more discriminative than a single modality or simple splicing.

[0029] In one embodiment, step S4, which involves segmenting at least one key anatomical structure surrounding the target lesion region, includes: S41, The key anatomical structures include at least one of blood vessels, bronchi and pleura; As described above, a local 3D region of interest (ROI) is defined centered on the 3D centroid of the target lung nodule segmented in the previous step. This ROI should be large enough to encompass neighboring structures that may interact with the nodule, such as a cube centered on the nodule with sides of 80 mm. This significantly reduces the computational scope for subsequent fine-grained segmentation. Within this local ROI, a specialized, trained deep learning segmentation model is used to perform pixel-level segmentation of various key anatomical structures: Vascular segmentation uses a 3D segmentation network sensitive to tubular structures (e.g., 3DU-Net with vascular enhancement filtering preprocessing). This model can extract the hierarchical vascular tree structure from the main pulmonary artery to the terminal pulmonary artery from CT images and output a 3D mask of the vascular lumen. Bronchial segmentation follows a similar principle, using a segmentation model optimized for airway structure (e.g., a model specifically trained on an airway dataset) to segment the bronchial tree from the main bronchus to the subsegmental level. Pleural segmentation: The pleura, a thin membrane covering the lung surface, appears as a bright linear structure on CT. A segmentation model or image processing algorithm (such as gradient- and morphology-based methods) capable of recognizing thin-layer linear features can be used to extract the surface of the visceral pleura. After segmentation, the system not only outputs independent masks for each anatomical structure but also calculates their basic spatial relationships with the target lesion. For example, for each vessel segment near the nodule, the system records its shortest distance to the nodule surface and determines whether there is contact or encapsulation; for the bronchus, it similarly determines whether its wall is adjacent to or invaded by the nodule. This preliminary spatial relationship data will provide the original geometric basis for constructing the edges in the heterogeneous map in the next step. For an 8mm nodule located in the upper lobe of the right lung, this step will first delineate a cubic region around the nodule. Then, the segmentation model identifies a small pulmonary artery branch about 2mm away from the nodule, a subsegmental bronchus close to the edge of the nodule, and the distant visceral pleura within this region. The system records relationships such as the vessel-nodule distance of 2mm and the bronchus-nodule contact.

[0030] In one embodiment, step S5, which involves constructing a heterogeneous map representing the spatial and structural relationships between the target lesion region and the surrounding key anatomical structures, includes: S51. The target lesion area and surrounding key anatomical structures are abstracted as nodes in the figure, and connecting edges are constructed according to the spatial and structural relationships between the nodes. S52. The connecting edges include edges that reflect spatial proximity, edges that reflect the continuity of the anatomical structure itself, and higher-order causal edges that reflect the deformation or invasion of the anatomical structure caused by the lesion. S53, wherein the higher-order causal edge includes deformation / tension edge defined based on deformation field relationship, and invasion / enclosure edge defined based on boundary ambiguity or enclosing geometry relationship.

[0031] As described in S51-S53 above, each anatomical entity of interest is abstracted as a node in the graph. The lesion node has the target lung nodule as its sole core, and its node feature vector can be composed of a portion of the depth image features extracted in step S2, or geometric features (such as volume and sphericity) calculated directly from its segmentation mask. Anatomical structure nodes: Each independent vascular segment, bronchial segment, and pleural patch segmented in step S4 and located within the lesion's local ROI is abstracted as an independent node. The feature vector of each anatomical node may include its own morphological features (such as vessel diameter and bronchial wall thickness), spatial relationship features with the lesion (such as centroid coordinates and the shortest distance vector to the lesion surface), and local radiomics features extracted from its original image. Based on the biophysical relationships between nodes, three types of directed or undirected connecting edges are constructed and assigned initial weights or features: spatial proximity edges are established between the lesion node and all its anatomical nodes, as well as between anatomical nodes whose spatial distance to each other is less than a set threshold (e.g., 5 mm). Edge weights can be calculated using the reciprocal of the Euclidean distance or other decay functions, quantifying spatial proximity. Anatomical continuity edges are established within the same anatomical structure (e.g., between two connected blood vessels or branches of the same bronchial tree), reflecting their inherent topological connectivity. This is automatically determined through connectivity analysis on a segmentation mask. Higher-order causal edges are key to this invention, aiming to model the pathological effects caused by lesions. This requires defining image-derived indices: Deformation / traction edges are defined by comparing the actual morphology of blood vessels / bronchioles near the lesion region with the estimated normal morphology (estimated through local image context interpolation or statistical shape models). The deformation field of the region surrounding the lesion is calculated through non-rigid image registration, quantifying the degree of displacement, bending, or traction of blood vessels / bronchioles, and quantifying this as the weight or feature of the edge. Invasion / encapsulation edges are defined by analyzing the image features of the boundary between the lesion and adjacent anatomical structures. The image gradient intensity, texture uniformity variation, or curvature of the contact surface in the area where the lesion boundary contacts the blood vessel / bronchial wall is calculated. If the boundaries become blurred, rough, or the lesion morphology shows a tendency to envelop, such edges are created, and their strength is measured by the quantified values ​​of the aforementioned features (such as the rate of reduction of gradient magnitude). For a lung nodule, the constructed heterogeneous graph contains one lesion node, two surrounding vascular nodes, and one bronchial node. The graph includes spatial proximity edges from the lesion to each anatomical node; continuous edges between vascular nodes; if a vessel is pushed and bent by the nodule, a deformable edge is created pointing from the lesion to the vessel, and its weight reflects the calculated degree of bending; if the boundary between the nodule and the bronchial wall is blurred, an invasion edge is created, and its weight reflects the degree of gradient blurring in the boundary region. Through the above steps, a heterogeneous graph containing multiple types of nodes, multiple types of edges, and their quantified attributes is accurately constructed.

[0032] In one embodiment, step S5, which uses a graph neural network to infer the heterogeneous graph and obtain updated features, includes: S54. A heterogeneous graph attention network is adopted, which uses an attention mechanism to assign different weights to different types of connection edges and neighbor nodes in order to aggregate information.

[0033] As described above, a heterogeneous graph attention network is specifically used for implementation. This network takes the aforementioned heterogeneous graph as input, and its core mechanism dynamically assigns different importance weights to different types of connecting edges and neighboring nodes within the neighborhood of each node through an attention mechanism, followed by information aggregation. The heterogeneous graph constructed in step S5 is input into the network. Each node in the graph (lesion node, blood vessel node, etc.) has its initial feature vector, and each type of edge (proximity edge, continuous edge, deformable edge, etc.) can also have initial weights or features. The network consists of multiple stacked graph attention layers. In each layer, for each target node (e.g., lesion nodule), neighbor sampling and grouping are performed based on the edge type, sampling or grouping the multi-hop neighbors of the target node. Type-dependent attention calculation: For each type of connecting edge (e.g., deformable edges are one type, proximity edges are another), attention weights are calculated independently. This is achieved through a learnable attention function, which takes the target node features, the neighboring node features on a certain type of edge, and the edge's own features (if any) as input, and outputs a scalar weight. This weight represents the importance of the neighbor node information connected to the target node through this type of edge in the current inference task. Based on the calculated attention weights, the feature information from all neighbor nodes via various edge types is weighted and summed, then aggregated onto the target node. The aggregated neighbor information is combined with the target node's own features (e.g., through concatenation or addition), and then a new feature representation of the target node at the current layer is generated using a learnable weight matrix and a non-linear activation function (such as ReLU). By stacking multiple such graph attention layers, the features of each node are fused with information from its multi-hop neighbors transmitted through various relational paths. After L layers of propagation, the updated feature representations of all nodes are finally obtained. The final layer feature representation of the lesion (target nodule) node is extracted separately as the output of the entire graph inference process, i.e., the updated feature. This feature vector deeply integrates the lesion's own attributes, the structural information of the surrounding environment, and the complex interaction patterns between them weighted by the attention mechanism. For the previously constructed graph containing lesions, blood vessels, and bronchial nodes, in the first attention layer, the lesion node calculates its attention weights for blood vessel neighbors connected by deformed edges and bronchial neighbors connected by invasive edges. The network may learn to assign higher attention to the severely stretched blood vessel (with a high weight on the deformed edge). After multiple propagations, the updated features of the lesion node not only include information about itself and its direct neighbors but also indirectly incorporate information about distant lung regions connected by bronchial continuity edges.

[0034] In one embodiment, step S5, which uses a graph neural network to infer the heterogeneous graph and obtain updated features, further includes: S55. A physical heuristic model is used to model and infer the interaction relationship between the nodes, wherein the target lesion area is modeled as a gravitational core, the key anatomical structure is modeled as an orbital entity affected by it, and the interaction process under the action of the gravitational field is simulated using dynamic equations.

[0035] As described above, this embodiment employs a physical heuristic model to model and reason about the interaction relationships between nodes. Its core idea is to analogize biological pathological processes such as invasion and traction to dynamic processes under the influence of a gravitational field in physics, thereby providing interpretable physical constraints and priors for graph reasoning. The target lesion region (pulmonary nodule) is modeled as a gravitational core with mass M. This mass can be quantified as an indicator of the nodule's radiographic malignancy potential, for example, a scalar value obtained by mapping its depth image features through a small perceptron. The surrounding key anatomical structural nodes (blood vessels, bronchial segments) are modeled as orbital entities influenced by the gravitational core. Each entity is assigned a mass mi (which can be initialized to a constant or based on its morphological characteristics) and a spatial position pi (taken from the centroid coordinates of its three-dimensional mask). A virtual gravitational field is defined between the lesion node and each anatomical node. According to Newton's law of universal gravitation, the gravitational force Fi exerted by the lesion node on the anatomical node i can be formally expressed as: Fi = G*(M*mi) / (||p0-pi||^2+ε), where: M: the mass of the target lesion node, a learnable scalar parameter, whose initial value can be obtained by mapping its depth image features through a single-layer perceptron, representing the potential malignancy or influence of the lesion. mi: the mass of the i-th key anatomical structure node, which can be set to a fixed constant (e.g., 1.0) or set according to its morphological characteristics (e.g., blood vessel diameter). p0, pi: the centroid coordinate vectors of the target lesion and the i-th anatomical node in three-dimensional space, respectively. ||p0-pi||: the Euclidean distance between the two points, representing the degree of spatial separation. G: a learnable scalar parameter, representing the global gravitational constant, controlling the overall strength of the interaction. ε: a very small positive smoothing term (e.g., 1e-6) to prevent the denominator from being zero and ensure numerical stability. The calculated vector Fi, directed from the lesion to the anatomical node, quantifies the intensity of the pathological force exerted by the lesion on that specific structure. This force is interpreted as a physical analogy of the pathological force exerted by the lesion on surrounding structures. This force can be further decomposed into two pathologically relevant components: the normal component may lead to displacement or deformation of the structure, simulating traction or displacement effects; the tangential component may lead to rotation or envelopment of the structure along the lesion boundary, simulating invasion or envelopment tendencies. By simulating the effect of this force within a small time step Δt, the equivalent displacement Δdi or deformation energy Ei of the anatomical node can be updated. The amplitude of Δdi or Ei is the physical quantification of the interaction intensity between the nodes. Interaction feature generation and update: The calculated physical interaction quantification values ​​(such as the amplitude of Fi, Δdi, Ei) are used as a new set of relational features and fused with the original node features. Specifically, for each anatomical node, its updated feature hi' can be generated by its original feature hi and the interaction features received from the lesion node. (Fi, Δdi) are combined (e.g., through feature concatenation): hi'=[hi, The characteristics of the lesion node itself can be updated by aggregating the feedback information of all applied forces, for example, h0'=[h0, ψ(ΣFi)], where ψ is the aggregation function. Ultimately, the updated feature representation h0' of the lesion node serves as the updated feature obtained through this physical heuristic model. For a metabolically active (high M value) malignant nodule, it acts as a gravitational core, generating a significant computational gravitational force Fi on a nearby blood vessel (node ​​i). Simulations show that this force primarily results in a significant radial Δdi, which is interpreted as physical evidence of vascular traction. The feature hi of this vascular node incorporates the quantitative information of this traction force. Simultaneously, the nodule node's own features also summarize its total force on all surrounding structures, reflecting its overall invasiveness.

[0036] In one embodiment, step S6, which involves fusing the integrated feature representation with the updated features to generate auxiliary decision-making information for the target lesion based on the fused features, includes: S61. The integrated feature representation is fused with the updated feature and input into the classifier to obtain the benign or malignant classification probability and risk level suggestion of the target lesion; S62. Generate interpretable output that reflects the basis of the auxiliary decision-making information; S63. The interpretability output includes visualizing the connections in the heterogeneous graph where the attention weights assigned by the graph neural network are higher than a predetermined threshold, and outputting an index that quantitatively assesses the intensity of the interaction between the lesion and the surrounding environment. As described in S61-S63 above, the comprehensive lesion feature representation (denoted as Fc) generated in step S3 is fused with the updated feature (denoted as Fu) generated in step S6. The fusion method can be concatenation, weighted summation, or deep interaction through a small fusion network (such as several fully connected layers). For example, Ffused=[Fc, Fu] indicates that the two feature vectors are directly concatenated to form a higher-dimensional fused feature vector Ffused. Ffused is then input into a classifier. This classifier is typically composed of a multilayer perceptron, with its last layer using the Softmax activation function. The classifier outputs a probability vector, such as [Pbenign, Pmalignant], where Pmalignant is the probability that the target lesion is malignant (e.g., 0.87). Based on the classification probability, the system can further map it to clinically commonly used risk level suggestions. For example, thresholds can be set: Pmalignant < 0.1 is low risk, 0.1 ≤ Pmalignant < 0.7 is medium risk, and Pmalignant ≥ 0.7 is high risk. Ultimately, the system outputs decision support information that explicitly includes a malignancy probability of 87% and a high-risk level recommendation. To enhance physicians' trust in the model's decisions, the system simultaneously generates interpretable output reflecting the basis for the decisions, specifically including: a retrospective analysis of the inference process of the graph neural network in the steps. The system extracts the final attention weights assigned to each connection edge in the graph by the heterogeneous graph attention network. Connection edges with attention weights higher than a preset threshold (e.g., the top 10% of weights or an absolute value greater than 0.1) are visually marked with highlighted, colored lines, or specific icons on the 3D reconstructed view or corresponding slice of the original CT image. For example, an edge representing invasion with a high attention weight is rendered as a thick red arrow pointing from the nodule to the invaded bronchial wall. This intuitively reveals to physicians the most important lesion-environment interaction relationship upon which the model bases its judgment. The system calculates one or more indices to quantitatively assess the overall interaction strength between the lesion and its surrounding environment. For example, it calculates the mean or sum of the attention weights of all edges emanating from the lesion node as an indicator of its overall environmental influence. If a physics-inspired model is used, the calculated total force amplitude or total deformation energy can be directly output as the interaction intensity index. The number of edges identified as invasive or deformable edges with high weights in the statistical graph is also considered. This index can be a numerical value (e.g., interaction intensity: 8.5), providing doctors with an objective and traceable quantitative assessment indicator. For the aforementioned case, the system ultimately outputs: malignancy probability: 87%, high risk; the nodule-bronchial edge identified as the key invasive relationship is highlighted on the CT image, and the interaction intensity index is displayed as 7.2.

[0037] In one embodiment, the method further includes the step of: The medical imaging data were subjected to whole-lung field effect analysis to extract the spatial distribution characteristics of lesions within the bilateral lungs and the radiomics characteristics of the background lung parenchyma. The extracted spatial distribution features of the lesions and the radiomics features of the background lung parenchyma are fused with the comprehensive feature representation and the updated features to generate the auxiliary decision information. As described above, a whole-lung segmentation model (e.g., a semantic segmentation network pre-trained on a large amount of CT data) is used to process the input entire chest CT sequence, accurately segmenting the contours of the left and right lungs to obtain a three-dimensional lung mask. Within this whole-lung mask, all target lesion regions identified in step S2 (including target nodules and other suspicious small nodules) are excluded, and the remaining lung tissue region is defined as background lung parenchyma for subsequent feature extraction. All visible lung nodules (including target nodules and other minor nodules) are detected and located throughout the entire lung. Based on this, a set of statistical features quantifying their spatial distribution is calculated, such as the total number of nodules in both lungs and the number of nodules per lobe. The mean distance or standard deviation of the distance between all nodule centroids is calculated to measure whether the nodules are clustered or dispersed. The ratio of the number or volume of nodules in the left and right lungs is compared. The distribution ratio of nodules in each lung lobe (upper, middle, and lower lobes) is statistically analyzed. Within the defined background lung parenchyma region, a set of standard radiomics features is extracted. This is typically achieved using specialized radiomics software libraries. Features include: first-order statistical features such as the mean, standard deviation, skewness, and kurtosis of CT values, used to assess overall density distribution; and texture features such as contrast, correlation, entropy, and homogeneity extracted from the gray-level co-occurrence matrix, used to quantify the texture roughness, uniformity, and structural complexity of lung parenchyma, aiding in the identification of interstitial changes or early emphysema. The extracted global features (the lesion spatial distribution feature vector Fglobaldist and the background lung parenchyma feature vector Fglobaltexture) are concatenated to form the global context feature Fglobal=[Fglobaldist, Fglobaltexture]. In the final decision fusion stage, the global context feature Fglobal is fused a second time with the original local fusion feature Ffused (obtained by fusing the comprehensive feature representation with the updated feature). The fusion method can be to directly concatenate them into a richer feature vector Ffinal=[Ffused, Fglobal] before inputting it into the classifier; or it can use a gated attention mechanism to allow the model to adaptively determine the weights assigned to local and global features during the final decision. The updated classifier will generate final benign / malignant probabilities and risk level recommendations based on Ffinal. Simultaneously, key indicators from the global features (such as multiple nodules in both lungs and a high background emphysema index) can also be provided as part of the interpretability output for clinicians' reference. One patient's target nodule was judged to be of moderate risk (65% malignancy probability) by the local model. However, whole-lung field effect analysis revealed multiple (>5) micronodules in both lungs of this patient, with a scattered spatial distribution (spatial distribution characteristics), and a significantly increased background lung parenchyma texture entropy value (suggesting interstitial inflammation).These global features are given high weights through the fusion network, and the system may eventually raise the overall risk level to high risk and note in the report: The decision took into account the characteristics of the target nodules and the background of multiple nodules in both lungs, and close follow-up or further examination is recommended.

[0038] Example 2, please refer to Figure 2 As shown in this embodiment, a medical auxiliary decision-making system based on multimodal deep learning includes: The data acquisition and processing module is used to acquire and process medical imaging data and non-imaging data, identify and segment target lesion areas, and extract depth imaging features and depth non-imaging features respectively. The feature fusion module is used to fuse the depth image features and depth non-image features to generate a comprehensive feature representation of the lesion; The microenvironment graph modeling and reasoning module is used to segment key anatomical structures around the target lesion, construct a heterogeneous graph, and use graph neural networks for reasoning to obtain updated features; The decision generation module is used to fuse the comprehensive feature representation with the updated features, and generate auxiliary decision information based on the fused features.

[0039] As described above, the data acquisition and processing module, as the system's input, is responsible for interfacing with the medical image archiving and communication system, hospital information system, and laboratory information system to acquire and coordinate the processing of multimodal data. This module receives and parses input medical image data (such as DICOM format chest CT sequences) and non-image data (such as structured electronic medical records, gene reports, or pathology texts). A built-in data preprocessing submodule performs standardization, denoising, and anonymization operations on the raw data to ensure data quality and privacy security; its processing flow is consistent with the privacy specifications in the aforementioned method embodiment S1. This module integrates a lesion recognition and segmentation engine (such as a 3DU-Net-based model) to automatically identify and segment target lesion regions (such as lung nodules) in images. Finally, this module uses a built-in deep feature extractor (such as a CNN encoder or NLP encoder) to extract standardized deep image features and deep non-image features from the segmented lesion images and related non-image data, respectively, and outputs these two feature vectors to the next module. The feature fusion module is responsible for high-level fusion of deep features from different modalities. This module receives two feature vectors from the data acquisition and processing module. Its core is a cross-modal attention fusion submodule. This submodule calculates cross-modal attention weights, guiding image features to focus on the most relevant non-image semantic information, and performs weighted aggregation. After fusion calculation, this module outputs a unified, multimodal comprehensive feature representation of the lesion. This feature representation, as a high-level encoding of the lesion's own attributes, is sent to the decision generation module. The microenvironment graph modeling and reasoning module is responsible for simulating the local anatomical microenvironment of the lesion and performing relational reasoning. This includes an anatomical structure segmentation submodule, which runs a specialized model within a local region centered on the target lesion, performing fine segmentation of key anatomical structures such as blood vessels, bronchi, and pleura. The heterogeneous graph construction submodule then abstracts the target lesion and each segmented anatomical structure entity as graph nodes, and constructs various types of connection edges based on their spatial, topological, and pathological relationships (such as proximity, continuity, invasion, and traction), thus forming a structured heterogeneous graph. Graph Inference Engine: This module includes at least one graph inference engine for deep analysis of the constructed heterogeneous graph: Heterogeneous Graph Attention Network Engine: Through a multi-layer attention mechanism, it dynamically learns the importance of different nodes and edge types, iteratively updates node features, and finally extracts updated features representing the complex interaction between lesions and the microenvironment. Optional Physics-Inspired Inference Engine: As an alternative or supplement, this module can be configured with another inference engine that models lesions as sources of gravity and anatomical structures as affected entities, quantifies the interaction intensity by simulating dynamic processes, and generates physics-inspired updated features.

[0040] This module outputs the updated features obtained from the inference to the decision generation module. The decision generation module is responsible for integrating all information to generate a final clinically applicable auxiliary decision-making report. This module receives the comprehensive feature representation from the feature fusion module and the updated features from the microenvironment graph modeling and inference module. Feature fusion and classification submodule: First, it fuses the two types of features (e.g., by splicing) and inputs the fused features into a classifier (e.g., a multilayer perceptron). The classifier outputs the benign or malignant probability of the target lesion and can map it to a risk level suggestion (e.g., low, medium, high risk) based on a preset threshold. Interpretability generation submodule: Simultaneously, this module generates interpretable outputs for the decision-making basis. For example, it extracts the key relationship edges with the highest attention weight from the graph inference engine and visually highlights them on the original image; or it calculates and outputs a lesion-environment interaction intensity index. Finally, this module packages the structured decision information (probability, risk level) and interpretable information (visual labels, quantitative indices) to form complete auxiliary decision-making information, which is presented to the user through a user interface (e.g., a doctor's workstation).

[0041] During operation, the data flow sequentially passes through the four modules mentioned above. The raw multimodal data is transformed into initial features by the data acquisition and processing module; the feature fusion module extracts the comprehensive features of the lesion itself; the microenvironment modeling and reasoning module analyzes and extracts the interaction features between the lesion and the environment; finally, the decision generation module aggregates all features to generate an auxiliary decision-making report that combines conclusions and evidence. Each module has a clear division of labor and progresses sequentially, jointly realizing intelligent and interpretable medical auxiliary decision-making functions.

[0042] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A medical decision support method based on multimodal deep learning, characterized in that, include: Acquire medical imaging and non-imaging data of the target object; Based on the medical imaging data and non-imaging data, the target lesion area is identified and segmented, and the depth imaging features and depth non-imaging features of the target lesion area are extracted respectively. By fusing the depth image features and the depth non-image features, a comprehensive feature representation of the lesion is generated; Centered on the target lesion area, at least one key anatomical structure around it is segmented; A heterogeneous graph representing the spatial and structural relationship between the target lesion region and the surrounding key anatomical structures is constructed, and a graph neural network is used to reason about the heterogeneous graph to obtain updated features; The integrated feature representation is fused with the updated feature, and auxiliary decision-making information for the target lesion is generated based on the fused feature.

2. The medical decision support method based on multimodal deep learning according to claim 1, characterized in that: The step of identifying and segmenting the target lesion region based on the medical imaging data and non-imaging data, and extracting the depth imaging features and depth non-imaging features of the target lesion region respectively, includes: The medical imaging data is a chest CT image, and the target lesion area is a pulmonary nodule; The non-image data includes at least one of electronic medical record text data, gene sequencing data, or pathology report text data; The depth image features refer to the features extracted from the image data of the segmented lung nodule region; The deep non-image features refer to the semantic or numerical features related to the lung nodules extracted from the non-image data.

3. The medical decision support method based on multimodal deep learning according to claim 1, characterized in that: The step of fusing the depth image features and the depth non-image features to generate a comprehensive feature representation of the lesion includes: The fusion of depth image features and depth non-image features is achieved through a cross-modal attention fusion mechanism.

4. The medical decision support method based on multimodal deep learning according to claim 1, characterized in that: The step of segmenting at least one key anatomical structure surrounding the target lesion region as the center includes: The key anatomical structures include at least one of blood vessels, bronchi, and pleura.

5. A medical decision support method based on multimodal deep learning according to claim 1, characterized in that: The step of constructing a heterogeneous map representing the spatial and structural relationships between the target lesion region and the surrounding key anatomical structures includes: The target lesion area and surrounding key anatomical structures are abstracted as nodes in the diagram, and connecting edges are constructed based on the spatial and structural relationships between the nodes. The connecting edges include edges that reflect spatial proximity, edges that reflect the continuity of the anatomical structure itself, and higher-order causal edges that reflect the deformation or invasion of the anatomical structure caused by the lesion. The higher-order causal edges include deformation and / or tension edges defined based on deformation field relations, and invasion and / or enclosing edges defined based on boundary ambiguity or enclosing geometry relations.

6. A medical decision support method based on multimodal deep learning according to claim 1, characterized in that: The step of using a graph neural network to infer the heterogeneous graph and obtain updated features includes: A heterogeneous graph attention network is adopted, which uses an attention mechanism to assign different weights to different types of connection edges and neighbor nodes in order to aggregate information.

7. A medical decision support method based on multimodal deep learning according to claim 5, characterized in that: The step of using a graph neural network to infer the heterogeneous graph and obtain updated features includes: A physics-inspired model is used to model and infer the interaction relationship between the nodes. The target lesion region is modeled as a gravitational core, and the key anatomical structure is modeled as an orbital entity affected by it. The dynamic equation is used to simulate the interaction process under the action of the pseudo-gravitational field between the gravitational core and the orbital entity.

8. A medical decision support method based on multimodal deep learning according to claim 1, characterized in that: The step of fusing the integrated feature representation with the updated features and generating auxiliary decision-making information for the target lesion based on the fused features includes: The integrated feature representation is fused with the updated feature and input into the classifier to obtain the benign or malignant classification probability and risk level suggestion of the target lesion; Generate interpretable output that reflects the basis of the auxiliary decision-making information; The interpretability output includes visualizing connections in the heterogeneous graph where the attention weights assigned by the graph neural network are higher than a predetermined threshold, and outputting an index that quantitatively assesses the intensity of interaction between the lesion and the surrounding environment.

9. A medical decision support method based on multimodal deep learning according to claim 1, characterized in that: The method further includes the following steps: The medical imaging data were subjected to whole-lung field effect analysis to extract the spatial distribution characteristics of lesions within the bilateral lungs and the radiomics characteristics of the background lung parenchyma. The extracted spatial distribution features of the lesions and the radiomics features of the background lung parenchyma are fused with the comprehensive feature representation and the updated features to generate the auxiliary decision information.

10. A medical decision support system based on multimodal deep learning, used to implement the medical decision support method based on multimodal deep learning as described in any one of claims 1-9, characterized in that, include: The data acquisition and processing module is used to acquire and process medical imaging data and non-imaging data, identify and segment target lesion areas, and extract depth imaging features and depth non-imaging features respectively. The feature fusion module is used to fuse the depth image features and depth non-image features to generate a comprehensive feature representation of the lesion; The microenvironment graph modeling and reasoning module is used to segment key anatomical structures around the target lesion, construct a heterogeneous graph, and use graph neural networks for reasoning to obtain updated features; The decision generation module is used to fuse the comprehensive feature representation with the updated features, and generate auxiliary decision information based on the fused features.

Images