Multi-modal medical image complex high-order correlation modeling method and device
Patent Information
- Application Number
- CN202311153046.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2026-09-08
- Estimated Expiration
- 2043-09-07
AI Technical Summary
[0004]本申请提供一种多模态医学影像复杂高阶关联建模方法、装置、电子设备及存储介质，以解决相关技术中关联结构与特征表示之间映射机理不清晰，可解释性较差，且缺乏对多模态医学影像高阶关联表示的一致性分析，对多模态影像的复杂高阶关联建模能力不足，无法在模态确实情景下进行预测等问题
[0022] This application embodiment can perform joint iterative optimization of unimodal and multimodal hypergraph models, and input the data to be predicted to output the prediction results of unknown samples for downstream clinical tasks. Thus, this application embodiment can realize an alternating optimization mechanism of unimodal and multimodal hypergraphs by establishing a hypergraph federated learning module. This enables the establishment of complex high-valence association representations within and between modalities, while achieving prediction tasks based on unimodal and multimodal medical images. This improves the modeling ability of complex high-order associations influenced by multimodal effects. The mapping mechanism between the association structure and the feature table is clear, highly interpretable, and meets the actual needs of prediction in modal contexts.
Smart Images
Figure CN117437173B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image analysis technology, and in particular to a method and apparatus for modeling complex high-order correlations in multimodal medical images. Background Technology
[0002] With the popularization of medical imaging equipment, digital medical technology and medical image management systems, the amount of multimodal medical image data has surged, and the data types are showing high-dimensional and diversified development. Among the related technologies, it is possible to understand and analyze the complex high-order correlations in multimodal medical image data, so as to reveal the essential correlations between multimodal data and eliminate information redundancy.
[0003] However, the methods in the related technologies lack consistency analysis of high-order association representations of multimodal medical images, have insufficient ability to model complex high-order associations of multimodal images, and cannot make predictions in modal real-world scenarios; at the same time, the mapping mechanism between association structure and feature representation is unclear, resulting in poor interpretability. Summary of the Invention
[0004] This application provides a method, apparatus, electronic device, and storage medium for modeling complex high-order associations in multimodal medical images, in order to solve the problems in related technologies such as unclear mapping mechanism between association structure and feature representation, poor interpretability, lack of consistency analysis of high-order association representations of multimodal medical images, insufficient ability to model complex high-order associations of multimodal images, and inability to make predictions in modal real-world scenarios.
[0005] The first aspect of this application provides a method for modeling complex high-order associations in multimodal medical images, comprising the following steps: extracting radiomics features and depth features from multimodal medical images; constructing different unimodal hypergraph models based on the radiomics features and depth features; aggregating the unimodal hypergraph models into a multimodal hypergraph model; performing joint iterative optimization on the unimodal hypergraph model and the multimodal hypergraph model; inputting the data to be predicted for downstream clinical tasks into the optimized unimodal hypergraph model and the multimodal hypergraph model; and outputting the prediction results for unknown samples of downstream clinical tasks.
[0006] Optionally, constructing different unimodal hypergraph models based on the radiomics features and the depth features includes: performing multi-type structural associations on the extracted radiomics features and depth features; defining a multi-dimensional hypergraph structural association method based on clinical information and feature distance measurement; establishing multi-dimensional hyperedge groups for the radiomics features and the depth features based on the hypergraph structural association method; defining different weights for different hyperedges based on the number of vertices connected by each hyperedge in the multi-dimensional hyperedge group and clinical information; fusing the radiomics feature hyperedge groups and depth feature hyperedge groups in different modalities to form a unimodal hyperedge group; and using the unimodal hyperedge group and corresponding weights to establish a unimodal hypergraph model.
[0007] Optionally, the step of aggregating each unimodal hypergraph model into a multimodal hypergraph model includes: using each unimodal hypergraph model as a local model and aggregating them using federated learning to obtain the multimodal hypergraph model.
[0008] Optionally, joint iterative optimization of the unimodal hypergraph model and the multimodal hypergraph model includes: iteratively optimizing each unimodal hypergraph model and updating the model parameters of the multimodal hypergraph model based on the model parameters of each unimodal hypergraph model after iterative optimization; and iteratively optimizing the multimodal hypergraph model and updating the model parameters of each unimodal hypergraph model based on the model parameters of the multimodal hypergraph model after iterative optimization.
[0009] Optionally, the step of inputting the data to be predicted for the downstream clinical task into the optimized unimodal hypergraph model and the multimodal hypergraph model, and outputting the prediction results of unknown samples for the downstream clinical task, includes: extracting radiomics features and depth features from the data to be predicted; establishing a hypergraph model and a hypergraph correlation matrix for the data to be predicted, and fusing the hypergraph model with the optimized unimodal hypergraph model and the multimodal hypergraph model respectively; inputting the hypergraph correlation matrix, the radiomics features, and the depth features into the fused unimodal hypergraph model and the multimodal hypergraph model, and outputting the prediction results of unknown samples for the downstream clinical task.
[0010] Optionally, the single-modal data is input into the corresponding single-modal hypergraph model to generate single-modal complex high-order association representation information, and the single-modal complex high-order association representation information is sent to a classifier or evaluation model for prediction.
[0011] Optionally, the multimodal hypergraph model establishes a multimodal hypergraph convolutional layer through the multimodal hypergraph association matrix and feature transformation parameters to generate multimodal complex high-order association representation information, and sends the multimodal complex high-order association representation information into a classifier or evaluation model for prediction.
[0012] A second aspect of this application provides a multimodal medical image complex high-order correlation modeling device, comprising: an extraction module for extracting radiomics features and depth features of multimodal medical images; an optimization module for constructing different unimodal hypergraph models based on the radiomics features and the depth features, aggregating the unimodal hypergraph models into a multimodal hypergraph model, and performing joint iterative optimization on the unimodal hypergraph model and the multimodal hypergraph model; and an output module for inputting the data to be predicted for downstream clinical tasks into the optimized unimodal hypergraph model and the multimodal hypergraph model, and outputting the prediction results of unknown samples for downstream clinical tasks.
[0013] Optionally, the optimization module is further configured to: perform multi-type structural association on the extracted radiomics features and depth features; define a multi-dimensional hypergraph structural association method based on clinical information and feature distance measurement; establish a multi-dimensional hyperedge group for the radiomics features and depth features based on the hypergraph structural association method; define different weights for different hyperedges based on the number of vertices connected by each hyperedge in the multi-dimensional hyperedge group and clinical information; fuse the radiomics feature hyperedge group and the depth feature hyperedge group in different modalities to form a single-modality hyperedge group; and establish a single-modality hypergraph model using the single-modality hyperedge group and corresponding weights.
[0014] Optionally, the optimization module is further configured to: use each unimodal hypergraph model as a local model and aggregate them using federated learning to obtain the multimodal hypergraph model.
[0015] Optionally, the optimization module is further configured to: perform iterative optimization on each unimodal hypergraph model, and update the model parameters of the multimodal hypergraph model according to the model parameters of each unimodal hypergraph model after iterative optimization; perform iterative optimization on the multimodal hypergraph model, and update the model parameters of each unimodal hypergraph model according to the model parameters of the multimodal hypergraph model after iterative optimization.
[0016] Optionally, the output module further includes: extracting radiomics features and depth features from the data to be predicted; establishing a hypergraph model and a hypergraph correlation matrix for the data to be predicted, and fusing the hypergraph model with the optimized unimodal hypergraph model and the multimodal hypergraph model respectively; inputting the hypergraph correlation matrix, the radiomics features, and the depth features into the fused unimodal hypergraph model and the multimodal hypergraph model, and outputting the prediction results for unknown samples in the downstream clinical task.
[0017] Optionally, the device is further configured to input single-modal data into the corresponding single-modal hypergraph model to generate single-modal complex high-order association representation information, and send the single-modal complex high-order association representation information into a classifier or evaluation model for prediction.
[0018] Optionally, the multimodal hypergraph model establishes a multimodal hypergraph convolutional layer through the multimodal hypergraph association matrix and feature transformation parameters to generate multimodal complex high-order association representation information, and sends the multimodal complex high-order association representation information into a classifier or evaluation model for prediction.
[0019] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal medical image complex high-order correlation modeling method as described in the above embodiments.
[0020] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the multimodal medical image complex high-order correlation modeling method as described in the above embodiments.
[0021] Therefore, this application has at least the following beneficial effects:
[0022] This application embodiment can perform joint iterative optimization of unimodal and multimodal hypergraph models, and input the data to be predicted to output the prediction results of unknown samples for downstream clinical tasks. Thus, this application embodiment can realize an alternating optimization mechanism of unimodal and multimodal hypergraphs by establishing a hypergraph federated learning module. This enables the establishment of complex high-valence association representations within and between modalities, while achieving prediction tasks based on unimodal and multimodal medical images. This improves the modeling ability of complex high-order associations influenced by multimodal effects. The mapping mechanism between the association structure and the feature table is clear, highly interpretable, and meets the actual needs of prediction in modal contexts.
[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0025] Figure 1 This is a flowchart illustrating a method for modeling complex high-order correlations in multimodal medical images, as described in an embodiment of this application.
[0026] Figure 2 This is a diagram illustrating the multimodal medical image complex high-order correlation modeling method and system architecture according to an embodiment of this application.
[0027] Figure 3 This is a schematic diagram of the hypergraph federated learning module in an embodiment of this application;
[0028] Figure 4 This is an example diagram of a multimodal medical image complex high-order correlation modeling device according to an embodiment of this application;
[0029] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0030] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0031] Specifically, Figure 1 This is a flowchart illustrating a method for modeling complex high-order correlations in multimodal medical images, as provided in an embodiment of this application.
[0032] like Figure 1 As shown, this method for modeling complex high-order correlations in multimodal medical images includes the following steps:
[0033] In step S101, radiomics features and depth features of multimodal medical images are extracted.
[0034] Among them, radiomics features may include shape, first-order statistical features, gray-level co-occurrence matrix, gray-level running length matrix, gray-level size region matrix, gray-level correlation matrix, and Hu invariant moments, etc.
[0035] It is understandable that, such as Figure 2 As shown, the embodiments of this application can extract features from given medical image data of different modalities, so as to facilitate the use of feature data in subsequent embodiments; wherein, the embodiments of this application can use at least one method to extract radiomics features and deep features, such as using radiomics methods to extract radiomics features and using deep learning methods to extract deep features (such as using the EfficientNet model pre-trained on the ImageNet dataset), etc.
[0036] In step S102, different unimodal hypergraph models are constructed based on radiomics features and depth features. The unimodal hypergraph models are then aggregated into a multimodal hypergraph model, and the unimodal and multimodal hypergraph models are jointly iteratively optimized.
[0037] In this application, the construction of the hypergraph structure can be achieved in at least one way. For example, the association information in clinical information and the feature measurement algorithm can be used to construct the hypergraph structure for different types of features of different modalities.
[0038] It is understood that embodiments of this application can use radiomics features and deep features to construct a unimodal hypergraph model, and then aggregate the corresponding unimodal hypergraph models to generate a multimodal hypergraph model, and perform iterative optimization by merging the unimodal and hypermodal hypergraph models; specifically as follows:
[0039] I. Constructing different unimodal hypergraph models based on radiomics features and depth features
[0040] In this embodiment, different unimodal hypergraph models are constructed based on radiomics features and depth features, including: performing multi-type structural association on the extracted radiomics features and depth features; defining a multi-dimensional hypergraph structural association method based on clinical information and feature distance measurement; establishing multi-dimensional hyperedge groups for radiomics features and depth features based on the hypergraph structural association method; defining different weights for different hyperedges based on the number of vertices connected by each hyperedge in the multi-dimensional hyperedge group and clinical information; fusing the radiomics feature hyperedge groups and depth feature hyperedge groups in different modalities to form a unimodal hyperedge group; and establishing a unimodal hypergraph model using the unimodal hyperedge group and corresponding weights.
[0041] It is understood that the construction of the single-modal hypergraph model in this application embodiment may include defining a hypergraph structure association method, establishing a multi-dimensional hyperedge group, defining hyperedge group weights, and establishing a hypergraph structure based on single-modal medical images, as detailed below:
[0042] (1) Define hypergraph structure association methods
[0043] In this embodiment, a multi-dimensional association method R = {R_1, R_2, ..., R_n, R_(n+1), ..., R_m} can be defined based on clinical information and feature distance measurement methods. The association method (R_1, R_2, ..., R_n) refers to connecting corresponding data points with the same attributes to form hyperedges based on clinical information. Clinical information may include demographic information, medical history, health behavior, clinical diagnosis and treatment information, etc. The association method {R_(n+1), ..., R_t} refers to calculating the distance between features of each data point using a feature distance measurement method (K-nearest neighbor algorithm), and connecting each data point with its K nearest neighbors to form a hyperedge according to a predefined K value.
[0044] (2) Establish a multidimensional hyperedge group
[0045] In this embodiment, multiple hyperedge group definitions can be established based on the multi-dimensional association method R. Representing n medical image modalities, using a multi-dimensional association method For each image modality, sub-association groups are established based on the generated image omics features and deep features. and Where i represents the modality number.
[0046] (3) Define the weights of the hyperedge group
[0047] The embodiments of this application can be based on the above embodiments. and Further establish corresponding hyperedge groups and Based on the number of vertices connected by each hyperedge and clinical information, different weights W = {W1, W2, ..., W...} are defined for different hyperedges. n W n+1 ,W2,…,W m}
[0048] (4) Establish a hypergraph structure based on single-modal medical images
[0049] This application embodiment can fuse image omics feature hyperedge groups and depth feature hyperedge groups from different modalities to form a single-modal hyperedge group, and use it to build a single-modal hypergraph model, represented as follows: in, V represents the hypergraph model for feature generation of the i-th image modality. i ε represents the combination of all data nodes in the i-th image modality. i W represents the combination of multi-dimensional hyperedge groups in the i-th image modality. i This represents the weight corresponding to each hyperedge in the i-th image modality multidimensional hyperedge group.
[0050] Define the correlation matrix H as the adjacency matrix of the i-th image mode:
[0051]
[0052] Where v refers to a data point, e refers to a hyperedge, and W represents the weight of the hyperedge.
[0053] Therefore, this application can construct a single-modality hyper-hypergraph model by using a hypergraph structure association method, establishing multidimensional hyperedge groups, defining hyperedge group weights, and establishing a hypergraph based on single-modality medical images.
[0054] II. Aggregate the individual unimodal hypergraph models into a multimodal hypergraph model.
[0055] In this embodiment of the application, the aggregation of each unimodal hypergraph model into a multimodal hypergraph model includes: using each unimodal hypergraph model as a local model and aggregating them using federated learning to obtain a multimodal hypergraph model.
[0056] It is understandable that, such as Figure 3As shown, the embodiments of this application can establish a hypergraph federated learning module, use each unimodal hypergraph as a local model, and use federated learning to aggregate hypergraph structures to obtain a multimodal hypergraph structure.
[0057] Specifically, in the embodiments of this application, each single-modal hypergraph can be used as a local model θ. i They are aggregated using a federated learning architecture to form a multimodal hypergraph model Θ = 1 / n∑ i θ i Where o represents the mode number, θ i Θ represents the parameters of mode i, and Θ represents the aggregated hypergraph model.
[0058] III. Joint Iterative Optimization of Single-Mode Hypergraph Model and Multi-Mode Hypergraph Model
[0059] In this embodiment of the application, joint iterative optimization of the unimodal hypergraph model and the multimodal hypergraph model includes: iterative optimization of each unimodal hypergraph model, updating the model parameters of the multimodal hypergraph model according to the model parameters of each unimodal hypergraph model after iterative optimization; iterative optimization of the multimodal hypergraph model, updating the model parameters of each unimodal hypergraph model according to the model parameters of the multimodal hypergraph model after iterative optimization.
[0060] It is understandable that, such as Figure 2 As shown in Iterative Optimization 1 and Iterative Optimization 2, this embodiment of the application can use a hypergraph federated learning module to iteratively optimize each unimodal hypergraph model and the multimodal hypergraph after aggregating each unimodal hypergraph structure. After optimization, the optimized model is distributed back to the multimodal hypergraph structure and the unimodal hypergraph structure to replace the current parameters, thereby achieving joint iterative optimization; specifically as follows:
[0061] (1) Iterative optimization of single-modal hypergraph model: The embodiments of this application can calculate the update of aggregate hypergraph model Θ←Θ+ΔΘ and distribute it to each single-modal hypergraph to replace the current parameters, thereby realizing the optimization of single-modal hypergraph structure.
[0062] (2) Iterative optimization of the multimodal hypergraph model: In this embodiment, a complex high-order association representation of the multimodal hypergraph can be generated first. A fully connected layer is added at a fixed point in the network to generate a basic feature representation. The basic feature transformation layer L0 is defined as θ0X, where θ0 represents an optimizable parameterized model. A multimodal hypergraph convolutional layer is established by fusing the multimodal hypergraph association matrix and feature transformation parameters to generate the corresponding complex high-order association representation.
[0063]
[0064] Where σ(·) represents the activation function, D v =∑ e∈ε w(e)H(v,e) and De =∑ v∈v h(v,e) represents the degree of the vertex and the hyperedge of the hypergraph, respectively, and H(v,e) represents the value at the corresponding position in the incidence matrix H.
[0065] After generating multimodal complex high-order associations using a multimodal hypergraph, embodiments of this application can use a loss function for iterative optimization, select a suitable model through the set convergence conditions, and obtain a hypergraph model containing representation information of complex high-order associations of multimodal medical images.
[0066] (3) Alternating optimization: In the embodiments of this application, after iterative optimization of each unimodal hypergraph model, the current parameters can be distributed back to the multimodal hypergraph structure to replace the current parameters, and after iterative optimization of the multimodal hypergraph model, the current parameters can be distributed back to the unimodal hypergraph structure to replace the current parameters, thereby achieving joint iterative optimization.
[0067] In step S103, the data to be predicted for the downstream clinical task is input into the optimized single-modal hypergraph model and multimodal hypergraph model, and the prediction results of the unknown samples of the downstream clinical task are output.
[0068] It is understood that the embodiments of this application can perform downstream clinical prediction tasks based on unimodal hypergraph models and multimodal hypergraph models, and output prediction results; the process of performing downstream clinical prediction tasks in the embodiments of this application can be specifically as follows:
[0069] In this embodiment, the data to be predicted for the downstream clinical task is input into the optimized unimodal hypergraph model and multimodal hypergraph model, and the prediction results of unknown samples for the downstream clinical task are output. This includes: extracting radiomics features and depth features from the data to be predicted; establishing a hypergraph model and a hypergraph correlation matrix for the data to be predicted, and fusing the hypergraph model with the optimized unimodal hypergraph model and multimodal hypergraph model respectively; inputting the hypergraph correlation matrix, radiomics features, and depth features into the fused unimodal hypergraph model and multimodal hypergraph model, and outputting the prediction results of unknown samples for the downstream clinical task.
[0070] Specifically, (1) extracting the image omics features and depth features of the data to be predicted:
[0071] The embodiments of this application can use at least one method to extract image omics features and depth features with prediction data. For example, image omics features can be extracted using image omics methods, and depth features can be extracted using deep learning methods, as described in the above embodiments.
[0072] (2) Establish the hypergraph model and hypergraph correlation matrix of the data to be predicted, and fuse the hypergraph model with the optimized unimodal hypergraph model and multimodal hypergraph model respectively:
[0073] This application embodiment can design a multimodal hypergraph label learning method to mine complex high-order association information between different modalities, and fuse the obtained hypergraph model with the optimized unimodal and multimodal hypergraph models respectively.
[0074] In this embodiment, single-modal data is input into the corresponding single-modal hypergraph model to generate single-modal complex high-order association representation information, and the single-modal complex high-order association representation information is sent to a classifier or evaluation model for prediction.
[0075] It is understood that, for unimodal hypergraph models, the embodiments of this application can predict downstream clinical tasks based on unimodal medical images; the features of the unimodal images are input into the optimized unimodal hypergraph. In this process, a complex high-order association representation of a single modality is generated, and a classifier or evaluation model is used to achieve tasks such as assisted diagnosis and survival prediction.
[0076] In this embodiment, the multimodal hypergraph model establishes a multimodal hypergraph convolutional layer through the multimodal hypergraph association matrix and feature transformation parameters, generates multimodal complex high-order association representation information, and sends the multimodal complex high-order association representation information into a classifier or evaluation model for prediction.
[0077] It is understood that, for multimodal hypergraph models, the embodiments of this application can be as shown in the above embodiments, where the generation of multimodal complex high-order association representations through iterative optimization of multimodal hypergraph models is described. Multimodal hypergraph convolutional layers are established using the multimodal hypergraph association matrix and feature transformation parameters to generate multimodal complex high-order association representation information.
[0078] (3) Input the hypergraph correlation matrix, radiomics features, and deep features into the fused single-modal hypergraph model and multimodal hypergraph model, and output the prediction results of unknown samples for downstream clinical tasks:
[0079] In this embodiment, the generated multimodal complex high-order association representation can be defined as X′, and input into a classifier for node category prediction:
[0080] Y = Cla(X′),
[0081] Where Cla(·) represents the classifier, and Y represents the predicted class label. Inputting this into the evaluation model can be used for tasks such as survival prediction.
[0082] risk = Surv(X′),
[0083] Where Surv(·) represents the survival prediction regression model, and risk represents the predicted risk value. Therefore, in this embodiment, the corresponding hypergraph correlation matrix and the feature input optimized hypergraph can be used to obtain the final downstream clinical task prediction results.
[0084] In summary, the multimodal medical image complex high-order association modeling method proposed in this application can jointly iteratively optimize unimodal hypergraph models and multimodal hypergraph models, and input the data to be predicted to output the prediction results of unknown samples for downstream clinical tasks. Thus, this application embodiment can realize the alternating optimization mechanism of unimodal and multimodal hypergraphs by establishing a hypergraph federated learning module, so that while establishing complex high-value association representations within and between modalities, prediction tasks based on unimodal and multimodal medical images can be realized, improving the modeling ability of multimodal influences on complex high-order associations. The mapping mechanism between the association structure and the feature table is clear, with strong interpretability, meeting the actual needs of prediction in modal contexts.
[0085] Next, referring to the accompanying drawings, a multimodal medical image complex high-order correlation modeling apparatus proposed according to an embodiment of this application is described.
[0086] Figure 4 This is a block diagram of a multimodal medical image complex high-order correlation modeling device according to an embodiment of this application.
[0087] like Figure 4 As shown, the multimodal medical image complex high-order correlation modeling device 10 includes: an extraction module 100, an optimization module 200, and an output module 300.
[0088] The extraction module 100 is used to extract radiomics features and depth features from multimodal medical images; the optimization module 200 is used to construct different unimodal hypergraph models based on radiomics features and depth features, aggregate the unimodal hypergraph models into a multimodal hypergraph model, and perform joint iterative optimization on the unimodal hypergraph model and the multimodal hypergraph model; the output module 300 is used to input the data to be predicted for downstream clinical tasks into the optimized unimodal hypergraph model and the multimodal hypergraph model, and output the prediction results of unknown samples for downstream clinical tasks.
[0089] Optionally, the optimization module 200 is further configured to: perform multi-type structural association on the extracted radiomics features and depth features; define a multi-dimensional hypergraph structural association method based on clinical information and feature distance measurement; establish a multi-dimensional hyperedge group for radiomics features and depth features based on the hypergraph structural association method; define different weights for different hyperedges based on the number of vertices connected by each hyperedge in the multi-dimensional hyperedge group and clinical information; fuse the radiomics feature hyperedge group and the depth feature hyperedge group in different modalities to form a single-modal hyperedge group; and establish a single-modal hypergraph model using the single-modal hyperedge group and corresponding weights.
[0090] Optionally, the optimization module 200 is further used to: aggregate each unimodal hypergraph model as a local model using federated learning to obtain a multimodal hypergraph model.
[0091] Optionally, the optimization module 200 is further configured to: perform iterative optimization on each unimodal hypergraph model, and update the model parameters of the multimodal hypergraph model based on the model parameters of each unimodal hypergraph model after iterative optimization; perform iterative optimization on the multimodal hypergraph model, and update the model parameters of each unimodal hypergraph model based on the model parameters of the multimodal hypergraph model after iterative optimization.
[0092] Optionally, the output module 300 further performs the following functions: extracting radiomics features and depth features from the data to be predicted; establishing a hypergraph model and hypergraph correlation matrix for the data to be predicted, and fusing the hypergraph model with the optimized unimodal hypergraph model and multimodal hypergraph model respectively; inputting the hypergraph correlation matrix, radiomics features, and depth features into the fused unimodal hypergraph model and multimodal hypergraph model, and outputting the prediction results for unknown samples in the downstream clinical task.
[0093] Optionally, the device 10 is further configured to input single-modal data into the corresponding single-modal hypergraph model to generate single-modal complex high-order association representation information, and to send the single-modal complex high-order association representation information into a classifier or evaluation model for prediction.
[0094] Optionally, the multimodal hypergraph model establishes a multimodal hypergraph convolutional layer through the multimodal hypergraph association matrix and feature transformation parameters to generate multimodal complex high-order association representation information, which is then fed into a classifier or evaluation model for prediction.
[0095] It should be noted that the foregoing explanation of the embodiment of the multimodal medical image complex high-order correlation modeling method also applies to the multimodal medical image complex high-order correlation modeling device of this embodiment, and will not be repeated here.
[0096] The multimodal medical image complex high-order association modeling device proposed in this application embodiment can jointly iteratively optimize unimodal hypergraph models and multimodal hypergraph models, and input the data to be predicted to output the prediction results of unknown samples for downstream clinical tasks. Thus, this application embodiment can realize the alternating optimization mechanism of unimodal hypergraph and multimodal hypergraph by establishing a hypergraph federated learning module, so that while establishing complex high-value association representations within and between modalities, prediction tasks based on unimodal and multimodal medical images can be realized, improving the modeling ability of multimodal influence on complex high-order associations. The mapping mechanism between the association structure and the feature table is clear, with strong interpretability, meeting the actual needs of prediction in modal contexts.
[0097] Figure 5A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:
[0098] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.
[0099] When the processor 502 executes the program, it implements the multimodal medical image complex high-order correlation modeling method provided in the above embodiments.
[0100] Furthermore, electronic devices also include:
[0101] Communication interface 503 is used for communication between memory 501 and processor 502.
[0102] The memory 501 is used to store computer programs that can run on the processor 502.
[0103] The memory 501 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.
[0104] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0105] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.
[0106] Processor 502 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of this application.
[0107] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for modeling complex high-order correlations in multimodal medical images.
[0108] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0109] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0110] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0111] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.
[0112] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0113] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A method for modeling complex high-order correlations in multimodal medical images, characterized in that, Includes the following steps: Extracting radiomics and depth features from multimodal medical images; Different unimodal hypergraph models are constructed based on the radiomics features and the depth features. The unimodal hypergraph models are then aggregated into a multimodal hypergraph model. The unimodal hypergraph model and the multimodal hypergraph model are then jointly iteratively optimized. The predicted data for downstream clinical tasks are input into the optimized unimodal hypergraph model and the multimodal hypergraph model, and the prediction results for unknown samples of downstream clinical tasks are output. The construction of different unimodal hypergraph models based on the radiomics features and the depth features includes: Multi-type structural associations are performed on the extracted radiomics features and depth features, and a multi-dimensional hypergraph structural association method is defined based on clinical information and feature distance measurement. Based on the hypergraph structure association method, a multi-dimensional hyperedge group is established for the radiomics features and the depth features. Based on the number of vertices connected by each hyperedge in the multi-dimensional hyperedge group and clinical information, different weights are defined for different hyperedges. After fusing image omics feature hyperedge groups and depth feature hyperedge groups from different modalities to form a single-modal hyperedge group, a single-modal hypergraph model is established using the single-modal hyperedge group and its corresponding weights. The aggregation of these single-modal hypergraph models into a multimodal hypergraph model includes: Each unimodal hypergraph model is used as a local model, and federated learning is used to aggregate them to obtain the multimodal hypergraph model. Joint iterative optimization of the unimodal and multimodal hypergraph models is then performed, including: The model parameters of the multimodal hypergraph model are updated based on the model parameters of each unimodal hypergraph model after iterative optimization. The multimodal hypergraph model is iteratively optimized, and the model parameters of each unimodal hypergraph model are updated based on the model parameters of the iteratively optimized multimodal hypergraph model.
2. The method for modeling complex high-order correlations in medical images according to claim 1, characterized in that, The process of inputting the data to be predicted from the downstream clinical task into the optimized unimodal hypergraph model and the multimodal hypergraph model, and outputting the prediction results for unknown samples of the downstream clinical task, includes: Extract the image omics features and depth features of the data to be predicted; Establish a hypergraph model and hypergraph association matrix for the data to be predicted, and then fuse the hypergraph model with the optimized unimodal hypergraph model and the multimodal hypergraph model respectively; The hypergraph correlation matrix, the radiomics features, and the deep features are input into the fused single-modal hypergraph model and the multimodal hypergraph model, and the prediction results of unknown samples for downstream clinical tasks are output.
3. The method for modeling complex high-order correlations in medical images according to any one of claims 1-2, characterized in that, The unimodal data is input into the corresponding unimodal hypergraph model to generate unimodal complex high-order association representation information, and the unimodal complex high-order association representation information is sent to a classifier or evaluation model for prediction.
4. The method for modeling complex high-order correlations in medical images according to any one of claims 1-2, characterized in that, The multimodal hypergraph model establishes a multimodal hypergraph convolutional layer through the multimodal hypergraph association matrix and feature transformation parameters, generates multimodal complex high-order association representation information, and feeds the multimodal complex high-order association representation information into a classifier or evaluation model for prediction.
5. A multimodal medical image complex high-order correlation modeling device, for implementing the multimodal medical image complex high-order correlation modeling method as described in any one of claims 1-4, characterized in that, include: The extraction module is used to extract radiomics features and depth features from multimodal medical images; The optimization module is used to construct different unimodal hypergraph models based on the image omics features and the depth features, aggregate the unimodal hypergraph models into a multimodal hypergraph model, and perform joint iterative optimization on the unimodal hypergraph model and the multimodal hypergraph model; The output module is used to input the data to be predicted from the downstream clinical task into the optimized unimodal hypergraph model and the multimodal hypergraph model, and output the prediction results of the unknown samples of the downstream clinical task.
6. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the multimodal medical image complex high-order correlation modeling method as described in any one of claims 1-4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the multimodal medical image complex high-order correlation modeling method as described in any one of claims 1-4.