Intelligent identification method and system of carotid artery plaque based on multimodal image fusion

Through multimodal image fusion and attention map convolution network model, the problem of insufficient image registration and generalization capabilities in carotid plaque recognition in the prior art is solved, automatic identification and prediction of plaques are realized, and quantitative Evaluate and visualize reports to improve diagnostic efficiency and accuracy.

CN118799693BActive Publication Date: 2024-11-15DONGGUAN SOUTHEAST CENTRAL HOSPITAL (DONGGUAN SOUTHEAST TRADITIONAL CHINESE MEDICINE MEDICAL SERVICE CENTER DONGGUAN FIRST HOSPITAL AFFILIATED TO GUANGDONG MEDICAL UNIVERSITY)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411295178.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-16
Publication Date
2024-11-15
Estimated Expiration
2044-09-16

AI Technical Summary

Technical Problem

The prior art lacks effective multimodal image registration and fusion methods in carotid plaque recognition, making it difficult to fully utilize the complementary information of different modal images, and the generalization ability of plaque recognition and prediction models is limited, making it difficult to adapt to different patients. and imaging conditions, lack of quantitative evaluation indicators, and it is difficult to provide an objective basis for clinical decision-making.

Method used

By acquiring carotid ultrasound and CT images, preprocessing and registration, extracting multimodal image features, using multimodal attention map convolutional network model for plaque identification and classification, and combining three-dimensional visualization technology to evaluate the danger of plaques. Generate intelligent identification reports for level and future development.

Benefits of technology

It realizes automatic, accurate, comprehensive identification and prediction of carotid plaques, provides intuitive visual results, improves diagnostic efficiency and accuracy, provides intelligent assistance for early diagnosis and treatment plan formulation, and reduces artificial subjective errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799693B_ABST
    Figure CN118799693B_ABST
Patent Text Reader

Abstract

The present invention provides a method and system for intelligent identification of carotid plaques based on multimodal image fusion, which relates to the field of image recognition technology, and includes acquiring carotid ultrasound images and carotid CT images to obtain a multimodal image feature set of carotid plaques; inputting the multimodal image feature set into a pre-trained plaque identification model to identify whether there is a plaque in the multimodal carotid image, and if there is a plaque, inputting the multimodal image feature set of the carotid plaque into a plaque type classification model to classify the type of the plaque; obtaining a plaque development prediction result based on the recognition result of the plaque recognition model, and using three-dimensional visualization technology to fuse the plaque recognition result, type classification result, plaque location, plaque volume, and plaque development prediction result with the original ultrasound image and CT image for display, so as to generate a carotid plaque intelligent identification report.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to image recognition technology, and in particular to a method and system for intelligently identifying carotid artery plaques based on multimodal image fusion. Background Art

[0002] Carotid plaque is one of the main causes of stroke. Early and accurate identification and prediction of the development of carotid plaque is of great significance for timely intervention and reducing the risk of stroke. Traditional carotid plaque identification mainly relies on doctors' naked eye observation and empirical judgment of single-modality medical images, which has problems such as strong subjectivity and poor repeatability, making it difficult to comprehensively and accurately assess the severity and future progression of plaques.

[0003] In recent years, the development of multimodal medical imaging technology and artificial intelligence methods has provided a new approach for the intelligent identification of carotid plaques. Some studies have attempted to fuse the image information of different imaging modalities and establish plaque recognition and prediction models through machine learning algorithms, and have achieved preliminary results. However, the existing methods still have the following shortcomings: First, there is a lack of effective multimodal image registration and fusion methods, which makes it difficult to fully utilize the complementary information provided by different modalities of images; second, the generalization ability of plaque recognition and prediction models is limited, and it is difficult to adapt to image changes under different patients and imaging conditions; third, there is a lack of quantitative evaluation indicators for the severity and development trend of plaques, which makes it difficult to provide an objective and quantitative basis for clinical decision-making.

[0004] Therefore, there is an urgent need for an intelligent identification method for carotid plaques based on multimodal image fusion. Through image registration, feature extraction, machine learning and other technologies, it can achieve automatic, accurate and comprehensive identification and prediction of plaques and provide intuitive visualization results, thereby providing intelligent assistance for early diagnosis, risk assessment, and treatment plan formulation of patients with carotid plaques. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for intelligently identifying carotid plaque based on multimodal image fusion, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provided is a method for intelligently identifying carotid plaque based on multimodal image fusion, comprising:

[0008] Acquire a carotid artery ultrasound image and a carotid artery CT image, preprocess the carotid artery ultrasound image and the carotid artery CT image, register the preprocessed carotid artery ultrasound image and the preprocessed carotid artery CT image to obtain a registered multimodal carotid artery image, extract image features of the registered multimodal carotid artery image, and obtain a multimodal image feature set of carotid artery plaque;

[0009] Inputting the multimodal image feature set into a pre-trained plaque recognition model to identify whether there is a plaque in the multimodal carotid artery image; if there is a plaque, inputting the multimodal image feature set of the carotid artery plaque into a plaque type classification model to classify the type of the plaque;

[0010] According to the recognition results of the plaque recognition model, the three-dimensional spatial position of the plaque in the carotid artery is determined in combination with the multimodal carotid artery images, and the volume of the plaque is calculated. According to the classification results of the plaque classification model, the three-dimensional spatial position of the plaque and the volume of the plaque, the risk level of the plaque is determined. If the risk level of the plaque is greater than the preset risk level threshold, the multimodal imaging feature set of the carotid plaque is input into the preset plaque development prediction model to predict the development of the plaque at different times in the future and obtain the plaque development prediction results. Using three-dimensional visualization technology, the plaque recognition results, type classification results, plaque location, plaque volume, and plaque development prediction results are fused with the original carotid ultrasound images and carotid CT images to generate a carotid plaque intelligent recognition report.

[0011] In an optional embodiment,

[0012] Extracting the image features of the registered multimodal carotid artery image to obtain a multimodal image feature set of carotid artery plaque includes:

[0013] Perform superpixel segmentation on each modality of the registered multimodal carotid artery images, divide the images into multiple locally consistent regions as nodes of the graph, extract the local texture features of each node as the initial feature representation of the node, and use the pre-trained deep convolutional neural network to perform semantic segmentation on the images of each modality, obtain pixel-level anatomical structure annotations as the semantic attributes of the nodes, calculate the attention weights of nodes between different modalities, consider the feature similarity and semantic consistency of nodes in different modalities, build cross-modal edge connections, and within each modality, build intra-modal edge connections based on the spatial adjacency and feature similarity between nodes to obtain the final multimodal image graph structure;

[0014] A multimodal attention graph convolutional network model is constructed. The constructed multimodal image graph structure is used as input. By introducing the modal attention mechanism and the cross-scale feature fusion strategy, the importance weights of different modalities are adaptively learned. The modal attention mechanism dynamically adjusts the weights of graph convolution by calculating the correlation between nodes of different modalities. The cross-scale feature fusion strategy transfers and interacts features between different convolutional layers through upsampling and downsampling operations to extract multi-scale semantic information.

[0015] In the multimodal attention graph convolutional network model, a graph attention pooling layer is set to adaptively compress the graph structure and aggregate features by learning the importance weights of nodes. At the same time, a multi-head attention module is designed to perform weighted fusion on graph-level features extracted from different modalities to obtain multimodal fusion features.

[0016] An end-to-end multi-task learning method is adopted, with patch segmentation and recognition as supervision information, to optimize the parameters of the multimodal attention graph convolutional network model and obtain a trained multimodal attention graph convolutional network model;

[0017] The multimodal carotid artery images to be analyzed are input into the trained multimodal attention graph convolutional network model to automatically extract multimodal fusion features and obtain the multimodal image feature set of carotid artery plaques.

[0018] In an optional embodiment,

[0019] An end-to-end multi-task learning method is adopted, with patch segmentation and recognition as supervision information, to optimize the parameters of the multimodal attention graph convolutional network model. The trained multimodal attention graph convolutional network model includes:

[0020] Self-supervised learning loss is introduced, and a joint loss function is constructed by combining multi-task learning loss. The model parameters are updated by minimizing the joint loss function, and the iteration is repeated until the preset number of iterations is met, and finally a trained multimodal attention graph convolutional network model is obtained;

[0021] The joint loss function formula is as follows:

[0022] ;

[0023] Among them, L represents the joint loss function, λ1 represents the weight coefficient of patch segmentation loss, M represents the total number of pixels, and p m represents the probability that the mth pixel predicted by the model belongs to a patch, g m represents the true label of the m-th pixel, λ2 represents the patch recognition loss, C represents the number of patch categories, and w c represents the weight of the cth class, y c Indicates the true label that the patch belongs to the cth class, represents the probability that the model predicts that the sample belongs to the cth class, λ3 represents the weight coefficient of the self-supervised learning loss, N represents the number of nodes, and s ij represents the similarity between node i and node j, d(·) represents the distance metric function, and h i represents the hidden representation of node i, h j represents the hidden representation of node j.

[0024] In an optional embodiment,

[0025] The multimodal image feature set is input into the pre-trained plaque recognition model to identify whether there is a plaque in the multimodal carotid artery image. If there is a plaque, the multimodal image feature set of the carotid artery plaque is input into the plaque type classification model to classify the type of plaque, including:

[0026] The acquired carotid artery multimodal image data are preprocessed, and a hybrid image segmentation algorithm combining the active contour model and the level set method is used to automatically extract the region of interest from the preprocessed multimodal image data;

[0027] Extracting multiple types of image features from the region of interest, the image features include morphological features, first-order texture features, second-order texture features and grayscale statistical features, and optimizing the extracted image features using a two-stage feature selection strategy. In the first stage, a feature sorting method based on maximum and minimum distances is used to preliminarily screen out a first feature subset. In the second stage, a redundant feature elimination method based on correlation coefficients is used to eliminate redundant information in the first feature subset to obtain a target feature subset.

[0028] The target feature subset is input into the plaque recognition model, which consists of a cascade classifier, a convolutional neural network and a support vector machine. The input target feature subset is screened by the cascade classifier, divided into multiple candidate areas and passed to the convolutional neural network and the support vector machine for parallel feature learning and classification. At the same time, the weight coefficients of the cascade classifier, the convolutional neural network and the support vector machine are dynamically adjusted, and the final plaque recognition result is obtained through a weighted fusion strategy, and the target plaque area is output;

[0029] Extract local and global multi-scale feature descriptors of the target plaque area, construct a multi-level plaque type representation, input the multi-level plaque type representation into a plaque type classification model that combines a multi-classification support vector machine and a deep belief network, and use the multi-classification support vector machine to classify the plaque type in a coarse-grained manner, and preliminarily classify the plaques into lipid plaques, fibrous plaques, and calcified plaques. At the same time, use the deep belief network to learn the discriminative features of the plaques through multiple hidden layers, and perform multi-classification prediction through the output layer;

[0030] The classification prediction results of multiple classification support vector machines are fused to obtain the final plaque type classification result.

[0031] In an optional embodiment,

[0032] According to the recognition results of the plaque recognition model, the three-dimensional spatial position of the plaque in the carotid artery is determined in combination with multimodal carotid artery images, and the volume of the plaque is calculated, including:

[0033] The registered multimodal carotid artery images are input into a pre-trained multi-scale 3D convolutional neural network. The local and global features of the multimodal images are extracted through multi-scale 3D convolutional layers and pooling layers. The gated attention mechanism is used to perform adaptive weighted fusion of features of different scales and modalities to obtain a fused multi-scale 3D feature map.

[0034] The fused multi-scale 3D feature map is input into the pre-trained convolutional neural network, the structural features of the patch area are extracted through the convolutional block and transition layer, and the patch area is classified and edge optimized at the pixel level using the conditional random field to obtain the segmented 3D patch mask;

[0035] The segmented three-dimensional plaque mask is superimposed on the original multimodal carotid artery image to obtain the three-dimensional spatial position information of the plaque area. The plaque area is reconstructed in three dimensions using a ray casting algorithm to obtain a three-dimensional model of the plaque. The reconstructed three-dimensional plaque model is then subjected to morphological analysis and hemodynamic simulation to calculate the volume of the plaque.

[0036] In an optional embodiment,

[0037] The multimodal imaging feature set of carotid artery plaques is input into the preset plaque development prediction model to predict the development of plaques at different times in the future. The plaque development prediction results obtained include:

[0038] Preprocess the multimodal imaging features in the multimodal imaging feature set of carotid plaques to extract standardized imaging omics features;

[0039] Construct a medical knowledge graph in the field of plaques, which includes the dimensions of anatomical structure, physiological function and pathological manifestation, explore the intrinsic relationship between plaque imaging features and clinical phenotypes through knowledge reasoning and semantic association, and construct a plaque imaging-clinical association knowledge base, which includes the association knowledge between plaque morphological features, plaque density features, plaque texture features, plaque structural features and clinical prognosis, and mine the association rules between plaque imaging features and clinical prognosis from historical case data;

[0040] The plaque radiomics features are integrated with the plaque field medical knowledge graph and the structured knowledge representation in the plaque radiomics-clinical association knowledge base. The image-knowledge complementary information is mined from the plaque radiomics features and structured knowledge representation through multi-view feature learning methods to obtain the plaque image-knowledge fusion features.

[0041] Constructing a plaque development prediction model, inputting the plaque image-knowledge fusion features into the plaque development prediction model, predicting the development of the current plaque at different time points in the future through a time series prediction algorithm, and obtaining a plaque development prediction result;

[0042] The plaque development prediction result is visualized and semantically interpreted according to the medical knowledge graph in the plaque field to obtain the plaque development prediction result.

[0043] According to a second aspect of the embodiments of the present invention,

[0044] Provided is a carotid artery plaque intelligent recognition system based on multimodal image fusion, comprising:

[0045] The first unit is used to obtain a carotid ultrasound image and a carotid CT image, preprocess the carotid ultrasound image and the carotid CT image, register the preprocessed carotid ultrasound image and the preprocessed carotid CT image to obtain a registered multimodal carotid image, extract image features of the registered multimodal carotid image, and obtain a multimodal image feature set of carotid plaque;

[0046] The second unit is used to input the multimodal image feature set into a pre-trained plaque recognition model to identify whether there is a plaque in the multimodal carotid artery image. If there is a plaque, the multimodal image feature set of the carotid artery plaque is input into a plaque type classification model to classify the type of the plaque;

[0047] The third unit is used to determine the three-dimensional spatial position of the plaque in the carotid artery based on the recognition result of the plaque recognition model and the multimodal carotid artery image, and calculate the volume of the plaque; determine the danger level of the plaque based on the classification result of the plaque classification model, the three-dimensional spatial position of the plaque and the volume of the plaque; if the danger level of the plaque is greater than the preset danger level threshold, input the multimodal imaging feature set of the carotid plaque into the preset plaque development prediction model to predict the development of the plaque at different times in the future, and obtain the plaque development prediction result; use three-dimensional visualization technology to fuse the plaque recognition result, type classification result, plaque position, plaque volume, and plaque development prediction result with the original carotid artery ultrasound image and carotid artery CT image for display, and generate a carotid plaque intelligent recognition report.

[0048] According to a third aspect of the embodiments of the present invention,

[0049] An electronic device is provided, comprising:

[0050] processor;

[0051] a memory for storing processor-executable instructions;

[0052] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0053] A fourth aspect of the embodiments of the present invention is:

[0054] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0055] In this embodiment, a multimodal carotid artery image is generated by preprocessing and registering the carotid artery ultrasound image and the carotid artery CT image. Ultrasound images can provide soft tissue and blood flow information, and CT images can provide clearer anatomical structures. By registering and integrating the advantages of two different modalities, the carotid artery structure and its lesions can be observed more comprehensively and accurately. Through the pre-trained plaque recognition model, plaques in the carotid artery can be automatically identified, which greatly improves the efficiency and accuracy of diagnosis and avoids the subjective errors of manual screening. Based on the multimodal image feature set, the plaque classification model can accurately classify the identified plaque types. According to the registration results of the multimodal images, the three-dimensional spatial position of the plaque in the carotid artery can be accurately located, and the plaque volume can be accurately calculated. By combining the classification results, three-dimensional position and volume of the plaque, the risk level of the plaque can be automatically evaluated. Doctors can judge the patient's disease progression based on these data and formulate targeted treatment plans, especially early intervention for high-risk plaques. Through the plaque development prediction model, the development of plaques at different times in the future can be predicted to help doctors monitor and evaluate the patient's long-term risks. This is of great significance for preventive intervention and personalized treatment. Through a series of intelligent means such as multimodal image fusion, automatic detection and classification of plaques, three-dimensional visualization, and development prediction, comprehensive analysis and management of carotid artery plaques have been achieved, greatly improving the accuracy, efficiency and clinical value of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of a method for intelligently identifying carotid plaques based on multimodal image fusion according to an embodiment of the present invention;

[0057] Figure 2 It is a structural schematic diagram of a carotid artery plaque intelligent recognition system based on multimodal image fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0060] Figure 1 FIG. 1 is a flow chart of an intelligent method for identifying carotid plaques based on multimodal image fusion according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0061] S101. Acquire a carotid artery ultrasound image and a carotid artery CT image, preprocess the carotid artery ultrasound image and the carotid artery CT image, align the preprocessed carotid artery ultrasound image and the preprocessed carotid artery CT image to obtain a registered multimodal carotid artery image, extract image features of the registered multimodal carotid artery image, and obtain a multimodal image feature set of carotid artery plaques.

[0062] Among them, ultrasound imaging and CT imaging are used to image the carotid artery to obtain multimodal imaging data of the carotid artery. Ultrasound imaging uses a high-frequency linear array probe to obtain B-mode ultrasound images and color Doppler ultrasound images of the cross-section and longitudinal section of the carotid artery. CT imaging uses a 320-row spiral CT to obtain plain scan CT images and CT angiography images of the carotid artery.

[0063] In order to improve the accuracy of subsequent image registration and feature extraction, the acquired multimodal images need to be preprocessed. For B-mode ultrasound images, an adaptive threshold segmentation algorithm is used to extract the carotid intima-media region, and the carotid cross-sectional and longitudinal cross-sectional images are obtained through morphological processing; color Doppler ultrasound images are segmented by chromaticity threshold to extract the blood flow signal area. For plain scan CT images, the carotid region is extracted by Hounsfield unit threshold segmentation, and the cross-sectional and longitudinal cross-sectional anatomical structure images are reconstructed; image subtraction and maximum density projection are performed on CT angiography images to obtain the vascular cavity images of the cross-sectional and longitudinal sections of the carotid artery.

[0064] Due to the differences between ultrasound imaging and CT imaging in imaging mechanism, resolution, imaging range, etc., in order to achieve fusion recognition of multimodal images, it is necessary to register the carotid artery images of different modalities. A multi-scale registration algorithm based on mutual information was used to register the preprocessed B-mode ultrasound images, color Doppler ultrasound images, plain scan CT images, and CT angiography images in the cross-section and longitudinal sections, respectively, to obtain the registered multi-modal carotid artery cross-section images and longitudinal section images. The algorithm achieves registration by maximizing the mutual information metric between images of different modalities, and adopts a multi-scale strategy to improve the registration speed and robustness.

[0065] In the registered multimodal carotid images, extracting imaging features that can reflect the characteristics of plaques is the basis for plaque recognition and classification. A variety of image feature descriptors are used to extract features of carotid plaques, including multi-scale local binary pattern (LBP) features, gray-level co-occurrence matrix (GLCM) features, Gabor wavelet features, and deep convolutional neural network (CNN) features. Among them, LBP features extract local texture information of plaques, GLCM features extract second-order texture statistics of plaques, Gabor wavelet features extract multi-scale and multi-directional texture information of plaques, and CNN features automatically learn deep feature representations of plaques through convolutional neural networks. After feature extraction, the maximum relevance minimum redundancy algorithm and deep feature selection network algorithm are used to select the extracted multimodal image features to obtain a multimodal image feature set of carotid plaques.

[0066] In an optional embodiment,

[0067] Extracting the image features of the registered multimodal carotid artery image to obtain a multimodal image feature set of carotid artery plaque includes:

[0068] Perform superpixel segmentation on each modality of the registered multimodal carotid artery images, divide the images into multiple locally consistent regions as nodes of the graph, extract the local texture features of each node as the initial feature representation of the node, and use the pre-trained deep convolutional neural network to perform semantic segmentation on the images of each modality, obtain pixel-level anatomical structure annotations as the semantic attributes of the nodes, calculate the attention weights of nodes between different modalities, consider the feature similarity and semantic consistency of nodes in different modalities, build cross-modal edge connections, and within each modality, build intra-modal edge connections based on the spatial adjacency and feature similarity between nodes to obtain the final multimodal image graph structure;

[0069] A multimodal attention graph convolutional network model is constructed. The constructed multimodal image graph structure is used as input. By introducing the modal attention mechanism and the cross-scale feature fusion strategy, the importance weights of different modalities are adaptively learned. The modal attention mechanism dynamically adjusts the weights of graph convolution by calculating the correlation between nodes of different modalities. The cross-scale feature fusion strategy transfers and interacts features between different convolutional layers through upsampling and downsampling operations to extract multi-scale semantic information.

[0070] In the multimodal attention graph convolutional network model, a graph attention pooling layer is set to adaptively compress the graph structure and aggregate features by learning the importance weights of nodes. At the same time, a multi-head attention module is designed to perform weighted fusion on graph-level features extracted from different modalities to obtain multimodal fusion features.

[0071] An end-to-end multi-task learning method is adopted, with patch segmentation and recognition as supervision information, to optimize the parameters of the multimodal attention graph convolutional network model and obtain a trained multimodal attention graph convolutional network model;

[0072] The multimodal carotid artery images to be analyzed are input into the trained multimodal attention graph convolutional network model to automatically extract multimodal fusion features and obtain the multimodal image feature set of carotid artery plaques.

[0073] Exemplarily, the registered multimodal carotid artery images are first preprocessed and the graph structure is constructed. Specifically, the superpixel segmentation algorithm is used for each modality image to divide the image into multiple locally consistent and semantically related regions as nodes of the graph structure. Superpixel segmentation can effectively reduce the number of nodes in the graph and improve the efficiency of subsequent feature extraction by clustering pixel points to generate compact local regions with regular boundaries.

[0074] When extracting the initial feature representation of the node, the local texture features and semantic attributes are comprehensively considered. For the local texture features, multi-scale and multi-directional texture descriptors such as Gabor filters are used to characterize the texture pattern and edge information in the node area. At the same time, in order to obtain the semantic attributes of the node, the pre-trained deep convolutional neural network, such as ResNet, is used to perform semantic segmentation on the image of each modality to obtain pixel-level anatomical structure annotations, such as lumen, wall, plaque, etc., which are used as the semantic labels of the nodes.

[0075] When constructing the edge connections of the graph, the complementarity and correlation of multimodal images are fully utilized. For nodes between different modalities, by calculating their similarities in feature space and semantic space, corresponding attention weights are assigned to construct cross-modal edge connections. Specifically, cosine similarity is used to measure the similarity between node feature vectors, and Jaccard coefficient is used to measure the consistency of node semantic labels. The weighted combination of the two is used as the weight of the cross-modal edge. This attention-weighted cross-modal edge connection can effectively model the semantic correspondence between different modalities and promote subsequent cross-modal feature fusion.

[0076] For nodes within the same modality, their spatial adjacency and feature similarity are comprehensively considered to construct edge connections within the modality. Adjacent nodes have local consistency in space, and nodes with similar features also have certain semantic correlation. Therefore, the spatial adjacency matrix and the feature similarity matrix are Hadamard-producted to obtain the weight matrix of the intra-modal edges. Through the above steps, a multimodal image graph structure that integrates local texture, semantic attributes, and cross-modal correlation can be obtained. This graph structure can comprehensively characterize the anatomical structure and pathological characteristics of carotid artery images, providing rich prior information for subsequent feature learning.

[0077] Based on the constructed multimodal image graph structure, a multimodal attention graph convolutional network (MA-GCN) model is proposed to adaptively extract multimodal fusion features. Compared with traditional graph convolutional networks, MA-GCN introduces a modal attention mechanism and a cross-scale feature fusion strategy, which can dynamically adjust the convolution weights according to the importance of different modalities and capture multi-scale semantic information.

[0078] In the MA-GCN model, each graph convolution layer contains multiple modality-specific convolution kernels. For each modality, the importance of the convolution kernel is dynamically adjusted by calculating the attention weight between the node features and the convolution kernel to highlight the contribution of the significant modality. This modality attention mechanism enables MA-GCN to adaptively fuse complementary information from different modalities and improve the robustness of feature representation.

[0079] At the same time, in order to capture multi-scale semantic features, MA-GCN adopts a cross-scale feature fusion strategy. Between adjacent graph convolutional layers, upsampling and downsampling operations are used to achieve the transmission and interaction of feature maps of different scales. Upsampling expands the size of feature maps through interpolation, and downsampling reduces the size of feature maps through pooling, thereby extracting and fusing semantic information at different levels of abstraction. Cross-scale feature fusion enhances the ability of MA-GCN to extract multi-scale and multi-level features, enabling it to comprehensively characterize the anatomical structure and pathological pattern of carotid artery images.

[0080] At the top layer of MA-GCN, a graph attention pooling layer is set to adaptively compress the graph structure and aggregate features by learning the importance weights of nodes. The graph attention pooling layer uses a self-attention mechanism to generate a weight coefficient for each node based on the similarity and significance of node features, and performs a weighted summation of node features to obtain a compact feature representation at the graph level. This pooling operation can effectively filter redundant information, extract features of key nodes, and improve the discriminability of features.

[0081] Finally, in order to obtain the global features of multimodal fusion, MA-GCN uses a multi-head attention module to perform weighted combination of graph-level features extracted from different modalities. Multi-head attention calculates the correlation between features of different modalities, generates multiple sets of attention weights, and concatenates the weighted features to obtain the final multimodal fusion feature vector. The multi-head attention mechanism enhances the ability of MA-GCN to process multimodal information, enabling it to flexibly adapt to the semantic associations between different modalities and improve the effectiveness of feature fusion.

[0082] In the model training phase, an end-to-end multi-task learning method is adopted, with patch segmentation and recognition as supervision information, to jointly optimize the parameters of MA-GCN. Multi-task learning promotes knowledge transfer and feature enhancement between different tasks by sharing feature extraction networks and learning multiple related tasks, such as segmentation and recognition. This end-to-end training method enables MA-GCN to take into account both local details and global semantics, and learn more robust and discriminative multimodal fusion features.

[0083] In an optional embodiment,

[0084] An end-to-end multi-task learning method is adopted, with patch segmentation and recognition as supervision information, to optimize the parameters of the multimodal attention graph convolutional network model. The trained multimodal attention graph convolutional network model includes:

[0085] Self-supervised learning loss is introduced, and a joint loss function is constructed by combining multi-task learning loss. The model parameters are updated by minimizing the joint loss function, and the iteration is repeated until the preset number of iterations is met, and finally a trained multimodal attention graph convolutional network model is obtained;

[0086] The joint loss function formula is as follows:

[0087] ;

[0088] Among them, L represents the joint loss function, λ1 represents the weight coefficient of patch segmentation loss, M represents the total number of pixels, and p m represents the probability that the mth pixel predicted by the model belongs to a patch, g m represents the true label of the m-th pixel, λ2 represents the patch recognition loss, C represents the number of patch categories, and w c represents the weight of the cth class, y c Indicates the true label that the patch belongs to the cth class, represents the probability that the model predicts that the sample belongs to the cth class, λ3 represents the weight coefficient of the self-supervised learning loss, N represents the number of nodes, and s ij represents the similarity between node i and node j, d(·) represents the distance metric function, and h i represents the hidden representation of node i, hj represents the hidden representation of node j.

[0089] In this embodiment, the fine segmentation of carotid artery images is achieved by combining superpixel segmentation and semantic segmentation. Superpixel segmentation divides the image into locally consistent areas, which helps to extract the boundaries of plaques more accurately. At the same time, semantic segmentation provides pixel-level anatomical structure annotation, making the location and identification of plaques more accurate. By constructing a cross-modal graph structure and introducing a modal attention mechanism and a cross-scale feature fusion strategy, the multimodal information of the carotid artery image is effectively fused. The modal attention mechanism adaptively adjusts the convolution weights of each modality by calculating the correlation between nodes to ensure that important information is focused on during the fusion process. At the same time, the cross-scale feature fusion strategy can extract semantic information at different scales through feature transfer and interaction between different convolutional layers, ensuring that details and overall features are taken into account. An end-to-end multi-task learning method is adopted to jointly optimize the parameters of the model through the supervision information of plaque segmentation and identification. This multi-task learning mechanism ensures the close correlation between the plaque segmentation and identification processes. The model can simultaneously learn the spatial distribution characteristics and classification information of the plaque, greatly improving the synergy of the tasks and the accuracy of the final model. By innovatively introducing multimodal graph convolutional networks, attention mechanisms, and cross-scale feature fusion strategies, efficient segmentation, recognition, and feature extraction of multimodal carotid artery images are achieved, improving the accuracy and intelligence of plaque analysis.

[0090] S102. Input the multimodal image feature set into a pre-trained plaque recognition model to identify whether there is a plaque in the multimodal carotid artery image. If there is a plaque, input the multimodal image feature set of the carotid artery plaque into a plaque type classification model to classify the type of plaque.

[0091] In an optional embodiment,

[0092] The multimodal image feature set is input into the pre-trained plaque recognition model to identify whether there is a plaque in the multimodal carotid artery image. If there is a plaque, the multimodal image feature set of the carotid artery plaque is input into the plaque type classification model to classify the type of plaque, including:

[0093] The acquired carotid artery multimodal image data are preprocessed, and a hybrid image segmentation algorithm combining the active contour model and the level set method is used to automatically extract the region of interest from the preprocessed multimodal image data;

[0094] Extracting multiple types of image features from the region of interest, the image features include morphological features, first-order texture features, second-order texture features and grayscale statistical features, and optimizing the extracted image features using a two-stage feature selection strategy. In the first stage, a feature sorting method based on maximum and minimum distances is used to preliminarily screen out a first feature subset. In the second stage, a redundant feature elimination method based on correlation coefficients is used to eliminate redundant information in the first feature subset to obtain a target feature subset.

[0095] The target feature subset is input into the plaque recognition model, which consists of a cascade classifier, a convolutional neural network and a support vector machine. The input target feature subset is screened by the cascade classifier, divided into multiple candidate areas and passed to the convolutional neural network and the support vector machine for parallel feature learning and classification. At the same time, the weight coefficients of the cascade classifier, the convolutional neural network and the support vector machine are dynamically adjusted, and the final plaque recognition result is obtained through a weighted fusion strategy, and the target plaque area is output;

[0096] Extract local and global multi-scale feature descriptors of the target plaque area, construct a multi-level plaque type representation, input the multi-level plaque type representation into a plaque type classification model that combines a multi-classification support vector machine and a deep belief network, and use the multi-classification support vector machine to classify the plaque type in a coarse-grained manner, and preliminarily classify the plaques into lipid plaques, fibrous plaques, and calcified plaques. At the same time, use the deep belief network to learn the discriminative features of the plaques through multiple hidden layers, and perform multi-classification prediction through the output layer;

[0097] The classification prediction results of multiple classification support vector machines are fused to obtain the final plaque type classification result.

[0098] Exemplarily, the acquired multimodal image data of the carotid artery are first preprocessed, and a hybrid image segmentation algorithm combining the active contour model and the level set method is used to automatically extract the region of interest from the preprocessed multimodal image data, and the segmentation result is optimized through morphological post-processing operations. Among them, the active contour model defines a dynamically changing closed curve, so that it evolves continuously and finally converges to the boundary of the region of interest under the drive of the image gradient and the energy of the inner and outer regions. The level set method implicitly represents the curve as a zero level set in a high-dimensional space, and realizes the evolution and segmentation of the curve by evolving the level set function. The hybrid image segmentation algorithm combines the global constraints of the active contour model and the topological adaptability of the level set method, which can effectively overcome the problems of local optimality and sensitivity to initialization of traditional methods, and realize the robust and accurate segmentation of the carotid artery vascular region. Morphological post-processing operations include morphological opening and closing operations, connected domain analysis, etc., which are used to eliminate small areas in the segmentation results, smooth boundaries and fill holes, and further improve the accuracy and completeness of segmentation.

[0099] After obtaining the region of interest, various types of image features are extracted from it, including morphological features, first-order texture features, second-order texture features, and grayscale statistical features. Morphological features reflect the geometric shape and size information of the patch area, such as area, perimeter, compactness, etc. The first-order texture feature describes its spatial distribution pattern by calculating the statistics of the grayscale of the patch area. Commonly used features include grayscale co-occurrence matrix, local binary pattern, etc. The second-order texture feature considers the correlation of the grayscale of the patch area in different directions and scales, and uses Gabor wavelet transform to extract the multi-scale and multi-directional texture information of the patch. Grayscale statistical features describe the grayscale distribution of the patch area from the perspective of histogram, including mean, variance, skewness, kurtosis, and entropy.

[0100] In order to select the most discriminative and informative feature subset from numerous image features, a two-stage feature selection strategy was used to optimize the extracted image features. In the first stage, a feature sorting method based on maximum and minimum distance was used to preliminarily screen out a feature subset with strong discriminativeness. This method evaluates its discriminative ability by calculating the maximum and minimum distance between each feature and the category. The larger the distance, the higher the feature's discrimination. By setting a distance threshold, features with a distance greater than the threshold were selected to form the first feature subset. In the second stage, a redundant feature elimination method based on correlation coefficient was used to eliminate redundant information in the first feature subset. This method calculated the Pearson correlation coefficient between any two features in the feature subset. When the correlation coefficient was greater than the set threshold, the two features were considered to have a strong correlation and contain redundant information. By setting a correlation threshold, redundant features with large correlation were gradually eliminated, and finally a target feature subset with both high discriminative power and independence was obtained.

[0101] The target feature subset is input into the plaque recognition model to locate and identify the plaque area. The plaque recognition model consists of three parts: cascade classifier, convolutional neural network and support vector machine. The input target feature subset is preliminarily screened by the cascade classifier to divide the image into multiple candidate areas. The cascade classifier is composed of multiple simple Adaboost classifiers in series. Each level of classifier further screens the candidate area according to the classification results of the previous level, gradually eliminates non-plaque areas, and narrows the scope of the candidate area. The screened candidate areas are passed to the convolutional neural network and support vector machine for parallel feature learning and classification discrimination. The convolutional neural network automatically learns the high-level and abstract feature representation of the candidate area through multi-layer convolution and pooling operations, and maps the learned features to the plaque category through the fully connected layer. The support vector machine divides the candidate area into two categories: plaque and non-plaque by finding the maximum margin hyperplane. At the same time, according to the complexity of the candidate area, the weight coefficients of the three models are dynamically adjusted. For relatively simple areas, the cascade classifier is given a larger weight, while for more complex areas, the convolutional neural network and support vector machine are given a larger weight. Finally, the outputs of the three models are weighted and fused to obtain the final patch recognition result, and the target patch area is output.

[0102] For the identified target patch area, its local and global multi-scale feature descriptors are further extracted to construct a multi-level patch type representation. The local feature descriptor characterizes the local characteristics of the patch by extracting information such as the local texture, edge and shape of the patch area. For example, by extracting the local binary pattern (LBP) feature of the patch area, the micro-texture information of the patch surface can be effectively captured. The global feature descriptor characterizes the morphology, structure and grayscale distribution of the patch as a whole. For example, by calculating the invariant moments feature of the patch area, the overall shape and symmetry of the patch can be described. The multi-scale feature descriptor extracts the feature information of the patch at different scales, captures the feature changes of the patch at different scales, and enhances the robustness and adaptability of the feature. By fusing local, global and multi-scale feature descriptors, a multi-level feature representation of the patch is constructed to comprehensively characterize the feature information of the patch.

[0103] The multi-level plaque type representation is input into the plaque type classification model that combines a multi-classification support vector machine and a deep belief network to identify and classify the plaque type. The multi-classification support vector machine achieves multi-category classification tasks by constructing multiple binary support vector machines. For each category, a binary support vector machine is trained, and the category is regarded as the positive class and the remaining categories are regarded as negative classes. By finding the optimal classification hyperplane, the samples are divided into positive and negative categories. In the prediction stage, the samples to be classified are input into all binary support vector machines, and the final category of the samples is determined by voting based on the output results of each support vector machine. Through the multi-classification support vector machine, coarse-grained classification of plaque types can be achieved, and plaques can be preliminarily divided into major categories such as lipid plaques, fibrous plaques, and calcified plaques.

[0104] In order to further improve the accuracy and granularity of plaque type classification, a deep belief network is introduced to perform fine-grained classification of plaque types. The deep belief network consists of multiple hidden layers, which learn the discriminative features between plaque types through layer-by-layer feature transformation and nonlinear mapping. The input of the network is a multi-level feature representation of the plaque. The input signal is transmitted to each hidden layer through forward propagation to extract the deep features of the plaque. In each hidden layer, nonlinear factors are introduced through the activation function to enhance the expression ability of the network. The output layer of the network uses the Softmax function for multi-classification prediction, and the plaques are subdivided into subclasses, such as vulnerable plaques and stable plaques in lipid plaques, thick-walled plaques and thin-walled plaques in fibrous plaques, etc. Through the hierarchical feature learning and classification prediction of the deep belief network, the type of plaque can be identified more accurately and fine-grained.

[0105] Finally, the classification prediction results of multiple multi-classification support vector machines and deep belief networks are fused by weighted voting to obtain the final plaque type classification results. For each plaque to be classified, it is input into all classifiers to obtain the prediction results of each classifier. According to the performance and credibility of the classifier, a weight is assigned to each classifier, and the prediction results of the classifier are weighted. The weighted prediction results are voted to obtain the probability distribution of the category to which the plaque belongs. The category with the highest probability is selected as the final type of the plaque. By fusing the prediction results of multiple classifiers by weighted voting, the accuracy and robustness of plaque type classification can be effectively improved.

[0106] In this embodiment, by combining the hybrid image segmentation algorithm of the active contour model and the level set method, the region of interest can be automatically and accurately extracted from the preprocessed multimodal image data, and various types of image features can be extracted from the region of interest. These features cover multi-dimensional information such as the geometric shape, texture complexity, and grayscale distribution of the image, providing a rich feature basis for subsequent plaque recognition and classification, and improving the model's ability to distinguish plaques. By using a two-stage step-by-step optimization strategy, the efficiency of feature selection is improved, feature redundancy is reduced, and the speed and accuracy of model training are improved. The combination strategy of the plaque recognition model can give full play to the advantages of each model and improve the accuracy and robustness of plaque recognition. The full-process technology of multimodal image processing, feature extraction, model optimization and classification based on this technical solution realizes the accurate recognition and classification of carotid artery plaques. Through automation, multi-model fusion, feature optimization and multi-layer fusion of classification results, the efficiency of plaque recognition and the accuracy of classification are improved.

[0107] S103. According to the recognition result of the plaque recognition model, the three-dimensional spatial position of the plaque in the carotid artery is determined in combination with the multimodal carotid artery image, and the volume of the plaque is calculated; according to the classification result of the plaque classification model, the three-dimensional spatial position of the plaque and the volume of the plaque, the danger level of the plaque is determined; if the danger level of the plaque is greater than the preset danger level threshold, the multimodal imaging feature set of the carotid plaque is input into the preset plaque development prediction model to predict the development of the plaque at different times in the future, and obtain the plaque development prediction result; using three-dimensional visualization technology, the plaque recognition result, type classification result, plaque position, plaque volume, plaque development prediction result and the original carotid artery ultrasound image and carotid artery CT image are fused and displayed to generate a carotid plaque intelligent recognition report.

[0108] In an optional embodiment,

[0109] According to the recognition results of the plaque recognition model, the three-dimensional spatial position of the plaque in the carotid artery is determined in combination with multimodal carotid artery images, and the volume of the plaque is calculated, including:

[0110] The registered multimodal carotid artery images are input into a pre-trained multi-scale 3D convolutional neural network. The local and global features of the multimodal images are extracted through multi-scale 3D convolutional layers and pooling layers. The gated attention mechanism is used to perform adaptive weighted fusion of features of different scales and modalities to obtain a fused multi-scale 3D feature map.

[0111] The fused multi-scale 3D feature map is input into the pre-trained convolutional neural network, the structural features of the patch area are extracted through the convolutional block and transition layer, and the patch area is classified and edge optimized at the pixel level using the conditional random field to obtain the segmented 3D patch mask;

[0112] The segmented three-dimensional plaque mask is superimposed on the original multimodal carotid artery image to obtain the three-dimensional spatial position information of the plaque area. The plaque area is reconstructed in three dimensions using a ray casting algorithm to obtain a three-dimensional model of the plaque. The reconstructed three-dimensional plaque model is then subjected to morphological analysis and hemodynamic simulation to calculate the volume of the plaque.

[0113] Exemplarily, first, the acquired multimodal carotid artery image data is preprocessed, including steps such as image denoising, correction and registration, to eliminate the noise, distortion and spatial misalignment problems between different modal images, and obtain a registered multimodal carotid artery image dataset. Then, the registered multimodal carotid artery image dataset is input into a pre-trained multi-scale three-dimensional convolutional neural network for feature extraction. The network consists of multiple three-dimensional convolutional layers and pooling layers. By setting different convolution kernel sizes and pooling windows, local and global image features can be extracted at different scales. At the same time, in order to effectively fuse features of different modalities and scales, a gated attention mechanism is introduced into the network. By learning the importance weight of each feature channel, the contribution of different features is adaptively adjusted to obtain a fused multi-scale three-dimensional feature map.

[0114] Next, the fused multi-scale 3D feature map is input into another pre-trained convolutional neural network to segment the patch area. The network adopts an encoder-decoder architecture. The encoder part consists of multiple convolutional blocks and transition layers to extract high-level semantic features of the patch area; the decoder part gradually restores the spatial resolution of the feature map through upsampling and jump connections, and generates a pixel-level patch segmentation mask. In order to further improve the accuracy and smoothness of the segmentation, a conditional random field is introduced in the output layer of the network for post-processing optimization. By modeling the spatial relationship and label consistency between pixels, the patch edges are refined and smoothed, and finally a segmented 3D patch mask is obtained.

[0115] After obtaining the plaque segmentation mask, it is superimposed with the original multimodal carotid artery image to obtain the position information of the plaque area in three-dimensional space. Then, the ray projection algorithm is used to reconstruct the plaque area in three dimensions. Specifically, according to the spatial position of the plaque mask, the original image is sampled and voxel voted along the projection direction to generate a three-dimensional point cloud and mesh model of the plaque area.

[0116] Finally, the reconstructed three-dimensional plaque model was subjected to morphological analysis and hemodynamic simulation. The size and irregularity of the plaque were quantitatively evaluated by calculating the volume, surface area, shape index and other geometric parameters of the plaque model. At the same time, computational fluid dynamics methods were used to perform numerical simulations of blood flow velocity, pressure and stress on the plaque model to analyze the impact of the plaque on the local hemodynamic environment and predict the risk of plaque rupture.

[0117] In this embodiment, local and global features can be extracted from the registered multimodal carotid artery images through a multiscale three-dimensional convolutional neural network. Convolutional layers and pooling layers of different scales ensure the capture of features from details to the whole, and the gated attention mechanism adaptively integrates features under different modalities and scales, improving the ability to characterize complex plaques. The conditional random field helps to optimize the edge of the plaque, reduce the impact of noise, improve the accuracy of the segmentation results, and ensure the clarity and accuracy of the plaque boundary. The segmented three-dimensional plaque mask is superimposed with the original multimodal image to clearly present the three-dimensional spatial position information of the plaque in the carotid artery. This provides accurate data support for subsequent plaque volume calculation and morphological analysis. The plaque area is reconstructed in three dimensions through a ray casting algorithm, and a high-precision three-dimensional plaque model can be generated. The three-dimensional model intuitively displays the spatial morphology of the plaque, which helps to conduct in-depth analysis of the morphology of the plaque. By performing hemodynamic simulation on the three-dimensional plaque model, the effect of the plaque on blood flow can be understood, and the degree of stenosis of the blood vessel and the blood flow abnormality that may be caused by the plaque can be evaluated. This simulation provides a quantitative analysis of the potential danger of plaques, which is particularly important for assessing whether plaques are likely to cause stroke or other cardiovascular events. In addition, through morphological analysis, the volume of plaques can be accurately calculated, and the quantitative volume information provides an important basis for clinical treatment. Through multi-scale feature extraction, precise plaque segmentation and reconstruction, combined with morphological analysis and hemodynamic simulation, a comprehensive, automated, and refined analysis of carotid artery plaques is achieved, which helps to improve the accuracy and efficiency of clinical diagnosis.

[0118] In an optional embodiment,

[0119] The multimodal imaging feature set of carotid artery plaques is input into the preset plaque development prediction model to predict the development of plaques at different times in the future. The plaque development prediction results obtained include:

[0120] Preprocess the multimodal imaging features in the multimodal imaging feature set of carotid plaques to extract standardized imaging omics features;

[0121] Construct a medical knowledge graph in the field of plaques, which includes the dimensions of anatomical structure, physiological function and pathological manifestation, explore the intrinsic relationship between plaque imaging features and clinical phenotypes through knowledge reasoning and semantic association, and construct a plaque imaging-clinical association knowledge base, which includes the association knowledge between plaque morphological features, plaque density features, plaque texture features, plaque structural features and clinical prognosis, and mine the association rules between plaque imaging features and clinical prognosis from historical case data;

[0122] The plaque radiomics features are integrated with the plaque field medical knowledge graph and the structured knowledge representation in the plaque radiomics-clinical association knowledge base. The image-knowledge complementary information is mined from the plaque radiomics features and structured knowledge representation through multi-view feature learning methods to obtain the plaque image-knowledge fusion features.

[0123] Constructing a plaque development prediction model, inputting the plaque image-knowledge fusion features into the plaque development prediction model, predicting the development of the current plaque at different time points in the future through a time series prediction algorithm, and obtaining a plaque development prediction result;

[0124] The plaque development prediction result is visualized and semantically interpreted according to the medical knowledge graph in the plaque field to obtain the plaque development prediction result.

[0125] For example, first, the multimodal imaging data of carotid plaques are preprocessed, including image denoising, normalization, and feature extraction. Image denoising uses wavelet transform and non-local mean filtering algorithms to remove high-frequency noise and speckle noise in the image; image normalization uses linear stretching and histogram equalization methods to unify the grayscale range and contrast of different modal images to the same scale space; feature extraction uses radiomics methods to extract a series of standardized morphological, density, and texture features from the plaque area to characterize the imaging characteristics of the plaque.

[0126] Then, a medical knowledge graph and an imaging genomics-clinical association knowledge base in the field of plaques are constructed. The medical knowledge graph adopts an ontological approach, with carotid plaques as the core, to organize and represent multi-dimensional knowledge related to plaques, such as anatomical structure, physiological function, and pathological manifestations, and reveals the intrinsic connection between different knowledge through semantic links and logical reasoning. For example, the knowledge graph can be used to infer the location of the carotid artery where the plaque is located, the histological composition of the plaque, and the effect of the plaque on hemodynamics. The imaging genomics-clinical association knowledge base uses data-driven methods such as association rule mining and decision tree learning to automatically discover the correspondence between imaging features and clinical outcomes from large-scale plaque clinical case data. For example, through association analysis, it can be found that the imaging characteristics of plaques, such as irregular shape, low-density lipid core, and rough surface texture, are highly correlated with adverse events such as plaque rupture and embolism.

[0127] After obtaining the radiomics features and structured medical knowledge, they are fused using multi-view feature learning methods to obtain a more comprehensive and integrated plaque feature representation. Specifically, the radiomics features and the structured features in the knowledge graph and associated knowledge base are regarded as plaque representations under different views. Through machine learning methods such as multi-kernel learning and subspace learning, the features under different views are mapped to a shared implicit space. By constraining the consistency and complementarity between different views, the fused plaque features can take into account both imaging information and medical knowledge, reflecting the characteristics of plaques from multiple angles.

[0128] Based on the fused plaque image-knowledge features, a time series prediction model is constructed to predict the development of plaques at different time points in the future. The model uses deep learning architectures such as long short-term memory neural networks to model the dynamic change process of plaques by learning the evolution pattern of plaque features over time. At the same time, the attention mechanism and medical domain knowledge are introduced as priors to constrain and guide the prediction process of the model, thereby improving the accuracy and interpretability of the prediction.

[0129] Finally, the predicted plaque development results are visualized and semantically explained. Through visualization technologies such as three-dimensional reconstruction and virtual endoscopy, the morphological structure, spatial position and other information of the plaque at different time points are intuitively presented. At the same time, the medical knowledge graph is used to reason and interpret the prediction results, automatically generate clinical decision-making recommendations, and assist doctors in qualitative positioning analysis of plaques and formulation of treatment plans.

[0130] For example, for a patient with carotid plaque, first obtain the multimodal imaging data of the carotid artery, such as CT, MRI, and ultrasound, and perform preprocessing such as denoising and normalization on these images. Then, extract a series of quantitative imaging genomics features from the plaque area, such as the volume, shape irregularity, density heterogeneity, texture roughness, etc. of the plaque. Then, using the pre-constructed plaque medical knowledge graph and imaging genomics-clinical association knowledge base, the extracted imaging genomics features are integrated with structured medical concepts and clinical rules to obtain a higher-level semantic feature representation. Then, the fused plaque image-knowledge features are input into the time series prediction model to predict the development of the patient's plaque at different time points in the next 1 year, 3 years, 5 years, etc., and obtain a set of quantitative morphological parameters and qualitative risk assessment results. Finally, through the visualization and semantic interpretation modules, a plaque development prediction report is generated, which intuitively displays the three-dimensional structural changes of the plaque and gives clinical decision-making suggestions, such as "the plaque is currently large in size and irregular in shape, located at the carotid bifurcation, with a complex hemodynamic environment and a high risk of rupture. It is recommended to perform endarterectomy immediately." Through such a complete set of analysis processes, doctors can fully understand the current status and future development of patients' carotid plaques, make personalized diagnosis and treatment decisions, and improve clinical prognosis.

[0131] In this embodiment, the medical knowledge graph in the field of plaques contains multi-dimensional information such as anatomical structure, physiological function and pathological manifestations. Through knowledge reasoning and semantic association, the relationship between plaque imaging features and clinical phenotypes is explored. This structured knowledge graph can help to deeply understand the intrinsic relationship between imaging features and plaque-related clinical manifestations, and expand the depth of understanding of plaque diseases. By constructing a plaque imaging genomics-clinical association knowledge base and combining historical case data, the imaging features of plaques can be mined, which not only helps to identify high-risk plaques, but also can provide strong support for clinical decision-making of plaques and improve the predictive ability of plaque prognosis. Through the multi-view feature learning method, the imaging genomics features are fused with the structured knowledge in the knowledge graph, and the complementary information of images and knowledge can be mined. This fusion feature contains both the representation information in the image and the semantic information of the domain knowledge, which improves the expression ability of plaque features and helps to more accurately describe the complexity and diversity of plaques. Through multimodal fusion of images and knowledge, time series prediction and semantic interpretation, the ability to identify, classify and predict development trends of plaques has been comprehensively improved, providing strong support for personalized treatment and clinical decision-making, and improving the efficiency and accuracy of clinical management.

[0132] Figure 2 FIG. 1 is a schematic diagram of the structure of a carotid plaque intelligent recognition system based on multimodal image fusion according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0133] The first unit is used to obtain a carotid ultrasound image and a carotid CT image, preprocess the carotid ultrasound image and the carotid CT image, register the preprocessed carotid ultrasound image and the preprocessed carotid CT image to obtain a registered multimodal carotid image, extract image features of the registered multimodal carotid image, and obtain a multimodal image feature set of carotid plaque;

[0134] The second unit is used to input the multimodal image feature set into a pre-trained plaque recognition model to identify whether there is a plaque in the multimodal carotid artery image. If there is a plaque, the multimodal image feature set of the carotid artery plaque is input into a plaque type classification model to classify the type of the plaque;

[0135] The third unit is used to determine the three-dimensional spatial position of the plaque in the carotid artery based on the recognition result of the plaque recognition model and the multimodal carotid artery image, and calculate the volume of the plaque; determine the danger level of the plaque based on the classification result of the plaque classification model, the three-dimensional spatial position of the plaque and the volume of the plaque; if the danger level of the plaque is greater than the preset danger level threshold, input the multimodal imaging feature set of the carotid plaque into the preset plaque development prediction model to predict the development of the plaque at different times in the future, and obtain the plaque development prediction result; use three-dimensional visualization technology to fuse the plaque recognition result, type classification result, plaque position, plaque volume, and plaque development prediction result with the original carotid artery ultrasound image and carotid artery CT image for display, and generate a carotid plaque intelligent recognition report.

[0136] According to a third aspect of the embodiments of the present invention,

[0137] An electronic device is provided, comprising:

[0138] processor;

[0139] a memory for storing processor-executable instructions;

[0140] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0141] A fourth aspect of the embodiments of the present invention is:

[0142] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0143] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent identification method for carotid artery plaque based on multimodal image fusion, It is characterized in that include: Acquire a carotid artery ultrasound image and a carotid artery CT image, preprocess the carotid artery ultrasound image and the carotid artery CT image, register the preprocessed carotid artery ultrasound image and the preprocessed carotid artery CT image to obtain a registered multimodal carotid artery image, extract image features of the registered multimodal carotid artery image, and obtain a multimodal image feature set of carotid artery plaque; Inputting the multimodal image feature set into a pre-trained plaque recognition model to identify whether there is a plaque in the multimodal carotid artery image; if there is a plaque, inputting the multimodal image feature set of the carotid artery plaque into a plaque type classification model to classify the type of the plaque; According to the recognition result of the plaque recognition model, the three-dimensional spatial position of the plaque in the carotid artery is determined in combination with the multimodal carotid artery image, and the volume of the plaque is calculated. According to the classification result of the plaque classification model, the three-dimensional spatial position of the plaque and the volume of the plaque, the risk level of the plaque is determined. If the risk level of the plaque is greater than the preset risk level threshold, the multimodal imaging feature set of the carotid plaque is input into the preset plaque development prediction model to predict the development of the plaque at different times in the future, and obtain the plaque development prediction result. The plaque recognition result, type classification result, plaque position, plaque volume, plaque development prediction result and the original carotid artery ultrasound image and carotid artery CT image are fused and displayed by using three-dimensional visualization technology to generate a carotid artery plaque intelligent recognition report; The multimodal image feature set is input into the pre-trained plaque recognition model to identify whether there is a plaque in the multimodal carotid artery image. If there is a plaque, the multimodal image feature set of the carotid artery plaque is input into the plaque type classification model to classify the type of plaque, including: The acquired carotid artery multimodal image data are preprocessed, and a hybrid image segmentation algorithm combining the active contour model and the level set method is used to automatically extract the region of interest from the preprocessed multimodal image data; Extracting multiple types of image features from the region of interest, the image features include morphological features, first-order texture features, second-order texture features and grayscale statistical features, and optimizing the extracted image features using a two-stage feature selection strategy. In the first stage, a feature sorting method based on maximum and minimum distances is used to preliminarily screen out a first feature subset. In the second stage, a redundant feature elimination method based on correlation coefficients is used to eliminate redundant information in the first feature subset to obtain a target feature subset. The target feature subset is input into the plaque recognition model, which consists of a cascade classifier, a convolutional neural network and a support vector machine. The input target feature subset is screened by the cascade classifier, divided into multiple candidate areas and passed to the convolutional neural network and the support vector machine for parallel feature learning and classification. At the same time, the weight coefficients of the cascade classifier, the convolutional neural network and the support vector machine are dynamically adjusted, and the final plaque recognition result is obtained through a weighted fusion strategy, and the target plaque area is output; Extract local and global multi-scale feature descriptors of the target plaque area, construct a multi-level plaque type representation, input the multi-level plaque type representation into a plaque type classification model that combines a multi-classification support vector machine and a deep belief network, and use the multi-classification support vector machine to classify the plaque type in a coarse-grained manner, and preliminarily classify the plaques into lipid plaques, fibrous plaques, and calcified plaques. At the same time, use the deep belief network to learn the discriminative features of the plaques through multiple hidden layers, and perform multi-classification prediction through the output layer; The classification prediction results of multiple classification support vector machines are integrated to obtain the final plaque type classification result; According to the recognition results of the plaque recognition model, the three-dimensional spatial position of the plaque in the carotid artery is determined in combination with multimodal carotid artery images, and the volume of the plaque is calculated, including: The registered multimodal carotid artery images are input into a pre-trained multi-scale 3D convolutional neural network. The local and global features of the multimodal images are extracted through multi-scale 3D convolutional layers and pooling layers. The gated attention mechanism is used to perform adaptive weighted fusion of features of different scales and modalities to obtain a fused multi-scale 3D feature map. The fused multi-scale 3D feature map is input into the pre-trained convolutional neural network, the structural features of the patch area are extracted through the convolutional block and transition layer, and the patch area is classified and edge optimized at the pixel level using the conditional random field to obtain the segmented 3D patch mask; The segmented three-dimensional plaque mask is superimposed with the original multimodal carotid artery image to obtain the three-dimensional spatial position information of the plaque area, and the plaque area is reconstructed in three dimensions using a ray casting algorithm to obtain a three-dimensional model of the plaque. The reconstructed three-dimensional plaque model is then subjected to morphological analysis and hemodynamic simulation to calculate the volume of the plaque. The multimodal imaging feature set of carotid artery plaques is input into the preset plaque development prediction model to predict the development of plaques at different times in the future. The plaque development prediction results obtained include: Preprocess the multimodal imaging features in the multimodal imaging feature set of carotid plaques to extract standardized imaging omics features; Construct a medical knowledge graph in the field of plaques, which includes the dimensions of anatomical structure, physiological function and pathological manifestation, explore the intrinsic relationship between plaque imaging features and clinical phenotypes through knowledge reasoning and semantic association, and construct a plaque imaging-clinical association knowledge base, which includes the association knowledge between plaque morphological features, plaque density features, plaque texture features, plaque structural features and clinical prognosis, and mine the association rules between plaque imaging features and clinical prognosis from historical case data; The plaque radiomics features are integrated with the plaque field medical knowledge graph and the structured knowledge representation in the plaque radiomics-clinical association knowledge base. The image-knowledge complementary information is mined from the plaque radiomics features and structured knowledge representation through multi-view feature learning methods to obtain the plaque image-knowledge fusion features. Constructing a plaque development prediction model, inputting the plaque image-knowledge fusion features into the plaque development prediction model, predicting the development of the current plaque at different time points in the future through a time series prediction algorithm, and obtaining a plaque development prediction result; The plaque development prediction result is visualized and semantically interpreted according to the medical knowledge graph in the plaque field to obtain the plaque development prediction result.

2. The method according to claim 1, It is characterized in that Extracting the image features of the registered multimodal carotid artery image to obtain a multimodal image feature set of carotid artery plaque includes: Perform superpixel segmentation on each modality of the registered multimodal carotid artery images, divide the images into multiple locally consistent regions as nodes of the graph, extract the local texture features of each node as the initial feature representation of the node, and use the pre-trained deep convolutional neural network to perform semantic segmentation on the images of each modality, obtain pixel-level anatomical structure annotations as the semantic attributes of the nodes, calculate the attention weights of nodes between different modalities, consider the feature similarity and semantic consistency of nodes in different modalities, build cross-modal edge connections, and within each modality, build intra-modal edge connections based on the spatial adjacency and feature similarity between nodes to obtain the final multimodal image graph structure; A multimodal attention graph convolutional network model is constructed. The constructed multimodal image graph structure is used as input. By introducing the modal attention mechanism and the cross-scale feature fusion strategy, the importance weights of different modalities are adaptively learned. The modal attention mechanism dynamically adjusts the weights of graph convolution by calculating the correlation between nodes of different modalities. The cross-scale feature fusion strategy transfers and interacts features between different convolutional layers through upsampling and downsampling operations to extract multi-scale semantic information. In the multimodal attention graph convolutional network model, a graph attention pooling layer is set to adaptively compress the graph structure and aggregate features by learning the importance weights of nodes. At the same time, a multi-head attention module is designed to perform weighted fusion on graph-level features extracted from different modalities to obtain multimodal fusion features. An end-to-end multi-task learning method is adopted, with patch segmentation and recognition as supervision information, to optimize the parameters of the multimodal attention graph convolutional network model and obtain a trained multimodal attention graph convolutional network model; The multimodal carotid artery images to be analyzed are input into the trained multimodal attention graph convolutional network model to automatically extract multimodal fusion features and obtain the multimodal image feature set of carotid artery plaques.

3. The method according to claim 2, It is characterized in that An end-to-end multi-task learning method is adopted, with patch segmentation and recognition as supervision information, to optimize the parameters of the multimodal attention graph convolutional network model. The trained multimodal attention graph convolutional network model includes: Self-supervised learning loss is introduced, and a joint loss function is constructed by combining multi-task learning loss. The model parameters are updated by minimizing the joint loss function, and the iteration is repeated until the preset number of iterations is met, and finally a trained multimodal attention graph convolutional network model is obtained; The joint loss function formula is as follows: ; Where L represents the joint loss function, λ 1 represents the weight coefficient of the patch segmentation loss, M represents the total number of pixels, and p m represents the probability that the mth pixel predicted by the model belongs to a patch, g m represents the true label of the m-th pixel, λ 2 represents the patch recognition loss, C represents the number of patch categories, and w c represents the weight of the cth class, y c Indicates the true label that the patch belongs to the cth class, represents the probability that the model predicts that the sample belongs to the cth class, λ 3 represents the weight coefficient of self-supervised learning loss, N represents the number of nodes, and s ij represents the similarity between node i and node j, d(·) represents the distance metric function, and h i represents the hidden representation of node i, h j represents the hidden representation of node j.

4. A carotid plaque intelligent recognition system based on multimodal image fusion, used to implement the method described in any one of claims 1 to 3. It is characterized in that include: The first unit is used to obtain a carotid ultrasound image and a carotid CT image, preprocess the carotid ultrasound image and the carotid CT image, register the preprocessed carotid ultrasound image and the preprocessed carotid CT image to obtain a registered multimodal carotid image, extract image features of the registered multimodal carotid image, and obtain a multimodal image feature set of carotid plaque; The second unit is used to input the multimodal image feature set into a pre-trained plaque recognition model to identify whether there is a plaque in the multimodal carotid artery image. If there is a plaque, the multimodal image feature set of the carotid artery plaque is input into a plaque type classification model to classify the type of the plaque; The third unit is used to determine the three-dimensional spatial position of the plaque in the carotid artery based on the recognition result of the plaque recognition model and the multimodal carotid artery image, and calculate the volume of the plaque; determine the danger level of the plaque based on the classification result of the plaque classification model, the three-dimensional spatial position of the plaque and the volume of the plaque; if the danger level of the plaque is greater than the preset danger level threshold, input the multimodal imaging feature set of the carotid plaque into the preset plaque development prediction model to predict the development of the plaque at different times in the future, and obtain the plaque development prediction result; use three-dimensional visualization technology to fuse the plaque recognition result, type classification result, plaque position, plaque volume, and plaque development prediction result with the original carotid artery ultrasound image and carotid artery CT image for display, and generate a carotid plaque intelligent recognition report.

5. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 3.

6. A computer-readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.