An artificial intelligence multimodal medical image processing system

Abstract

The present invention discloses an artificial intelligence multimodal medical image processing and diagnosis and treatment system, which relates to the field of medical image processing and diagnosis and treatment technology. It collects multimodal brain image data from multiple angles, extracts brain region features based on the multimodal brain images and formulates a normal brain model. The acquired brain region feature set is fused to generate a three-dimensional structural model of the brain region. The three-dimensional structural model is compensated using the patient's medical data to obtain patient characteristics, and the patient characteristics are analyzed to obtain a diagnosis report. The present invention constructs a three-dimensional brain structural model by fusing multimodal image features and patient medical data, simulates the pathophysiological process of brain diseases, thereby improving the accuracy and reliability of diagnosis, and providing technical support for early diagnosis and personalized treatment of brain diseases.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and diagnosis and treatment technology, and in particular to an artificial intelligence multimodal medical image processing and diagnosis and treatment system. Background Art

[0002] Currently, medical image processing relies primarily on the experience and expertise of physicians. However, with the continuous development of medical imaging technology, the amount of image data has increased dramatically, and traditional manual analysis methods can no longer meet the needs of efficient and accurate diagnosis. In recent years, artificial intelligence technology, especially deep learning technology, has made significant progress in the field of medical image analysis.

[0003] However, most existing AI medical image processing systems target single-modality image data, such as CT or MRI images, ignoring the complementarity and correlation between multimodal image data, which limits the accuracy and comprehensiveness of diagnosis.

[0004] Therefore, it is of great significance to design a multimodal medical imaging intelligent assisted diagnosis and treatment system for brain diseases. Summary of the Invention

[0005] The purpose of the present invention is to provide an artificial intelligence multimodal medical image processing and diagnosis system to address the shortcomings of the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solutions: an artificial intelligence multimodal medical image processing and diagnosis system, comprising:

[0007] Data processing module: collects multi-modal brain imaging data from multiple angles, extracts brain regional features based on multi-modal brain images, and develops a normal brain model;

[0008] Feature fusion module: Connected to the data processing module, it fuses the acquired brain region feature set to generate a three-dimensional structural model of the brain region. The three-dimensional structural model is then compensated with the patient's medical data to obtain the patient's characteristics.

[0009] Diagnostic decision module: connected with the feature fusion module to analyze the patient's characteristics and produce a diagnosis report.

[0010] In a preferred embodiment, the data processing module includes:

[0011] Collect multi-modal brain imaging data from multiple angles, including MRI images, CT images, and positron emission tomography images, and annotate image information including shooting angle and timestamp;

[0012] The multimodal brain imaging data are registered to the same space, and the MRT image is selected as the reference image;

[0013] Extracting brain region features of the multimodal brain imaging data as a brain region feature set based on the specificity of the brain imaging data;

[0014] A normal brain model is developed including brain organs, gray matter, white matter, ventricles, and blood vessels, and the brain region is divided into multiple sub-regions based on the normal brain model.

[0015] In a preferred embodiment, the step of extracting brain regional features of multimodal brain imaging data as a regional feature set based on the specificity of the brain imaging data is:

[0016] The convolutional neural network is used to extract the anatomical structure features and functional connectivity features of MRI images, and the density features and vascular features of CT images;

[0017] The anatomical structural features include the volume, morphology, and location of brain organs, gray matter, white matter, and cerebrospinal fluid; the functional connectivity features include the functional connectivity strength between brain regions; the density features include the CT value of brain tissue and the CT value of lesions; and the vascular features include the diameter, morphology, and blood flow velocity of blood vessels.

[0018] Metabolic features of positron emission tomography images were extracted using a recurrent neural network. These features included the metabolic rate and glucose uptake rate of brain tissue.

[0019] The anatomical structure features, functional connectivity features, density features, vascular features and metabolic features are integrated to obtain a brain region feature set.

[0020] In a preferred embodiment, the feature fusion module includes:

[0021] Based on image information and normal brain models, a neural network algorithm is used to construct a three-dimensional structural model of the brain area;

[0022] Dividing the three-dimensional structure model into multiple voxels based on a preset volume;

[0023] The brain region feature set is fused to obtain a fused feature, and the fused feature is assigned to each voxel based on the normal brain model;

[0024] The voxels at the connection points between sub-regions are used as gates, and the voxels on the outer contours of each brain region are used as the edge layers of each brain region. The gates have the functions of receiving data, outputting data, and controlling weights, and the edge layers are used to predict whether brain organs will deform.

[0025] In a preferred embodiment, the steps of fusing the brain region feature set to obtain a fused feature and assigning the fused feature to each voxel based on a normal brain model are:

[0026] Defining preset weights for each feature of the brain region feature set, and fusing each feature based on the preset weights to obtain a fused feature;

[0027] Read the normal brain model, obtain the brain region to which each voxel belongs, establish the spatial correspondence between the fusion feature and the voxel and perform spatial mapping, and assign the fusion feature to the corresponding voxel based on the spatial correspondence;

[0028] If there is a voxel containing multiple fusion features, the range of the sub-region occupied by the voxel is calculated, and different weights are assigned to the fusion features based on the size of the sub-region occupied, and the fusion features are weightedly fused based on the weights.

[0029] In a preferred embodiment, the steps of using voxels at the junctions of two sub-regions as gates and using voxels on the outer contours of each brain region as the edge layer of each brain region are as follows:

[0030] Obtain the patient's real-time brain region characteristics as consultation data;

[0031] After mapping the patient's medical data to the corresponding brain region features to the corresponding gate, the voxel feature changes are observed to obtain the voxel feature, and the voxel feature is input into the neural network for quantification to obtain the quantitative voxel feature;

[0032] At the same time, the voxel diffusion in the edge layer is observed to predict the deformation of brain organs and obtain the deformation trend of brain organs;

[0033] Quantified voxel features and brain organ deformation trends are used as patient characteristics.

[0034] In a preferred embodiment, the step of observing the voxel diffusion of the edge layer to predict the deformation of the brain organ is:

[0035] Extract the structural features and functional connectivity features of the voxels in the edge layer, and use polynomial regression to obtain the trend of changes in the structural features and functional connectivity features;

[0036] A change threshold is preset. When the structural characteristics and functional connection characteristics of the voxels in the edge layer exceed the change threshold, it is determined that the brain organ will be deformed.

[0037] In a preferred embodiment, the diagnosis decision module includes:

[0038] Generate a diagnostic report based on the patient's brain imaging data, medical records, and patient characteristics;

[0039] The diagnostic report includes basic patient information, imaging description, and quantitative analysis results, among which the quantitative analysis results are patient characteristics.

[0040] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0041] 1. The present invention integrates multimodal brain imaging data, including MRI, CT, and PET, and extracts their specific features through deep learning technology. MRI images provide high-resolution anatomical structural features and functional connectivity features, which can accurately present the morphology, volume, and location of brain organs, as well as the strength of functional connectivity between brain regions. CT images provide density and vascular features, which help detect abnormal brain tissue density and vascular lesions. PET images provide metabolic features, which reflect the metabolic activity level and glucose uptake of brain tissue. By fusing these features from different modalities, a more comprehensive and in-depth understanding of brain diseases can be obtained. Compared with single-modality imaging data, multimodal fusion can complement each other and improve the accuracy and reliability of diagnosis. In addition, the solution also integrates patient medical data, including clinical data, genetic data, and biochemical indicators, further enriching patient characteristics and providing a more comprehensive basis for diagnostic decision-making. This multi-dimensional data fusion method can more effectively capture the complexity and heterogeneity of diseases, thereby improving the diagnostic performance of intelligent assisted diagnosis and treatment systems. Especially in the diagnosis of brain diseases, imaging data from different modalities often reflect different aspects of the disease. Multimodal fusion can more comprehensively assess the condition, thereby improving the accuracy and reliability of diagnosis.

[0042] 2. This invention constructs a three-dimensional structural model of the brain region and divides the model into voxels, enabling detailed analysis of the local characteristics of brain tissue. By assigning fused features to each voxel and introducing the concepts of "gates" and "edge layers," this approach can simulate the spread and evolution of brain diseases. The gate design simulates information transmission and regulation between brain regions, while the edge layer simulates cell diffusion and the spread of lesions. By observing the diffusion of voxels in the edge layer, the deformation trend of brain organs can be predicted, providing valuable information for early diagnosis and prognosis of the disease. This voxel-based three-dimensional structural model and edge layer analysis method can more realistically reflect the pathophysiological processes of brain diseases, thereby improving diagnostic accuracy and predictive power. For example, in the diagnosis of brain tumors, by observing the diffusion of voxels in the edge layer, the growth rate and invasion range of the tumor can be predicted, providing a basis for developing personalized treatment plans. In addition, this approach uses a neural network to quantify voxel features, extracting more representative patient characteristics, further improving the efficiency and accuracy of diagnostic decisions. This method of combining biological knowledge with artificial intelligence technology provides new ideas and directions for intelligent assisted diagnosis and treatment of brain diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0044] Figure 1 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] Example 1, please refer to Figure 1 As shown, the artificial intelligence multimodal medical image processing and diagnosis system described in this embodiment includes:

[0047] Data processing module: collects multi-modal brain imaging data from multiple angles, extracts brain regional features based on multi-modal brain images, and develops a normal brain model;

[0048] Feature fusion module: Connected to the data processing module, it fuses the acquired brain region feature set to generate a three-dimensional structural model of the brain region. The three-dimensional structural model is then compensated with the patient's medical data to obtain the patient's characteristics.

[0049] Diagnostic decision module: connected with the feature fusion module, analyzes patient characteristics and produces a diagnostic report;

[0050] Further explanation: Currently, medical image processing mainly relies on the experience and expertise of doctors. However, with the continuous development of medical imaging technology, the amount of image data has increased dramatically, and traditional manual analysis methods can no longer meet the needs of efficient and accurate diagnosis. In recent years, artificial intelligence technology, especially deep learning technology, has made significant progress in the field of medical image analysis.

[0051] However, most existing AI medical image processing systems target single-modality image data, such as CT or MRI images, ignoring the complementarity and correlation between multimodal image data, which limits the accuracy and comprehensiveness of diagnosis.

[0052] The present invention integrates multimodal brain imaging data, including MRI, CT, and PET, and extracts their specific features through deep learning technology. MRI images provide high-resolution anatomical structural features and functional connectivity features, which can accurately present the morphology, volume, and position of brain organs, as well as the strength of functional connectivity between brain regions. CT images provide density and vascular features, which help detect abnormal brain tissue density and vascular lesions. PET images provide metabolic features, which reflect the metabolic activity level and glucose uptake of brain tissue. By fusing these features from different modalities, a more comprehensive and in-depth understanding of brain diseases can be obtained. Compared with single-modality imaging data, multimodal fusion can complement each other and improve the accuracy and reliability of diagnosis. In addition, the solution also integrates patient medical data, including clinical data, genetic data, and biochemical indicators, further enriching patient characteristics and providing a more comprehensive basis for diagnostic decision-making. This multi-dimensional data fusion method can more effectively capture the complexity and heterogeneity of diseases, thereby improving the diagnostic performance of intelligent assisted diagnosis and treatment systems. Especially in the diagnosis of brain diseases, imaging data from different modalities often reflect different aspects of the disease. Multimodal fusion can more comprehensively assess the condition, thereby improving the accuracy and reliability of diagnosis.

[0053] By constructing a 3D structural model of the brain region and dividing it into voxels, this approach enables detailed analysis of local features within brain tissue. By assigning fused features to each voxel and introducing the concepts of "gates" and "edge layers," this approach simulates the spread and evolution of brain diseases. The gates simulate information transmission and regulation between brain regions, while the edge layers simulate cell diffusion and lesion spread. By observing the diffusion of voxels in the edge layer, deformation trends of brain organs can be predicted, providing valuable information for early diagnosis and prognosis. This voxel-based 3D structural model and edge layer analysis method can more realistically reflect the pathophysiological processes of brain diseases, thereby improving diagnostic accuracy and predictive power. For example, in the diagnosis of brain tumors, observing the diffusion of voxels in the edge layer can predict tumor growth rate and invasion range, providing a basis for customized treatment planning. Furthermore, this approach utilizes a neural network to quantify voxel features, extracting more representative patient characteristics, further improving the efficiency and accuracy of diagnostic decisions. This method of combining biological knowledge with artificial intelligence technology provides new ideas and directions for intelligent assisted diagnosis and treatment of brain diseases.

[0054] In one embodiment, the data processing module includes:

[0055] Collect multi-modal brain imaging data from multiple angles, including MRI images, CT images, and positron emission tomography images, and annotate image information including shooting angle and timestamp;

[0056] The multimodal brain imaging data are registered to the same space, and the MRT image is selected as the reference image;

[0057] Extracting brain region features of the multimodal brain imaging data as a brain region feature set based on the specificity of the brain imaging data;

[0058] Develop a normal brain model including brain organs, gray matter, white matter, ventricles, and blood vessels, and divide the brain region into multiple subregions based on the normal brain model;

[0059] Further explanation: multimodal imaging data is collected. MRI-magnetic resonance imaging can provide high-resolution brain soft tissue structure information and can clearly display brain organs, gray matter, white matter, ventricles and other structures. MRI can use a variety of scanning sequences, such as T1WI, T2WI, FLAIR, DWI, etc. to obtain different tissue contrast information. CT images provide clear images of brain bones and calcified tissues, as well as rapid detection of cerebral hemorrhage. CT images have high density resolution and can quantitatively analyze the CT value of brain tissue. Positron emission tomography reflects the metabolic activity and neurotransmitter function of brain tissue by tracking the distribution of radioactive drugs in the brain. Commonly used PET imaging agents include FDG, which is used to evaluate brain glucose metabolism rate. In order to obtain more comprehensive brain structure information, scanning at different angles is required. MRI and CT usually use axial, coronal and sagittal scans. PET can perform three-dimensional scanning to obtain metabolic distribution information of the whole brain, record the scanning direction and tilt angle of the image, and record the acquisition time of the image for subsequent image registration and time series analysis. The brain imaging data of different modalities are converted to the same spatial coordinate system for subsequent feature extraction and fusion. MRI images are usually selected as the reference image because they have higher spatial resolution and soft tissue contrast, which can provide more accurate anatomical information. Images of other modalities can also be selected as the reference image according to the specific application scenario. Different registration methods are selected for different brain structures. Rigid registration only performs rigid transformations such as translation, rotation, and scaling, which is suitable for cases with small modal differences. Affine registration adds affine transformations such as shearing and tilting on top of rigid transformations, which is suitable for cases with slightly larger modal differences. Nonlinear registration uses more complex transformation models such as B-splines and thin plate splines, which can correct for larger deformations and distortions and is suitable for cases with large modal differences. Feature extraction is performed based on the respective advantages of multimodal brain imaging data. The extracted features are integrated into a brain region feature set for subsequent brain modeling. At the same time, different brain regions are segmented to obtain a normal brain model including brain organs, gray matter, white matter, ventricles, tumors, and blood vessels, providing a framework for subsequent model construction.

[0060] In one embodiment, the step of extracting brain regional features of the multimodal brain image data as a regional feature set based on the specificity of the brain image data is:

[0061] The convolutional neural network is used to extract the anatomical structure features and functional connectivity features of MRI images, and the density features and vascular features of CT images;

[0062] The anatomical structural features include the volume, morphology, and location of brain organs, gray matter, white matter, and cerebrospinal fluid; the functional connectivity features include the functional connectivity strength between brain regions; the density features include the CT value of brain tissue and the CT value of lesions; and the vascular features include the diameter, morphology, and blood flow velocity of blood vessels.

[0063] Metabolic features of positron emission tomography images were extracted using a recurrent neural network. These features included the metabolic rate and glucose uptake rate of brain tissue.

[0064] The anatomical structure features, functional connectivity features, density features, vascular features and metabolic features are integrated to obtain a brain region feature set;

[0065] It is further explained that a deep learning method is used to extract features from multimodal brain images using convolutional neural networks and recurrent neural networks, and these features are integrated into a set of brain region features. Specifically, CNN training starts with data preprocessing, and performs motion correction and spatial smoothing on MRI data sets and CT image data sets; then, the image data is input into the convolution layer, and point multiplication operations are performed through the convolution kernel to extract features in the image; the pooling layer is responsible for down-sampling the feature map, reducing network parameters, and controlling computational complexity; the extracted features are sent to the fully connected layer for classification output; during the training process, the weights of the convolution kernel are continuously adjusted through the backpropagation algorithm to minimize The error between the predicted results and the true labels. Through multiple iterations, CNN gradually learns and optimizes the feature extraction capabilities, and finally forms a feature extraction model that can accurately identify and classify brain image features. The trained feature extraction model is used to process MRI and CT images to extract anatomical structure, functional connectivity, density, and blood vessel features. For MRI images, the feature extraction model is used to identify and quantify anatomical structure features such as the volume, morphology, and position of brain organs, gray matter, white matter, and cerebrospinal fluid, as well as the functional connectivity strength between brain regions. For CT images, the feature extraction model is used to extract density and vascular features such as the CT value of brain tissue, lesion CT value, vascular diameter, vascular morphology, and blood flow velocity. In addition, a metabolic feature extraction model is trained using a recurrent neural network using an LSTM network to convert PET image data into a time series. The LSTM network receives serialized brain image data and gradually adjusts the weights and biases within the network to minimize the prediction error. The difference between the model prediction and the actual label is measured by the loss function, and the network parameters are optimized by the back propagation algorithm. Through multiple iterative training, the LSTM network can learn and extract the key features hidden in the brain image time series, and use the trained metabolic feature extraction model to process PET images to extract metabolic features such as the metabolic rate and glucose uptake rate of brain tissue. Furthermore, this study can also introduce the Transformer architecture to use the self-attention mechanism in the image It captures the dependencies between different regions and is particularly suitable for processing complex cross-regional feature interactions in brain images. In addition, graph neural networks (GNNs) are used to model the spatial relationships between brain regions, optimize the segmentation and feature integration of brain regions, and further improve the correlation between internal and external information in brain regions. Finally, all extracted anatomical structure features, functional connectivity features, density features, vascular features, and metabolic features are integrated to form a comprehensive brain region feature set. This feature set can comprehensively describe the structure, function, and metabolic state of brain tissue, providing rich information for subsequent diagnostic decisions. The automatic feature extraction through deep learning methods avoids the subjectivity and limitations of traditional manual feature extraction, and improves the efficiency and accuracy of feature extraction.

[0066] In one embodiment, the feature fusion module includes:

[0067] Based on image information and normal brain models, a neural network algorithm is used to construct a three-dimensional structural model of the brain area;

[0068] Dividing the three-dimensional structure model into multiple voxels based on a preset volume;

[0069] The brain region feature set is fused to obtain a fused feature, and the fused feature is assigned to each voxel based on the normal brain model;

[0070] The voxels at the junctions of two sub-regions are used as gates, and the voxels on the outer contours of each brain region are used as the edge layer of each brain region. The gates have the functions of receiving data, outputting data, and controlling weights. The edge layer is used to predict whether brain organs will deform.

[0071] Further explanation: First, using image information and a normal brain model, a three-dimensional structural model of the brain area is constructed through a 3D-CNN based on an encoder-decoder structure. The steps are to obtain high-quality brain image data and perform preprocessing, including denoising, debiasing, alignment and standardization, to prepare a normal brain model. This model is an average brain structure based on the atlas. Based on the 3D-CNN of the encoder-decoder structure, the encoder is responsible for mapping the extracted image features to the latent space, and the decoder is responsible for reconstructing the three-dimensional structural model from the latent space. An adversarial loss function is defined to measure the difference between the reconstructed three-dimensional structural model and the real brain structure. The reconstructed three-dimensional structural model is post-processed, such as smoothing, denoising and surface reconstruction, to improve the visualization effect and accuracy of the model. The model accurately reflects the spatial structural relationship of various brain regions. Subsequently, the three-dimensional structural model is divided into cubes of preset volume, and each cube is a voxel, which serves as the basic unit for subsequent feature allocation and analysis. Next, the previously extracted brain region feature sets are fused to obtain a comprehensive fused feature vector. The fusion method may include weighted averaging, splicing, or more complex neural network fusion. Then, based on the normal brain model, the fused feature vector is assigned to the corresponding voxel to ensure that each voxel carries feature information related to the brain region where it is located. To simulate the interactions and information transfer between brain regions, voxels at the junctions of brain regions are defined as "gates." These gates have the functions of receiving and outputting data and controlling weights, regulating the flow of information between different brain regions. Different gate weights can be set for patients of different age groups. For example, the gate weights in the cerebellum of older patients may be lower, while the gate weights in the brainstem of younger patients with active brain regions may be increased. In other words, a gate is a port with edge computing capabilities. In addition, a certain number of voxels are selected at a preset distance at the edge of each brain region as an "edge layer." This edge layer is used to simulate processes such as cell diffusion and lesion spread. Changes in its voxel features can reflect the deformation trends of brain organs. In this way, the feature fusion module not only integrates multimodal imaging features but also constructs a complex model that can simulate brain structure and function, providing richer and more biologically meaningful information for subsequent diagnostic decisions. This module combines geometric modeling, feature engineering, and neural network technologies to capture the complex pathophysiological processes of brain diseases, thereby improving the accuracy and reliability of diagnosis.

[0072] In one embodiment, the steps of fusing the brain region feature set to obtain a fused feature and assigning the fused feature to each voxel based on a normal brain model are:

[0073] Defining preset weights for each feature of the brain region feature set, and fusing each feature based on the preset weights to obtain a fused feature;

[0074] Read the normal brain model, obtain the brain region to which each voxel belongs, establish the spatial correspondence between the fusion feature and the voxel and perform spatial mapping, and assign the fusion feature to the corresponding voxel based on the spatial correspondence;

[0075] If there is a voxel containing multiple fusion features, the range of the sub-region occupied by the voxel is calculated, and different weights are assigned to the fusion features based on the size of the sub-region occupied, and the fusion features are weighted fused based on the weights;

[0076] To further explain, first, preset weights are defined for each feature in the brain region feature set. These weights reflect the diagnostic importance of different features and can be determined based on prior knowledge, expert experience, or data-driven methods (such as learned weights). Then, based on the preset weights, the features are weightedly fused to produce a comprehensive fused feature vector. For example, if the weights for anatomical structure features, functional connectivity features, density features, vascular features, and metabolic features are 0.3, 0.25, 0.2, 0.15, and 0.1, respectively, the fused feature vector can be calculated by taking a weighted average of these features. Next, a normal brain model is read, which contains the label information for the brain region to which each voxel belongs. By analyzing the normal brain model, the spatial location of each voxel and its associated brain region can be determined. Then, a spatial correspondence is established between the fused features and the voxels, and spatial mapping is performed, assigning the fused feature vector to the corresponding voxel. If a voxel is completely within a brain region, the fused feature vector for that region is directly assigned to that voxel. However, if a voxel lies at the junction of multiple brain regions, meaning it contains multiple fused features, further processing is required. In this case, the proposed scheme calculates the proportion of each subregion occupied by the voxel and assigns different weights to the fused features based on the proportion of each subregion. Regions with larger proportions receive higher weights for their corresponding fused features. Finally, based on these weights, the multiple fused features are weighted together to produce the final feature vector for that voxel. For example, if a voxel occupies regions A, B, and C with a proportion of 1, 2, and 3, respectively, the corresponding weights can be calculated using a softmax function or a simple inverse proportional function. These weights are then applied to the fused feature vectors of regions A, B, and C, and a weighted average is performed to produce the final feature vector for that voxel. In this way, the proposed scheme can precisely assign brain regional features to each voxel and take into account the spatial relationships between the voxel and multiple brain regions, thereby more accurately describing the local characteristics of brain tissue. This process combines weighted fusion, spatial mapping, and distance calculation techniques to improve the accuracy and robustness of feature assignment.

[0077] In one embodiment, the steps of using voxels at the junctions of two sub-regions as gates and using voxels on the outer contours of each brain region as the edge layer of each brain region are as follows:

[0078] Obtain the patient's real-time brain region characteristics as consultation data;

[0079] After mapping the patient's medical data to the corresponding brain region features to the corresponding gate, the voxel feature changes are observed to obtain the voxel feature, and the voxel feature is input into the neural network for quantification to obtain the quantitative voxel feature;

[0080] At the same time, the voxel diffusion in the edge layer is observed to predict the deformation of brain organs and obtain the deformation trend of brain organs;

[0081] Quantified voxel features and brain organ deformation trends are used as patient characteristics;

[0082] To further explain, first, the patient's real-time brain region features are obtained and used as medical data; in order to integrate the medical data into the three-dimensional structural model of the brain, the voxels at the connection of each brain region are defined as "gates", and the function of these gates is to receive feature information from different brain regions and the patient's medical data; specifically, a neural network such as a multi-layer perceptron can be used to map the patient's medical data to the corresponding gates, and the mapping method can be a simple feature splicing or a more complex nonlinear transformation; through the gate, the patient's medical data can affect the feature changes of adjacent voxels; then, the feature changes of the voxels are observed, including the structural features, functional connectivity features and metabolic features of the voxels, etc. These feature changes can be quantified by calculating indicators such as the gradient, variance or entropy of the voxel features; in order to extract more representative voxel features, the voxel features are input into the neural network for quantization to obtain quantified voxel features. Quantification methods can include clustering, encoding, or dimensionality reduction. Furthermore, to simulate the spread and evolution of brain diseases, the solution selects a certain number of voxels at the edge of each brain region, at a preset distance (e.g., 1-2 voxels), as an "edge layer." These voxels are used to simulate processes such as cell diffusion and lesion spread. By observing the changes in the characteristics of these voxels, the deformation trend of brain organs can be predicted. For example, the structural and functional connectivity features of these voxels can be extracted, and the temporal trends of these features can be analyzed using polynomial regression or other time series analysis methods. If the characteristic changes of these voxels exceed a preset threshold, the brain organ is considered to have deformed. Finally, the quantified voxel features and the deformation trends of the brain organ are used as patient characteristics for subsequent diagnostic decision-making. In this way, the solution not only integrates multimodal imaging features and patient data but also simulates the pathophysiological processes of brain diseases, thereby improving the accuracy and reliability of diagnosis. By utilizing techniques such as data mapping, feature quantification, edge layer analysis, and deformation prediction, the solution aims to capture the complex pathophysiological processes of brain diseases, thereby improving the accuracy and reliability of diagnosis.

[0083] In one embodiment, the step of observing the voxel diffusion of the edge layer to predict the deformation of the brain organ is:

[0084] Extract the structural features and functional connectivity features of the voxels in the edge layer, and use polynomial regression to obtain the trend of changes in the structural features and functional connectivity features;

[0085] A change threshold is preset. When the structural characteristics and functional connectivity characteristics of the voxels in the edge layer exceed the change threshold, it is determined that the brain organ will undergo deformation.

[0086] To further illustrate, brain organ deformation is predicted by analyzing characteristic changes in voxels in the marginal layer. First, structural features of the marginal layer voxels, such as volume, surface area, and shape, and functional connectivity features, such as the strength of connections with adjacent brain regions, are extracted. Then, a polynomial regression model or other time series analysis method, such as the ARIMA model, is used to fit the time-varying curves of these features to obtain the changing trends of the structural and functional connectivity features, such as linear growth, exponential growth, or cyclical changes. To determine whether a brain organ will deform, a threshold is preset. This threshold can be determined based on prior knowledge, historical data, or statistical analysis. If the magnitude of the change in the structural or functional connectivity features of the marginal layer voxels exceeds the preset threshold, the brain organ is considered to have deformed. For example, if the volume of the marginal layer voxels of the hippocampus decreases significantly over a short period of time, exceeding the preset threshold, the hippocampus is predicted to have atrophied. In this way, the solution can quantitatively assess the risk of brain organ deformation, providing a basis for early diagnosis and prognostic assessment of diseases. This method combines feature extraction, time series analysis, and threshold judgment techniques to predict the deformation trends of brain organs, thereby improving the accuracy and timeliness of diagnosis.

[0087] In one embodiment, the diagnosis decision module includes:

[0088] Generate a diagnostic report based on the patient's brain imaging data, medical records, and patient characteristics;

[0089] The diagnostic report includes basic patient information, imaging descriptions, and quantitative analysis results, where the quantitative analysis results are patient characteristics;

[0090] Further explanation: the diagnostic decision module receives the patient's brain imaging data, medical data, and previously extracted patient characteristics including quantitative voxel features and brain organ deformation trends, and integrates this information to generate a diagnostic report. First, the patient's basic information, such as name, age, gender, medical history, etc., is collected and added to the diagnostic report. Then, the brain imaging data is described, including the visual features of MRI, CT, and PET images, such as the location, size, morphology, density, and metabolic activity of the lesions. These imaging descriptions can be provided by radiologists or generated by automatic image analysis algorithms. Next, the quantitative analysis results such as quantitative voxel features and brain organ deformation trends are combined. Added to the diagnostic report, these quantitative features can be compared with known disease patterns to assist in diagnostic decision-making. Finally, based on all this information, a final diagnostic conclusion is generated and corresponding treatment recommendations are provided. The diagnostic conclusion can include the type, severity and prognosis of the disease, and the treatment recommendations can include drug therapy, surgical treatment and radiotherapy, etc. The diagnostic decision module can use a variety of methods to generate diagnostic conclusions and treatment recommendations, such as rule-based reasoning, machine learning algorithms or deep learning models; the final diagnostic report will contain basic patient information, imaging descriptions and quantitative analysis results, providing clinicians with comprehensive and objective diagnostic information to assist them in making decisions.

[0091] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An artificial intelligence multimodal medical image processing system, characterized by: Data processing module: collects multi-modal brain imaging data from multiple angles, extracts brain region features based on the multi-modal brain images and develops a normal brain model, generates a brain region feature set, and divides the brain region into multiple sub-regions based on the normal brain model; Feature fusion module: This module is connected to the data processing module. Based on the brain region feature set and the normal brain model, it uses a neural network algorithm to construct a three-dimensional structural model of the brain region. The three-dimensional structural model is divided into multiple voxels based on the preset volume. The brain region feature set is fused to obtain a fused feature, and the fused feature is assigned to each voxel based on the normal brain model. The steps of fusing the brain region feature set to obtain a fused feature and assigning the fused feature to each voxel based on the normal brain model are as follows: Defining preset weights for each feature of the brain region feature set, and fusing each feature based on the preset weights to obtain a fused feature; Read the normal brain model, obtain the brain region to which each voxel belongs, establish a spatial correspondence between the fusion feature and the voxel, and perform spatial mapping. According to the spatial correspondence, the fusion feature is assigned to the corresponding voxel. If a voxel contains multiple fusion features, the range of the sub-region occupied by the voxel is calculated, and different weights are assigned to the fusion features based on the size of the sub-region occupied. The fusion features are then weighted fused based on the weights. The voxels at the junctions of two sub-regions are used as gates, and the voxels on the outer contours of each brain region are used as the edge layer of each brain region. The patient's medical data is used to compensate the three-dimensional structural model to obtain patient characteristics. The gate has the functions of receiving data, outputting data, and controlling weights, regulating the flow of information between different sub-regions. The edge layer is used to predict whether brain organs will deform. The steps of using the voxels at the connection points between the two sub-regions as gates and using the voxels on the outer contours of each brain region as the edge layer of each brain region are as follows: Obtain the patient's real-time brain region characteristics as consultation data; After mapping the brain region features corresponding to the patient's medical data to the corresponding gate, the voxel features are obtained according to the feature changes of the voxels, and the voxel features are input into the neural network for quantification to obtain the quantitative voxel features; At the same time, the deformation of brain organs is predicted based on the voxel diffusion of the edge layer to obtain the deformation trend of brain organs; Diagnostic decision module: connected with the feature fusion module to analyze the patient's characteristics and produce a diagnosis report.

2. The artificial intelligence multimodal medical image processing system according to claim 1, characterized in that: The data processing module includes: Collect multi-modal brain imaging data from multiple angles, including MRI images, CT images, and positron emission tomography images, and annotate image information including shooting angle and timestamp; The multimodal brain imaging data are registered to the same space, and the MRT image is selected as the reference image; Extracting brain region features of the multimodal brain imaging data as a brain region feature set based on the specificity of the brain imaging data; A normal brain model is developed including brain organs, gray matter, white matter, ventricles, and blood vessels, and the brain region is divided into multiple sub-regions based on the normal brain model.

3. The artificial intelligence multimodal medical image processing system according to claim 2, characterized in that: The step of extracting brain region features of the multimodal brain image data as a brain region feature set based on the specificity of the brain image data is as follows: The convolutional neural network is used to extract the anatomical structure features and functional connectivity features of MRI images, and the density features and vascular features of CT images; The anatomical structural features include the volume, morphology, and location of brain organs, gray matter, white matter, and cerebrospinal fluid; the functional connectivity features include the functional connectivity strength between brain regions; the density features include the CT value of brain tissue and the CT value of lesions; and the vascular features include the diameter, morphology, and blood flow velocity of blood vessels. Metabolic features of positron emission tomography images were extracted using a recurrent neural network. These features included the metabolic rate and glucose uptake rate of brain tissue. The anatomical structure features, functional connectivity features, density features, vascular features and metabolic features are integrated to obtain a brain region feature set.

4. The artificial intelligence multimodal medical image processing system according to claim 1, characterized in that: The step of predicting the deformation of the brain organ based on the voxel diffusion of the edge layer is: Extract the structural features and functional connectivity features of the voxels in the edge layer, and use polynomial regression to obtain the trend of changes in the structural features and functional connectivity features; A change threshold is preset. When the structural characteristics and functional connection characteristics of the voxels in the edge layer exceed the change threshold, it is determined that the brain organ will be deformed.

5. The artificial intelligence multimodal medical image processing system according to claim 1, characterized in that: The diagnosis decision module includes: Generate a diagnostic report based on the patient's brain imaging data, medical records, and patient characteristics; The diagnostic report includes basic patient information, imaging description, and quantitative analysis results, among which the quantitative analysis results are patient characteristics.

Images