Artificial intelligence multi-mode medical image processing diagnosis and treatment system

Abstract

The invention discloses an artificial intelligence multi-modal medical image processing diagnosis and treatment system, and relates to the technical field of medical image processing and diagnosis and treatment, and the system collects multi-angle multi-modal brain image data, extracts brain region features based on multi-modal brain images, formulates a brain normal model, carries out the set fusion of the obtained brain region features, and carries out the diagnosis and treatment of the brain normal model. Generating a three-dimensional structure model of the brain area, compensating the three-dimensional structure model by using the treatment data of the patient to obtain patient features, and analyzing the patient features to obtain a diagnosis report; according to the method, the brain three-dimensional structure model is constructed by fusing the multi-modal image features and the patient treatment data, and the pathophysiological process of the brain diseases is simulated, so that the diagnosis accuracy and reliability are improved, and technical support is provided for early diagnosis and personalized treatment of the brain diseases.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing and diagnosis and treatment, and particularly relates to an artificial intelligence multi-modal medical image processing and diagnosis and treatment system. Background Art

[0002] At present, medical image processing mainly relies on doctors' experience and professional knowledge. However, with the continuous development of medical imaging technology, the volume of image data has increased sharply, and traditional manual analysis methods are difficult to meet the requirements of efficient and accurate diagnosis. In recent years, artificial intelligence technology, especially deep learning technology, has made remarkable progress in the field of medical image analysis.

[0003] However, most of the existing artificial intelligence medical image processing systems are aimed at single-modal image data, such as only processing CT or MRI images, ignoring the complementarity and relevance between multi-modal image data, which limits the accuracy and comprehensiveness of diagnosis.

[0004] Therefore, it is of great significance to design a multi-modal intelligent auxiliary diagnosis and treatment system for brain diseases in medical images. Summary of the Invention

[0005] The purpose of the present invention is to provide an artificial intelligence multi-modal medical image processing and diagnosis and treatment system to solve the deficiencies in the background art.

[0006] To achieve the above purpose, the present invention provides the following technical solutions: An artificial intelligence multi-modal medical image processing and diagnosis and treatment system, comprising:

[0007] A data processing module: collecting multi-modal brain image data from multiple angles, extracting brain region features based on the multi-modal brain images and formulating a normal brain model;

[0008] A feature fusion module: connected to the data processing module, fusing the obtained set of brain region features to generate a three-dimensional structure model of the brain region, and compensating the three-dimensional structure model with the patient's medical data to obtain patient features;

[0009] A diagnosis and decision-making module: connected to the feature fusion module, analyzing the patient features to obtain a diagnosis report.

[0010] In a preferred embodiment, the data processing module includes:

[0011] Collecting multi-modal brain image data from multiple angles includes MRI images, CT images, and positron emission tomography images, and annotating image information including the shooting angle and timestamp;

[0012] Registering the multi-modal brain image data into the same space, and selecting the MRT image as the reference image;

[0013] Specifically extract the brain region features of multi-modal brain imaging data based on brain imaging data as a set of brain region features;

[0014] Formulate a normal brain model including brain organs, gray matter, white matter, ventricles, and blood vessels, and divide the brain region into multiple sub-regions based on the normal brain model.

[0015] In a preferred embodiment, the step of specifically extracting the brain region features of multi-modal brain imaging data based on brain imaging data as a set of regional features is as follows:

[0016] Extract the anatomical structure features and functional connectivity features of the MRI image through a convolutional neural network, and extract the density features and vascular features of the CT image;

[0017] Among them, the anatomical structure features include the volume, morphology, and location of brain organs, gray matter, white matter, and cerebrospinal fluid, the functional connectivity features include the functional connectivity strength between brain regions, the density features include the CT values of brain tissue and lesions, and the vascular features include the diameter, morphology, and blood flow velocity of blood vessels;

[0018] Extract the metabolic features of the positron emission tomography image through a recurrent neural network, and the metabolic features include the metabolic rate and glucose uptake rate of brain tissue;

[0019] Integrate the anatomical structure features, functional connectivity features, density features, vascular features, and metabolic features to obtain a set of brain region features.

[0020] In a preferred embodiment, the feature fusion module includes:

[0021] Construct a three-dimensional structure model of the brain region using a neural network algorithm based on image information and the normal brain model;

[0022] Divide the three-dimensional structure model into multiple voxels based on a preset volume;

[0023] Fuse the set of brain region features to obtain a fused feature, and assign the fused feature to each voxel based on the normal brain model;

[0024] Take the voxels at the connections between two sub-regions as gates respectively, and take the voxels on the outer contours of each brain region as the edge layers of each brain region. Among them, the gates have the functions of receiving data, outputting data, and weight control, and the edge layers are used to predict whether the brain organs will deform.

[0025] In a preferred embodiment, the step of fusing the set of brain region features to obtain a fused feature and assigning the fused feature to each voxel based on the normal brain model is as follows:

[0026] Define the preset weights for each feature in the set of brain region features, and fuse each feature based on the preset weights to obtain a fused feature;

[0027] Read the normal brain model, obtain the brain region to which each voxel belongs, establish a spatial correspondence between the fused feature and the voxel and perform spatial mapping, and assign the fused feature to the corresponding voxel according to the spatial correspondence;

[0028] If there are voxels containing multiple fused features, calculate the range of the sub-region occupied by the voxel, assign different weights to the fused features based on the size of the occupied sub-region range, and perform weighted fusion on the fused features based on the weights.

[0029] In a preferred embodiment, the steps of using the voxels at the joints of each pair of sub-regions as gates respectively and using the voxels on the outer contours of each brain region as the edge layer of each brain region are as follows:

[0030] Obtain the real-time brain region features of the patient as the medical treatment data;

[0031] After mapping the medical treatment data of the patient corresponding to the brain region features to the corresponding gates, observe the feature changes of the voxels to obtain voxel features, and input the voxel features into a neural network for quantization to obtain quantized voxel features;

[0032] At the same time, observe the voxel diffusion situation of the edge layer to predict the deformation of the brain organ to obtain the deformation trend of the brain organ;

[0033] Take the quantized voxel features and the deformation trend of the brain organ as the patient features.

[0034] In a preferred embodiment, the steps of observing the voxel diffusion situation of the edge layer to predict the deformation of the brain organ are as follows:

[0035] Extract the structural features and functional connection features of the voxels in the edge layer, and use polynomial regression to obtain the trends of changes in the structural features and functional connection features;

[0036] Preset a change threshold, and when the structural features and functional connection features of the voxels in the edge layer exceed the change threshold, it is determined that the brain organ will deform.

[0037] In a preferred embodiment, the diagnostic decision-making module includes:

[0038] Generate a diagnostic report through the patient's brain image data, medical treatment data, and patient features;

[0039] The diagnostic report includes the patient's basic information, imaging description, and quantitative analysis results, where the quantitative analysis results are the patient features.

[0040] In the above technical solution, the technical effects and advantages provided by the present invention:

[0041] 1. The present invention integrates multi-modal brain imaging data, including MRI, CT, and PET, and extracts their respective specific features through deep learning techniques. MRI images provide high-resolution anatomical structure features and functional connectivity features, which can finely present the morphology, volume, and location of brain organs, as well as the functional connectivity strength between brain regions. CT images provide density features and vascular features, which are helpful for detecting abnormal tissue density and vascular lesions in the brain. PET images provide metabolic features, reflecting the metabolic activity level and glucose uptake of brain tissues. By fusing these features from different modalities, a more comprehensive and in-depth understanding of brain diseases can be obtained. Compared with single-modal imaging data, multi-modal fusion can complement each other's advantages, improving the accuracy and reliability of diagnosis. In addition, this solution also integrates the patient's medical data, including clinical data, genetic data, and biochemical indicators, further enriching the patient characteristics and providing a more comprehensive basis for diagnostic decisions. This multi-dimensional data fusion method can more effectively capture the complexity and heterogeneity of diseases, thereby improving the diagnostic performance of intelligent assisted diagnosis and treatment systems. Especially in the diagnosis of brain diseases, different-modal imaging data often reflect different aspects of the disease, and multi-modal fusion can more comprehensively evaluate the condition, thus improving the accuracy and reliability of diagnosis;

[0042] 2. The present invention constructs a three-dimensional structure model of the brain region and divides the model into voxels, enabling fine analysis of the local features of brain tissues. By assigning the fused features to each voxel and introducing the concepts of "gates" and "marginal layers", this solution can simulate the spread and evolution process of brain diseases. The design of the gates simulates the information transmission and regulation between brain regions, while the marginal layer simulates cell diffusion and lesion spread; by observing the voxel diffusion in the marginal layer, the deformation trend of brain organs can be predicted, providing valuable information for early disease diagnosis and prognosis assessment. This voxel-based three-dimensional structure model and marginal layer analysis method can more realistically reflect the pathophysiological process of brain diseases, thereby improving the accuracy and predictive ability of diagnosis; for example, in the diagnosis of brain tumors, by observing the voxel diffusion in the marginal layer, the growth rate and invasion range of tumors can be predicted, providing a basis for formulating individualized treatment plans. In addition, this solution also uses neural networks to quantify voxel features, extracting more representative patient characteristics and further improving the efficiency and accuracy of diagnostic decisions. This method of combining biological knowledge with artificial intelligence technology provides new ideas and directions for the intelligent assisted diagnosis and treatment of brain diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0044] Figure 1 It is the system block diagram of the present invention. Specific embodiments

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0046] Embodiment 1. Please refer to Figure 1 As shown, a kind of artificial intelligence multi-modal medical image processing diagnosis and treatment system described in this embodiment includes:

[0047] Data processing module: Collect multi-modal brain image data from multiple angles, extract brain region features based on the multi-modal brain images, and establish a normal brain model;

[0048] Feature fusion module: Connected to the data processing module, fuse the obtained set of brain region features to generate a three-dimensional structure model of the brain region, and make compensation for the three-dimensional structure model using the patient's medical data to obtain patient characteristics;

[0049] Diagnosis decision module: Connected to the feature fusion module, analyze the patient characteristics to obtain a diagnosis report;

[0050] Further explanation, at present, medical image processing mainly relies on doctors' experience and professional knowledge. However, with the continuous development of medical imaging technology, the amount of image data has increased sharply, and traditional manual analysis methods are difficult to meet the requirements of efficient and accurate diagnosis. In recent years, artificial intelligence technology, especially deep learning technology, has made remarkable progress in the field of medical image analysis;

[0051] However, most of the existing artificial intelligence medical image processing systems are for single-modal image data, such as only processing CT or MRI images, ignoring the complementarity and relevance between multi-modal image data, which limits the accuracy and comprehensiveness of diagnosis;

[0052] The present invention integrates multi-modal brain imaging data, including MRI, CT, and PET, and extracts their respective specific features through deep learning techniques. MRI images provide high-resolution anatomical structure features and functional connectivity features, enabling the fine presentation of the morphology, volume, and location of brain organs, as well as the functional connectivity strength between brain regions. CT images provide density features and vascular features, which are helpful for detecting abnormal tissue density and vascular lesions in the brain. PET images provide metabolic features, reflecting the metabolic activity level and glucose uptake of brain tissues. By fusing these features from different modalities, a more comprehensive and in-depth understanding of brain diseases can be obtained. Compared with single-modal imaging data, multi-modal fusion can complement each other's advantages, improving the accuracy and reliability of diagnosis. In addition, the solution also integrates the patient's medical data, including clinical data, genetic data, and biochemical indicators, further enriching the patient characteristics and providing a more comprehensive basis for diagnostic decisions. This multi-dimensional data fusion method can more effectively capture the complexity and heterogeneity of diseases, thereby enhancing the diagnostic performance of intelligent assisted diagnosis and treatment systems. Especially in the diagnosis of brain diseases, different modalities of imaging data often reflect different aspects of the disease, and multi-modal fusion can more comprehensively evaluate the condition, thus improving the accuracy and reliability of diagnosis;

[0053] By constructing a three-dimensional structure model of the brain region and dividing the model into voxels, the local features of brain tissues can be finely analyzed. By assigning the fused features to each voxel and introducing the concepts of "gates" and "edge layers", the solution can simulate the diffusion and evolution process of brain diseases. The design of the gates simulates the information transmission and regulation between brain regions, while the edge layer simulates cell diffusion and lesion spread. By observing the voxel diffusion in the edge layer, the deformation trend of brain organs can be predicted, providing valuable information for early disease diagnosis and prognosis assessment. This voxel-based three-dimensional structure model and edge layer analysis method can more realistically reflect the pathophysiological process of brain diseases, thereby improving the accuracy and predictive ability of diagnosis. For example, in the diagnosis of brain tumors, by observing the voxel diffusion in the edge layer, the growth rate and invasion range of the tumor can be predicted, providing a basis for formulating individualized treatment plans; in addition, the solution also uses neural networks to quantify voxel features, extracting more representative patient characteristics and further enhancing the efficiency and accuracy of diagnostic decisions. This method of combining biological knowledge with artificial intelligence technology provides new ideas and directions for the intelligent assisted diagnosis and treatment of brain diseases.

[0054] In one embodiment, the data processing module includes:

[0055] Collect multi-modal brain imaging data from multiple angles, including MRI images, CT images, and positron emission tomography images, and label the image information including the shooting angle and timestamp;

[0056] Register multi-modal brain image data into the same space, and select the MRT image as the reference image;

[0057] Extract the brain region features of the multi-modal brain image data based on the specificity of the brain image data as the brain region feature set;

[0058] Formulate a normal brain model including brain organs, gray matter, white matter, ventricles and blood vessels, and divide the brain region into multiple sub-regions based on the normal brain model;

[0059] Further explanation, collect multi-modal image data. MRI - magnetic resonance imaging can provide high-resolution brain soft tissue structure information, and can clearly display structures such as brain organs, gray matter, white matter, ventricles, etc. MRI can use a variety of scanning sequences, such as T1WI, T2WI, FLAIR, DWI, etc., to obtain different tissue contrast information. CT images provide clear images of brain bones and calcified tissues, as well as rapid detection of cerebral hemorrhage. CT images have high density resolution and can quantitatively analyze the CT values of brain tissues. Positron emission tomography reflects the metabolic activities and neurotransmitter functions of brain tissues by tracking the distribution of radioactive drugs in the brain. Commonly used PET imaging agents include FDG, which is used to evaluate the cerebral glucose metabolic rate. In order to obtain more comprehensive brain structure information, scans need to be performed at different angles. MRI and CT usually use axial, coronal and sagittal scans. PET can perform three-dimensional scans to obtain the metabolic distribution information of the whole brain, record the scan direction and tilt angle of the images, and record the acquisition time of the images for subsequent image registration and time series analysis. Convert brain image data from different modalities and different angles into the same spatial coordinate system for subsequent feature extraction and fusion. Usually, the MRI image is selected as the reference image because the MRI image has high spatial resolution and soft tissue contrast and can provide more accurate anatomical structure information. Other modality images can also be selected as the reference image according to specific application scenarios. Different registration methods are selected for different brain structures. Rigid registration only performs rigid transformations such as translation, rotation and scaling, and is suitable for cases with small modality differences. Affine registration adds affine transformations such as shear and tilt on the basis of rigid transformations and is suitable for cases with slightly larger modality differences. Non-linear registration uses more complex transformation models, such as B-splines, thin plate splines, etc., which can correct large deformations and distortions and is suitable for cases with large modality differences. Feature extraction is performed based on the respective advantages of multi-modal brain image data, and the extracted features are integrated into a brain region feature set for subsequent brain modeling. At the same time, different regions of the brain are segmented to obtain a normal brain model including brain organs, gray matter, white matter, ventricles, tumors and blood vessels, providing a framework basis for the subsequent construction of the model.

[0060] In one embodiment, the step of extracting the brain region features of the multi-modal brain image data based on the brain image data as the region feature set is as follows:

[0061] Extract the anatomical structure features and functional connectivity features of the MRI image through a convolutional neural network, and extract the density features and vascular features of the CT image;

[0062] Among them, the anatomical structure features include the volume, morphology, and location of brain organs, gray matter, white matter, and cerebrospinal fluid. The functional connectivity features include the functional connectivity strength between brain regions. The density features include the CT values of brain tissue and the CT values of lesions. The vascular features include the diameter, morphology, and blood flow velocity of blood vessels;

[0063] Extract the metabolic features of the positron emission tomography image through a recurrent neural network. The metabolic features include the metabolic rate and glucose uptake rate of brain tissue;

[0064] Integrate the anatomical structure features, functional connectivity features, density features, vascular features, and metabolic features to obtain the brain region feature set;

[0065] Further elaboration, using deep learning methods, convolutional neural networks and recurrent neural networks are respectively used to extract features from multimodal brain images, and these features are integrated into a set of brain region features. Specifically, the training of CNN starts from data preprocessing. For MRI data sets and CT image data sets, steps such as motion correction and spatial smoothing are carried out; subsequently, the image data is input into the convolutional layer, and dot product operations are performed through convolutional kernels to extract features in the images; the pooling layer is responsible for downsampling the feature maps, reducing network parameters and controlling computational complexity; the extracted features are sent to the fully connected layer for classification output; during the training process, the weights of the convolutional kernels are continuously adjusted through the backpropagation algorithm, aiming to minimize the error between the prediction result and the true label. Through multiple iterations, CNN gradually learns and optimizes the feature extraction ability, and finally forms a feature extraction model that can accurately identify and classify brain image features. The trained feature extraction model is used to process MRI and CT images to extract features such as anatomical structures, functional connections, density, and blood vessels; for MRI images, the feature extraction model is used to identify and quantify anatomical structure features such as the volume, shape, and location of brain organs, gray matter, white matter, and cerebrospinal fluid, as well as the functional connection strength between brain regions; for CT images, the feature extraction model is used to extract density and blood vessel features such as CT values of brain tissue, CT values of lesions, blood vessel diameters, blood vessel shapes, and blood flow velocities. In addition, a recurrent neural network is used to train a metabolic feature extraction model using the LSTM network, converting PET image data into time series. The LSTM network receives serialized brain image data and gradually adjusts the weights and biases inside the network to minimize the prediction error. The difference between the model prediction and the actual label is measured through a loss function, and the network parameters are optimized through the backpropagation algorithm. Through multiple iterations of training, the LSTM network can learn and extract key features hidden in the time series of brain images. The trained metabolic feature extraction model is used to process PET images to extract metabolic features such as the metabolic rate and glucose uptake rate of brain tissue; furthermore, this study can also introduce the Transformer architecture, which captures the dependencies between different regions in the image through the self-attention mechanism, and is particularly suitable for processing complex cross-region feature interactions in brain images; in addition, graph neural networks (GNNs) are applied to model the spatial relationships between brain regions, optimize the segmentation and feature integration of brain regions, and further improve the correlation of information inside and outside brain regions; finally, all the extracted anatomical structure features, functional connection features, density features, blood vessel features, and metabolic features are integrated to form a comprehensive set of brain region features. This set of features can comprehensively describe the structure, function, and metabolic state of brain tissue, providing rich information for subsequent diagnostic decisions. Automatically extracting features through deep learning methods avoids the subjectivity and limitations of traditional manual feature extraction and improves the efficiency and accuracy of feature extraction.

[0066] In one embodiment, the feature fusion module includes:

[0067] Construct a three-dimensional structure model of the brain region using a neural network algorithm based on image information and a normal brain model;

[0068] Divide the three-dimensional structure model into multiple voxels based on a preset volume;

[0069] Fuse the brain region feature sets to obtain fused features, and assign the fused features to each voxel based on the normal brain model;

[0070] Take the voxels at the connections between pairs of sub-regions as gates respectively, and take the voxels on the outer contours of each brain region as the edge layers of each brain region. Among them, the gates have functions of receiving data, outputting data, and weight control, and the edge layers are used to predict whether the brain organs will deform;

[0071] Further explanation: First, using the image information and the normal brain model, a three-dimensional structural model of the brain region is constructed through 3D-CNN based on the encoder-decoder structure. The steps are as follows: Obtain high-quality brain image data and perform preprocessing, including noise reduction, debiasing, registration, and normalization. Prepare the normal brain model, which is the average brain structure based on the atlas. For the 3D-CNN based on the encoder-decoder structure, the encoder is responsible for mapping the extracted image features to the latent space, and the decoder is responsible for reconstructing the three-dimensional structural model from the latent space. Define the adversarial loss function to measure the difference between the reconstructed three-dimensional structural model and the real brain structure. Perform post-processing on the reconstructed three-dimensional structural model, such as smoothing, denoising, and surface reconstruction, etc., to improve the visualization effect and accuracy of the model. This model accurately reflects the spatial structure relationship of each brain region. Subsequently, divide the three-dimensional structural model into cubes of a preset volume, and each cube serves as a voxel, which is the basic unit for subsequent feature assignment and analysis. Then, fuse the previously extracted set of brain region features to obtain a comprehensive fused feature vector. The fusion method can include weighted average, concatenation, or more complex neural network fusion. Then, according to the normal brain model, assign the fused feature vector to the corresponding voxels to ensure that each voxel carries the feature information related to the brain region where it is located. To simulate the interaction and information transmission between brain regions, the voxels at the junctions of each brain region are defined as "gates". These gates have the functions of receiving data, outputting data, and weight control, and can regulate the flow of information between different brain regions. Different weights of the gates can be set for patients of different ages. For example, the weight of the gate in the cerebellum of older patients is appropriately lower, and the weight of the gate in the brainstem of young people with active brain regions is appropriately increased. That is, the gate is a port with edge computing ability. In addition, at the edge of each brain region, a certain number of voxels are selected as the "edge layer" according to a preset distance. The edge layer is used to simulate processes such as cell diffusion and lesion spread. The change of the voxel features can reflect the deformation trend of the brain organ. In this way, the feature fusion module not only integrates multi-modal imaging features but also constructs a complex model that can simulate the brain structure and function, providing richer and more biologically significant information for subsequent diagnostic decisions. This module combines geometric modeling, feature engineering, and neural network technology, aiming to capture the complex pathophysiological processes of brain diseases, thereby improving the accuracy and reliability of diagnosis.

[0072] In one embodiment, the step of fusing the set of brain region features to obtain a fused feature and assigning the fused feature to each voxel based on the normal brain model is as follows:

[0073] Define the preset weights of the features in the set of brain region features, and fuse the features based on the preset weights to obtain a fused feature;

[0074] Read the normal brain model, obtain the brain region to which each voxel belongs, establish the spatial correspondence between the fused features and the voxels and perform spatial mapping, and assign the fused features to the corresponding voxels according to the spatial correspondence;

[0075] If there are voxels containing multiple fused features, calculate the range of the voxel in the sub-region, assign different weights to the fused features based on the range of the occupied sub-region, and perform weighted fusion on the fused features based on the weights;

[0076] To further illustrate, first define preset weights for each feature in the brain region feature set. These weights reflect the importance of different features in diagnosis and can be determined based on prior knowledge, expert experience, or data-driven methods (such as learning weights). Then, based on the preset weights, perform weighted fusion on each feature to obtain a comprehensive fused feature vector. For example, if the weights of anatomical structure features, functional connectivity features, density features, vascular features, and metabolic features are 0.3, 0.25, 0.2, 0.15, and 0.1 respectively, the fused feature vector can be calculated by performing a weighted average on these features. Next, read the normal brain model, which contains the brain region label information of each voxel. By analyzing the normal brain model, the spatial position of each voxel and the brain region to which it belongs can be determined. Then, establish the spatial correspondence between the fused features and the voxels and perform spatial mapping, that is, assign the fused feature vector to the corresponding voxel. If a voxel is completely located within a certain brain region, directly assign the fused feature vector of that brain region to the voxel. However, if a voxel is located at the junction of multiple brain regions, that is, contains multiple fused features, further processing is required. For this case, the scheme calculates the proportion of the voxel in each sub-region and assigns different weights to the fused features based on the proportion of each sub-region. The larger the proportion of the brain region, the higher the weight of the corresponding fused feature. Finally, based on these weights, perform weighted fusion on the multiple fused features to obtain the final feature vector of the voxel. For example, if the proportions of a voxel in brain regions A, B, and C are 1, 2, and 3 respectively, the corresponding weights can be calculated by the softmax function or using a simple inverse proportional function. Then, apply these weights to the fused feature vectors of brain regions A, B, and C and perform weighted average to obtain the final feature vector of the voxel; in this way, the scheme can accurately assign the brain region features to each voxel and consider the spatial relationship between the voxel and multiple brain regions, so as to more accurately describe the local features of the brain tissue. This process combines techniques such as weighted fusion, spatial mapping, and distance calculation, aiming to improve the accuracy and robustness of feature assignment.

[0077] In one embodiment, the step of using the voxels at the pairwise connections of the sub-regions as gates respectively and using the voxels on the outer contours of each brain region as the edge layer of each brain region is as follows:

[0078] Obtain the real-time brain region features of the patient as the visit data;

[0079] Map the visit data corresponding to the brain region features of the patient to the corresponding gates, observe the feature changes of the voxels to obtain voxel features, and input the voxel features into a neural network for quantization to obtain quantized voxel features;

[0080] At the same time, observe the voxel diffusion in the edge layer to predict the deformation of the brain organ to obtain the brain organ deformation trend;

[0081] Take the quantized voxel features and the brain organ deformation trend as the patient features;

[0082] Further explanation: First, obtain the real-time brain region features of the patient and use the real-time brain region features as the visit data; to integrate the visit data into the brain three-dimensional structure model, define the voxels at the joints of each brain region as "gates", and the role of these gates is to receive the feature information from different brain regions and the visit data of the patient; specifically, a neural network such as a multi-layer perceptron can be used to map the visit data of the patient to the corresponding gates, and the mapping method can be simple feature splicing or more complex non-linear transformation; through the gates, the visit data of the patient can affect the feature changes of adjacent voxels; then, observe the feature changes of the voxels, including the structural features, functional connection features, and metabolic features of the voxels, etc., and these feature changes can be quantified by calculating indicators such as the gradient, variance, or entropy of the voxel features; to extract more representative voxel features, input the voxel features into a neural network for quantization to obtain quantized voxel features. The quantization method can be clustering, encoding, or dimensionality reduction, etc.; at the same time, to simulate the diffusion and evolution process of brain diseases, this solution also selects a certain number of voxels at the edge of each brain region at a preset distance, such as 1-2 voxels, as the "edge layer"; the voxels in the edge layer are used to simulate processes such as cell diffusion and lesion spread. By observing the feature changes of the voxels in the edge layer, the deformation trend of the brain organ can be predicted. For example, the structural features and functional connection features of the voxels in the edge layer can be extracted, and polynomial regression or other time series analysis methods can be used to obtain the trend of these features changing over time. If the feature changes of the voxels in the edge layer exceed the preset change threshold, it is determined that the brain organ will deform. Finally, take the quantized voxel features and the brain organ deformation trend as the patient features, and these features will be used for subsequent diagnostic decisions. In this way, this solution not only integrates multi-modal imaging features and the visit data of the patient, but also simulates the pathophysiological process of brain diseases, thereby improving the accuracy and reliability of diagnosis. Using technologies such as data mapping, feature quantization, edge layer analysis, and deformation prediction, it aims to capture the complex pathophysiological process of brain diseases, thereby improving the accuracy and reliability of diagnosis.

[0083] In one embodiment, the steps of predicting the deformation of the brain organ by observing the voxel diffusion of the edge layer are as follows:

[0084] Extract the structural features and functional connection features of the voxels in the edge layer, and use polynomial regression to obtain the trends of changes in the structural features and functional connection features;

[0085] Preset a change threshold. When the structural features and functional connection features of the voxels in the edge layer exceed the change threshold, it is determined that the brain organ will deform;

[0086] Further explanation, the deformation of the brain organ is predicted by analyzing the characteristic changes of the voxels in the edge layer. First, extract the structural features of the voxels in the edge layer, such as volume, surface area, shape, and functional connection features, such as the connection strength with adjacent brain regions. Then, use a polynomial regression model or other time series analysis methods, such as the ARIMA model, to fit the curves of these features changing over time, so as to obtain the trends of changes in the structural features and functional connection features, for example, linear growth, exponential growth, or periodic changes. To determine whether the brain organ will deform, a change threshold is preset, and this threshold can be determined based on prior knowledge, historical data, or statistical analysis; if the change amplitude of the structural features or functional connection features of the voxels in the edge layer exceeds the preset change threshold, it is determined that the brain organ will deform. For example, if the volume of the voxels in the edge layer of the hippocampus significantly shrinks in the short term and exceeds the preset threshold, it is predicted that the hippocampus may atrophy. In this way, this solution can quantitatively evaluate the deformation risk of the brain organ and provide a basis for the early diagnosis and prognosis evaluation of diseases. This method combines techniques such as feature extraction, time series analysis, and threshold judgment, aiming to predict the deformation trend of the brain organ, thereby improving the accuracy and timeliness of diagnosis.

[0087] In one embodiment, the diagnostic decision-making module includes:

[0088] Generate a diagnostic report based on the patient's brain imaging data, medical treatment data, and patient characteristics;

[0089] The diagnostic report includes the patient's basic information, imaging description, and quantitative analysis results, where the quantitative analysis results are the patient characteristics;

[0090] Further elaboration: The diagnostic decision-making module receives the patient's brain imaging data, medical visit data, and previously extracted patient characteristics including quantified voxel features and brain organ deformation trends, and generates a diagnostic report by integrating this information. First, the basic information of the patient, such as name, age, gender, medical history, etc., is collected and added to the diagnostic report. Then, the brain imaging data is described, including the visual features of MRI, CT, and PET images, such as the location, size, shape, density, and metabolic activity of the lesions; these imaging descriptions can be provided by radiologists or generated through automatic image analysis algorithms. Next, the quantitative analysis results such as quantified voxel features and brain organ deformation trends are added to the diagnostic report, and these quantitative features can be compared with known disease patterns to assist in diagnostic decision-making. Finally, based on all this information, a final diagnostic conclusion is generated and corresponding treatment suggestions are provided. The diagnostic conclusion can include the type, severity, and prognosis assessment of the disease, and the treatment suggestions can include drug treatment, surgical treatment, radiotherapy, etc. The diagnostic decision-making module can use various methods to generate diagnostic conclusions and treatment suggestions, such as rule-based reasoning, machine learning algorithms, or deep learning models; the finally generated diagnostic report will contain the patient's basic information, imaging descriptions, and quantitative analysis results, providing comprehensive and objective diagnostic information for clinicians to assist them in making decisions.

[0091] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claimed rights.

Claims

1. An artificial intelligence multi-modal medical image processing and diagnosis system, characterized in that: Data processing module: Collect multi-modal brain image data from multiple angles, extract brain region features based on the multi-modal brain images, and establish a normal brain model; Feature fusion module: Connected to the data processing module, fuse the obtained set of brain region features to generate a three-dimensional structure model of the brain region, and make compensation for the three-dimensional structure model using the patient's medical data to obtain patient features; Diagnostic decision-making module: Connected to the feature fusion module, analyze the patient features to obtain a diagnostic report.

2. The artificial intelligence multi-modal medical image processing diagnosis and treatment system according to claim 1, wherein: The data processing module includes: Collecting multi-modal brain image data from multiple angles includes MRI images, CT images, and positron emission tomography images, and annotating image information including the shooting angle and timestamp; Register the multi-modal brain image data into the same space, and select the MRT image as the reference image; Extract the brain region features of the multi-modal brain image data based on the specificity of the brain image data as a set of brain region features; Establishing a normal brain model includes brain organs, gray matter, white matter, ventricles, and blood vessels, and dividing the brain region into multiple sub-regions based on the normal brain model.

3. An artificial intelligence multi-modal medical image processing and diagnosis and treatment system according to claim 2, characterized in that: The step of extracting the brain region features of the multi-modal brain image data based on the specificity of the brain image data as a set of regional features is: Extract the anatomical structure features and functional connectivity features of the MRI image through a convolutional neural network, and extract the density features and vascular features of the CT image; Among them, the anatomical structure features include the volume, shape, and position of brain organs, gray matter, white matter, and cerebrospinal fluid, the functional connectivity features include the functional connectivity strength between brain regions, the density features include the CT values of brain tissue and lesions, and the vascular features include the diameter, shape, and blood flow velocity of blood vessels; Extract the metabolic features of the positron emission tomography image through a recurrent neural network, and the metabolic features include the metabolic rate and glucose uptake rate of brain tissue; Integrate the anatomical structure features, functional connectivity features, density features, vascular features, and metabolic features to obtain a set of brain region features.

4. An artificial intelligence multi-modal medical image processing and diagnosis and treatment system according to claim 1, characterized in that: The feature fusion module includes: Construct a three-dimensional structure model of the brain region using a neural network algorithm based on the image information and the normal brain model; Divide the three-dimensional structure model into multiple voxels based on a preset volume; Fuse the set of brain region features to obtain fusion features, and distribute the fusion features to each voxel based on the normal brain model; Take the voxels at the joints of the sub-regions in pairs as gates, and take the voxels on the outer contours of each brain region as the edge layers of each brain region, where the gates have functions of receiving data, outputting data, and weight control, and the edge layers are used to predict whether the brain organs will deform.

5. An artificial intelligence multi-modal medical image processing and diagnosis and treatment system according to claim 4, characterized in that: The step of fusing the set of brain region features to obtain fusion features and distributing the fusion features to each voxel based on the normal brain model is: Define the preset weights of the features in the set of brain region features, and fuse the features based on the preset weights to obtain fusion features; Read the normal brain model, obtain the brain regions to which each voxel belongs, establish the spatial correspondence between the fused features and the voxels and perform spatial mapping, and assign the fused features to the corresponding voxels according to the spatial correspondence; If there are voxels containing multiple fused features, calculate the range of the sub-regions occupied by the voxels, assign different weights to the fused features based on the range of the occupied sub-regions, and perform weighted fusion on the fused features based on the weights.

6. An artificial intelligence multi-modal medical image processing and diagnosis and treatment system according to claim 4, characterized in that: The steps of taking the voxels at the joints of the sub-regions pairwise as gates and taking the voxels on the outer contours of the brain regions as the edge layers of the brain regions are as follows: Obtain the real-time brain region features of the patient as the medical data; After mapping the medical data of the patient corresponding to the brain region features to the corresponding gates, observe the feature changes of the voxels to obtain voxel features, and input the voxel features into a neural network for quantization to obtain quantized voxel features; At the same time, observe the voxel diffusion situation of the edge layer to predict the deformation of the brain organs to obtain the deformation trend of the brain organs; Take the quantized voxel features and the deformation trend of the brain organs as the patient features.

7. An artificial intelligence multi-modal medical image processing diagnosis and treatment system according to claim 6, characterized in that: The steps of observing the voxel diffusion situation of the edge layer to predict the deformation of the brain organs are as follows: Extract the structural features and functional connection features of the voxels in the edge layer, and use polynomial regression to obtain the trends of changes in the structural features and functional connection features; Preset a change threshold, and when the structural features and functional connection features of the voxels in the edge layer exceed the change threshold, it is determined that the brain organ will deform.

8. An artificial intelligence multi-modal medical image processing and diagnosis and treatment system according to claim 1, characterized in that: The diagnosis decision module includes: Generate a diagnosis report through the patient's brain imaging data, medical data, and patient features; The diagnosis report includes the patient's basic information, imaging description, and quantitative analysis results, where the quantitative analysis results are the patient features.

Images