Imaging medical multimodal data fusion processing system

Abstract

The application relates to the technical field of image medical treatment, and discloses an image medical multi-modal data fusion processing system. The system comprises a multi-modal data acquisition module, a dynamic PET parameterization analysis module, a dose-image quality optimization module, a multi-target fusion model and a hierarchical processing control module. The multi-modal data acquisition module acquires a dynamic PET time-activity curve, the dynamic PET parameterization analysis module models and generates a dynamic parameterization image; the dose-image quality optimization module constructs a correlation model and outputs an optimal dose allocation scheme; the multi-target fusion model performs feature-level fusion to generate a multi-modal fusion image; and the hierarchical processing control module realizes lesion region segmentation, image edge enhancement and image reconstruction parameter adjustment through a global optimization layer, a local correction layer and an execution layer. The system effectively fuses multi-modal image information, optimizes image quality and improves the accuracy and efficiency of disease diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image medical processing, in particular to an image medical multi-modal data fusion processing system. BACKGROUND

[0002] In the modern medical field, image diagnosis has become one of the key means for disease diagnosis and treatment plan formulation. With the rapid development of technology, various medical imaging modalities have emerged, such as positron emission tomography (PET), computed tomography (CT), magnetic resonance imaging (MRI), etc. Each modality has its unique imaging principle and advantages. However, the image of a single modality often cannot provide sufficient and accurate information, limiting the early diagnosis and precise treatment of diseases.

[0003] PET can reflect the metabolic function of the human body from the molecular level, and has important value in early detection of tumors, efficacy evaluation, etc. However, the spatial resolution of PET images is relatively low, and the anatomical structure is not clear enough, which can easily cause errors in judging the location and range of lesions. For example, when diagnosing brain tumors, it is difficult to accurately distinguish the boundary between tumors and surrounding normal tissues relying only on PET images, which may lead to inaccurate surgical resection range and affect treatment effect.

[0004] CT has high spatial resolution and can clearly present the anatomical structures of human bones, organs, etc. However, CT mainly images based on tissue density differences, and has limited ability to detect some functional lesions. In the diagnosis of cardiovascular diseases, CT is difficult to directly reflect the metabolic status of myocardium, and cannot accurately judge whether there is functional abnormality such as myocardial ischemia.

[0005] MRI performs well in soft tissue imaging and can provide rich soft tissue contrast information. However, MRI examination time is relatively long, and the cooperation of patients is required. The imaging process is easily disturbed by motion artifacts. When examining children or patients who cannot maintain stillness for a long time, motion artifacts may seriously affect image quality, leading to misdiagnosis or missed diagnosis.

[0006] In addition, some problems in the image acquisition process also need to be solved. For example, PET imaging requires the use of radioactive drugs, and the selection of drug dosage has always been a difficult problem. If the dosage is too high, it will increase the risk of radiation exposure for patients and harm their health; if the dosage is too low, it will lead to decreased image quality, increased noise, reduced lesion contrast, and affect diagnostic accuracy. Currently, the determination of radioactive drug dosage in clinical practice often lacks scientific and precise methods, and mostly relies on the experience of doctors, which makes it difficult to ensure that each examination can obtain the best image quality and diagnostic effect.

[0007] In the aspect of image processing and analysis, the traditional method is mostly for processing single modality image, which cannot fully utilize the complementary information of multi-modality image. Moreover, in the key links of lesion region segmentation, image edge enhancement and image reconstruction parameter optimization, there are many deficiencies in the existing technical means. The traditional lesion segmentation algorithm has low accuracy and is prone to segmentation errors or omissions; the image edge enhancement method may introduce noise, resulting in loss of image details; the adjustment of image reconstruction parameters lacks effective strategies, which cannot guarantee the reconstruction quality while taking into account the calculation efficiency. SUMMARY

[0008] The purpose of the present application is to provide an image medical multi-modality data fusion processing system to solve the problems raised in the background art.

[0009] To achieve the above purpose, the present application provides the following technical scheme: an image medical multi-modality data fusion processing system, the system comprising:

[0010] A multi-modality data acquisition module is used to acquire dynamic PET time-activity curves through a dynamic PET scanner;

[0011] A dynamic PET parameterization analysis module is used to parameterize modeling of the dynamic PET time-activity curves based on a hemodynamic model, extract blood flow features using a convolutional neural network, and generate dynamic parameterization images by fusing hemodynamic parameters at multiple time points through a time series attention mechanism;

[0012] A dose-image quality optimization module is used to construct a correlation model of radioactive drug dose and image quality, generate simulated PET images under low dose using a generative adversarial network, optimize a dose-image quality trade-off function through a reinforcement learning strategy, and output an optimal dose allocation scheme;

[0013] A multi-objective fusion model is used to perform feature-level fusion of multi-modality image data based on an improved multi-objective particle swarm optimization algorithm, which introduces adaptive inertia weight and dynamic learning factor to optimize the fusion weight matrix and generate multi-modality fusion images;

[0014] A hierarchical processing control module is used to adopt a hierarchical adaptive control strategy, including a global optimization layer, a local correction layer and an execution layer. The global optimization layer performs lesion region segmentation based on multi-modality fusion images, the local correction layer performs image edge enhancement based on the segmentation results, and the execution layer adjusts image reconstruction parameters through a model predictive control algorithm.

[0015] Preferably, the parameterization modeling of the dynamic PET time-activity curves based on the hemodynamic model comprises:

[0016] A two-compartment model is constructed to describe the metabolism of the radiopharmaceutical in the tissue, the two-compartment model comprising a plasma compartment and a tissue compartment, and the rate of change of the drug concentration is calculated by a differential equation;

[0017] The convolutional neural network comprises a time convolution layer, a max pooling layer and a full connection layer, the time convolution layer has a convolution kernel size of 3x1 and a step length of 1;

[0018] The multi-head attention mechanism is used to fuse blood flow parameters at different time points, the importance weight of each time point parameter is calculated, and a fused dynamic parameterized image is generated;

[0019] A post-processing network of the parameterized image is constructed, the residual structure is used to suppress image noise, and the channel attention mechanism is used to enhance the weight of the blood flow feature channel.

[0020] Preferably, the constructed association model of the radiopharmaceutical dose and the image quality comprises:

[0021] A dose-image quality trade-off function is defined, the trade-off function comprises a dose penalty term, a noise level term and a lesion contrast term, wherein the dose penalty term is calculated by the square sum of the injection dose, the noise level term is calculated by the standard deviation of the PET image, and the lesion contrast term is calculated by the SUV ratio of the lesion region to the background;

[0022] A generative adversarial network is constructed, the generator takes low-dose PET projection data as input and outputs high-dose simulation images, and the discriminator adopts a multi-scale convolution structure to distinguish the authenticity of the images;

[0023] A deep deterministic policy gradient algorithm is used to optimize the dose allocation strategy, the state space is the current dose distribution and the image quality index, the action space is the dose adjustment amount, and the reward function is based on the weighted combination of the trade-off function.

[0024] Preferably, the feature-level fusion of multi-modal image data based on the improved multi-objective particle swarm algorithm comprises:

[0025] A multi-objective optimization function is defined, comprising a feature similarity term, an information entropy term and a modality complementarity term, the feature similarity term is calculated by the structural similarity of the feature maps between the modalities, the information entropy term is calculated by the information entropy value of the fused image, and the modality complementarity term is calculated by the covariance matrix of the feature difference between the modalities;

[0026] An adaptive inertia weight update rule is designed, the adaptive inertia weight exponentially decays with the iteration number, and the decay factor is negatively correlated with the particle diversity index;

[0027] A non-dominated sorting strategy is used to screen the Pareto optimal solution set, and the optimal fusion weight matrix is ​​selected from the solution set by the entropy weight method.

[0028] Preferably, the global optimization layer performs lesion region segmentation based on multimodal fusion images, including:

[0029] A U-Net++ segmentation network is constructed, which includes an encoder-decoder structure and dense skip connections. The encoder uses ResNet-50 to extract multi-scale features, and the decoder gradually restores the resolution through transposed convolution.

[0030] Design a multi-task loss function, including Dice loss, edge-aware loss, and lesion shape constraint term. The edge-aware loss calculates the gradient difference of the segmentation boundary using the Sobel operator, and the lesion shape constraint term is measured by the compactness of the lesion region.

[0031] A course-based learning strategy is used to train the segmentation network. In the initial stage, the focus is on learning global features, and the weights of complex samples are gradually increased.

[0032] Preferably, the local correction layer enhances image edges based on the segmentation results, including:

[0033] An edge-guided filtering network is constructed, which is guided by a segmentation mask and uses deformable convolution to adaptively adjust the shape of the filtering kernel;

[0034] The edge enhancement loss function is designed, including a high-frequency component retention term and a smoothing term. The high-frequency component retention term is calculated using the Laplacian operator response of the image, and the smoothing term is calculated using the gray-level difference between adjacent pixels.

[0035] An iterative optimization algorithm is used to alternately update the filter network parameters and edge enhancement weights until the convergence condition is met.

[0036] Preferably, the execution layer adjusts the image reconstruction parameters through a model prediction control algorithm, including:

[0037] Establish a state-space model of image reconstruction parameters. The state variables include the number of iterations, the regularization coefficient, and the filter kernel size. The control variables are the parameter adjustment amounts.

[0038] Construct a prediction cost function, including reconstruction error term, parameter stability term and computational efficiency term, and solve for the optimal control sequence through quadratic programming;

[0039] A rolling time-domain strategy is adopted to dynamically update the parameter adjustment amount, and only the first control instruction is executed in each optimization.

[0040] Preferably, the calculation of the drug concentration change rate using a differential equation includes:

[0041] The plasma chamber concentration equation is defined as a convolution integral of the input function and the clearance rate, and the tissue chamber concentration equation is the product of the blood flow rate and the extraction fraction;

[0042] The fourth-order Runge-Kutta method is used to numerically solve the differential equation set, and the time step is adaptively adjusted according to the curve slope;

[0043] The model parameters are estimated by the Bayesian optimization algorithm, the parameter posterior probability distribution is constructed, and the parameter estimation value is generated by Markov chain Monte Carlo sampling.

[0044] Preferably, the generator structure of the generative adversarial network comprises:

[0045] The input layer receives the sinogram of the low-dose PET projection data;

[0046] The encoder part adopts 3 layers of convolution layers to compress the feature dimension, and each layer of convolution is followed by instance normalization and LeakyReLU activation;

[0047] The decoder part adopts 3 layers of transpose convolution layers to restore the spatial resolution, and the last layer uses the Tanh activation function to constrain the output range;

[0048] The spectral normalization technique is introduced to stabilize the generator training process, and the weight matrix spectral norm of the convolution layer is limited.

[0049] Preferably, the non-dominated sorting strategy for screening the Pareto optimal solution set comprises:

[0050] The dominance relationship of each particle is calculated, and the non-dominated individual is marked as the first front layer;

[0051] The solutions in the same front layer are sorted by using the crowding degree sorting algorithm, and the crowding degree is calculated by the difference in the hyper volume of adjacent solutions in the objective space;

[0052] An elite reservation strategy is designed, and the top 20% of non-dominated solutions in each generation are reserved into the next generation population.

[0053] Compared with the prior art, the beneficial effects of the present application are:

[0054] At the data acquisition and analysis level, the multi-modal data acquisition module cooperates with the dynamic PET parameterization analysis module to deeply mine the value of PET data. By constructing a double-chamber model to describe the metabolism of radiopharmaceuticals and using the fourth-order Runge-Kutta method to solve the differential equation, the drug concentration change rate can be accurately obtained, and the Bayesian optimization algorithm is used to estimate the model parameters to ensure the accuracy of modeling. The convolutional neural network and the time sequence attention mechanism work together to effectively extract blood flow features and fuse multiple time point parameters to generate high-quality dynamic parameterization images, providing doctors with more rich metabolic information, which helps to early detect small lesions and improve the sensitivity of disease diagnosis.

[0055] The dose-image quality optimization module plays a key role in ensuring patient safety and image quality. The constructed radiopharmaceutical dose and image quality correlation model comprehensively considers factors such as dose penalty, noise level, and lesion contrast by defining a scientific trade-off function. The generative adversarial network can generate simulated PET images at low dose, and the deep deterministic policy gradient algorithm optimizes the dose allocation strategy, reducing the potential harm of radiation to the patient's body while maintaining or even improving image quality, making PET examination safer and more reliable.

[0056] The multi-objective fusion model is based on an improved multi-objective particle swarm optimization algorithm to realize feature-level fusion of multi-modal images. The adaptive inertia weight and dynamic learning factor optimize the fusion weight matrix, and the multi-objective optimization function considers feature similarity, information entropy, and modality complementarity to generate multi-modal fusion images that can fully integrate the advantages of different modalities. This not only improves image clarity and information quantity, but also enhances the doctor's ability to identify complex lesions, helping to more accurately determine the nature, location, and extent of the lesion, reducing misdiagnosis and missed diagnosis.

[0057] The global optimization layer of the hierarchical processing control module uses the U-Net++ segmentation network and multi-task loss function, combined with the curriculum learning strategy, to accurately segment the lesion area. The local correction layer uses an edge-oriented filtering network and iterative optimization algorithm to enhance image edges without introducing noise, highlighting lesion details. The execution layer uses model predictive control algorithm to establish a reasonable state space model and prediction cost function, dynamically updating image reconstruction parameters to improve reconstruction quality and computational efficiency, providing strong support for clinical rapid diagnosis.

[0058] Overall, the system realizes efficient fusion and processing of multi-modal data through the collaborative work of each module, providing doctors with more comprehensive, accurate, and clear image information, improving the accuracy and efficiency of disease diagnosis, and assisting the development of precision medicine. It has broad prospects in clinical applications and can effectively improve patient medical experience and treatment effect. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The working principle diagram of the image medical multi-modal data fusion processing system described in the present application;

[0060] Figure 2 The working principle diagram of the dynamic PET parameterization analysis module;

[0061] Figure 3 The flowchart of the global optimization layer;

[0062] Figure 4 The working principle diagram of the image reconstruction parameter execution layer dynamic adjustment. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Please see Figures 1-4 This invention provides a multimodal data fusion processing system for medical imaging, the overall implementation of which is as follows:

[0065] The system first operates through a multimodal data acquisition module, using a dynamic PET scanner to acquire dynamic PET time-activity curves. These curves contain information on the metabolism of the radiopharmaceutical in vivo over time, serving as crucial foundational data for subsequent analysis.

[0066] After acquiring dynamic PET time-activity curves, the dynamic PET parametric analysis module performs parametric modeling on the acquired curves based on a hemodynamic model. It uses a convolutional neural network to extract blood flow features and fuses hemodynamic parameters from multiple time points using a temporal attention mechanism, ultimately generating dynamic parametric images. These images can more intuitively display hemodynamic changes within tissues, providing doctors with richer diagnostic information.

[0067] The dose-image quality optimization module constructs a correlation model between radiopharmaceutical dose and image quality. It generates simulated PET images at low doses using a generative adversarial network (GAN), and then optimizes the dose-image quality tradeoff function using reinforcement learning to output the optimal dose allocation scheme. This approach ensures that image quality meets diagnostic requirements while minimizing the radiopharmaceutical dose received by the patient, thus reducing radiation hazards.

[0068] A multi-objective fusion model processes multimodal image data. This model performs feature-level fusion based on an improved multi-objective particle swarm optimization algorithm, optimizing the fusion weight matrix by introducing adaptive inertial weights and dynamic learning factors, thereby generating a multimodal fused image. This fused image integrates the advantages of multiple modalities, providing more comprehensive and accurate medical information.

[0069] Finally, the hierarchical processing control module further processes the multi-modal fusion image. The module adopts a hierarchical adaptive control strategy, including a global optimization layer, a local correction layer, and an execution layer. The global optimization layer performs lesion region segmentation based on the multi-modal fusion image to determine the approximate position and range of the lesion; the local correction layer performs image edge enhancement according to the segmentation result to make the lesion edge clearer; and the execution layer adjusts the image reconstruction parameters through a model predictive control algorithm to optimize the reconstruction effect of the image and improve the image quality.

[0070] The implementation of the present application will be further described below in connection with Examples 1 to 5.

[0071] Example 1

[0072] This example mainly focuses on the parametric modeling of dynamic PET time-activity curves based on a hemodynamic model in the dynamic PET parametric analysis module. First, a two-compartment model is constructed to describe the metabolism of the radiopharmaceutical in the tissue, which consists of a plasma compartment and a tissue compartment. In the plasma compartment, the change in drug concentration is affected not only by the input function but also by the clearance rate. Assuming that the plasma compartment concentration is , the input function is , and the clearance rate is , the plasma compartment concentration equation is: where the integral represents the cumulative effect of the product of the input function and the clearance rate exponential function from the start to time In the tissue compartment, assuming that the tissue compartment concentration is , the blood flow rate is , and the extraction fraction is , the tissue compartment concentration equation is which reflects the relationship between the tissue compartment concentration and the plasma compartment concentration, as well as the blood flow rate and the extraction fraction. The fourth-order Runge-Kutta method is used to numerically solve the above differential equation set, and in the solving process, the time step is adaptively adjusted according to the curve slope. If the curve slope changes greatly, it means that the drug concentration changes rapidly, and in this case, the time step is reduced to improve the solving accuracy; otherwise, the time step is appropriately increased to improve the calculation efficiency. At the same time, the model parameters are estimated by the Bayesian optimization algorithm to construct the parameter posterior probability distribution, and the Markov Chain Monte Carlo sampling is used to generate parameter estimation values. After completing the model construction, a convolutional neural network is used to extract features from the time-activity curve. The convolutional neural network includes a time series convolution layer, a max pooling layer, and a fully connected layer, where the size of the convolution kernel of the time series convolution layer is and the step size is The time convolution layer extracts features on the time sequence by convolving the time-activity curve in the time dimension with a specific size of convolution kernel. The max-pooling layer filters the features after convolution, retains the main features, and reduces the data volume. The fully connected layer integrates the processed features and outputs the feature vector. Then, the multi-head attention mechanism is used to fuse the blood flow parameters at different time points. The multi-head attention mechanism calculates the importance weight of each time point parameter. Assuming that the weight of the first time point is , the fused dynamic parameterized image is generated by weighted sum of parameters at different time points, i.e. . Finally, the post-processing network of the parameterized image is constructed. The residual structure is used to suppress image noise, and the channel attention mechanism is used to enhance the weight of the blood flow feature channel, further optimizing the image quality.

[0073] Embodiment 2

[0074] This embodiment focuses on the construction of the correlation model between the dose of radiopharmaceuticals and the image quality in the dose-image quality optimization module. First, the dose-image quality trade-off function is defined, which includes the dose penalty term, the noise level term and the lesion contrast term. Let the injection dose be , the dose penalty term is calculated by the square sum of the injection dose, i.e. the dose penalty term , where represents the dose of each injection, and the square sum of all injection doses is calculated to punish high dose. Assuming that the standard deviation of the PET image is , the noise level term is calculated by the standard deviation of the PET image, i.e. the noise level term , the larger the standard deviation, the more the image noise. Assuming that the SUV value of the lesion area is , and the SUV value of the background is , the lesion contrast term is calculated by the SUV ratio of the lesion area to the background, i.e. the lesion contrast term , the larger the ratio, the higher the distinction between the lesion and the background. The trade-off function is the combination of the three terms, as shown in , where , , ​are weight coefficients used to adjust the importance of each term in the trade-off function. Then a generative adversarial network is constructed, the generator takes low-dose PET projection data as input and outputs high-dose simulated images. The structure of the generator includes: the input layer receives the sinogram of low-dose PET projection data; the encoder part adopts 3 layers of convolutional layers to compress the feature dimension, and each layer of convolution is followed by instance normalization and LeakyReLU activation; the decoder part adopts 3 layers of transpose convolutional layers to restore the spatial resolution, and the last layer uses the Tanh activation function to constrain the output range. The spectral normalization technique is introduced to stabilize the generator training process, and the spectral norm of the weight matrix of the convolutional layer is limited. The discriminator adopts a multi-scale convolutional structure to distinguish the authenticity of the image. The deep deterministic policy gradient algorithm is used to optimize the dose allocation strategy, the state space is the current dose distribution and the image quality index, the action space is the dose adjustment amount, and the reward function is based on the weighted combination of the trade-off function. Through continuous iteration optimization, the optimal dose allocation scheme is found.

[0075] Embodiment 3

[0076] This embodiment details the process of feature-level fusion of multi-modal image data based on an improved multi-objective particle swarm optimization algorithm in a multi-objective fusion model. First, a multi-objective optimization function is defined, including a feature similarity term, an information entropy term, and a modality complementarity term. Let the feature maps of modalities and modalities be and respectively, the feature similarity term is calculated by the structural similarity of the feature maps between modalities, and the structural similarity index is calculated as , where , are the mean values of the feature maps , , , are the standard deviations of the feature maps , , is the covariance of and , , are constants used to maintain the stability of the calculation. The information entropy term is calculated by the information entropy value of the fused image, the greater the information entropy, the more information the image contains. Let the feature difference of modalities and modalities be , the modality complementarity term is calculated by the covariance matrix of the feature difference between modalities, assuming that the covariance matrix is , then the modality complementarity term is related to . An adaptive inertia weight update rule is designed, and the inertia weight decays exponentially with the number of iterations. The decay factor The particle diversity index is negatively correlated with the particle diversity index, The inertia weight decays more slowly, which can maintain the global search ability of particles in the early stage of the algorithm and gradually enhance the local search ability in the later stage. The non-dominated sorting strategy is used to screen the Pareto optimal solution set, calculate the dominance relationship of each particle, and mark the non-dominated individuals as the first front layer. The crowding degree sorting algorithm is used to sort the solutions in the same front layer, and the crowding degree is calculated by the difference in hyper volume of adjacent solutions in the target space. An elite reservation strategy is designed to reserve the top 20% non-dominated solutions in each generation to enter the next generation population. Finally, the optimal fusion weight matrix is selected from the solution set by entropy weight method.

[0077] Embodiment 4:

[0078] This embodiment is used to describe the implementation of the global optimization layer in the hierarchical processing control module based on multi-modal fusion image for lesion region segmentation. Specifically, it includes:

[0079] A U-Net++ segmentation network is constructed, which includes an encoder-decoder structure and dense skip connection. The encoder uses ResNet-50 to extract multi-scale features. ResNet-50 extracts features from the input multi-modal fusion image through a series of convolutional layers and residual blocks, which can obtain feature information at different scales of the image. The decoder gradually restores the resolution through transposed convolution, upsamples the features extracted by the encoder, and finally outputs the segmentation result. A multi-task loss function is designed, including Dice loss, edge perception loss and lesion shape constraint term. Let the segmentation result be , the true label be , the Dice loss be This loss function is used to measure the overlap between the segmentation result and the true label. The closer the value is to 1, the better the segmentation effect. The Sobel operator is used to calculate the gradient difference of the segmentation boundary, and the edge perception loss is designed accordingly. Assuming that the boundary gradient of the segmentation result calculated by the Sobel operator is , the true label boundary gradient is , and the edge perception loss is This loss function can make the segmentation boundary closer to the true boundary. The shape constraint term measures the compactness of the lesion region. Assuming that the area of the lesion region is , the perimeter is , and the shape constraint term is The closer the value is to 1, the closer the lesion region is to a circle, i.e., the more regular the shape is. A curriculum learning strategy is used to train the segmentation network. In the initial stage, focus on global feature learning, give simple samples greater weight, and gradually increase the weight of complex samples as the training progresses, so that the network can learn more complex features and improve the segmentation accuracy.

[0080] Example 5:

[0081] This embodiment mainly introduces the process that the local correction layer in the hierarchical processing control module performs image edge enhancement based on the segmentation result and the execution layer adjusts the image reconstruction parameters through the model predictive control algorithm. Specifically, it includes:

[0082] The local correction layer constructs an edge-oriented filtering network. The network is guided by the segmentation mask and uses deformable convolution to adaptively adjust the shape of the filter kernel. Deformable convolution can dynamically adjust the sampling position of the convolution kernel according to the local features of the image, better capturing edge information. An edge enhancement loss function is designed, including a high-frequency component preservation term and a smoothing term. Let the image be , the high-frequency component preservation term is calculated by the image Laplacian operator response. The Laplacian operator is sensitive to the high-frequency components of the image. The high-frequency component preservation term , the larger the value, the better the preservation of the high-frequency components of the image. The smoothing term is calculated by the gray difference of adjacent pixels. Assuming that the gray values of adjacent pixels are and , the smoothing term is , the smaller the value, the smoother the image. An iterative optimization algorithm is used to alternately update the filtering network parameters and edge enhancement weights until the convergence condition is met, so that the image edge is enhanced. The execution layer establishes a state space model of the image reconstruction parameters. The state variables include the number of iterations, the regularization coefficient and the filter kernel size, and the control variable is the parameter adjustment amount. A prediction cost function is constructed, including a reconstruction error term, a parameter stability term and a computational efficiency term. Let the difference between the reconstructed image and the original image be , the reconstruction error term is ; let the change before and after the parameter adjustment be , the parameter stability term is ; the computational efficiency term is related to the calculation time, and the optimal control sequence is solved by quadratic programming. A rolling horizon strategy is used to dynamically update the parameter adjustment amount. Only the first step control instruction is executed at each optimization, and the image reconstruction parameters are constantly adjusted to optimize the image reconstruction effect.

[0083] It should be noted that in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0084] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. An image medical multi-modality data fusion processing system, characterized by, include: The multimodal data acquisition module is used to acquire dynamic PET time-activity curves using a dynamic PET scanner; The dynamic PET parametric analysis module parametrically models the dynamic PET time-activity curve based on a hemodynamic model, extracts blood flow features using a convolutional neural network, and fuses hemodynamic parameters from multiple time points through a temporal attention mechanism to generate a dynamic parametric image. The dose-image quality optimization module constructs a correlation model between radiopharmaceutical dose and image quality, uses generative adversarial networks to generate simulated PET images under low doses, optimizes the dose-image quality trade-off function through reinforcement learning strategies, and outputs the optimal dose allocation scheme. A multi-objective fusion model is based on an improved multi-objective particle swarm optimization algorithm to perform feature-level fusion of multimodal image data. The improved multi-objective particle swarm optimization algorithm introduces adaptive inertial weights and dynamic learning factors to optimize the fusion weight matrix and generate a multimodal fused image. The hierarchical processing control module adopts a hierarchical adaptive control strategy, including a global optimization layer, a local correction layer, and an execution layer. The global optimization layer performs lesion region segmentation based on multimodal fusion images, the local correction layer enhances image edges based on the segmentation results, and the execution layer adjusts image reconstruction parameters through a model prediction control algorithm.

2. The image medical multi-modality data fusion processing system of claim 1, wherein, The parametric modeling of the dynamic PET time-activity curve based on the hemodynamic model includes: A two-compartment model was constructed to describe the metabolic process of radiopharmaceuticals in tissues. The two-compartment model includes a plasma compartment and a tissue compartment. The rate of change of drug concentration was calculated using differential equations. A convolutional neural network is used to extract features from the time-activity curve. The convolutional neural network includes a temporal convolutional layer, a max pooling layer, and a fully connected layer. The kernel size of the temporal convolutional layer is 3×1, and the stride is 1. By fusing blood flow parameters at different time points through a multi-head attention mechanism, the importance weight of each parameter at each time point is calculated, and a fused dynamic parameterized image is generated. A post-processing network for parameterized images is constructed, and residual structures are used to suppress image noise. The weights of blood flow feature channels are enhanced through a channel attention mechanism.

3. The image medical multi-modality data fusion processing system of claim 1, wherein, The construction of the correlation model between radiopharmaceutical dosage and image quality includes: Define a dose-image quality tradeoff function, which includes a dose penalty term, a noise level term, and a lesion contrast term. The dose penalty term is calculated by the sum of squares of the injection dose, the noise level term is calculated by the standard deviation of the PET image, and the lesion contrast term is calculated by the SUV ratio of the lesion area to the background. A generative adversarial network is constructed. The generator takes low-dose PET projection data as input and outputs high-dose simulated images. The discriminator uses a multi-scale convolutional structure to determine the authenticity of the images. A deep deterministic gradient algorithm is used to optimize the dose allocation strategy. The state space is constructed as the current dose distribution and image quality index, the action space is the dose adjustment amount, and the reward function is based on a weighted combination of trade-off functions.

4. The image medical multi-modality data fusion processing system of claim 1, wherein, Feature-level fusion of multimodal image data based on an improved multi-objective particle swarm optimization algorithm includes: A multi-objective optimization function is defined, including a feature similarity term, an information entropy term and a modality complementarity term, the feature similarity term is calculated by the structural similarity of inter-modal feature maps, the information entropy term is calculated by the information entropy value of the fused image, and the modality complementarity term is calculated by the covariance matrix of inter-modal feature difference; An adaptive inertia weight update rule is designed, the adaptive inertia weight exponentially decays with the iteration number, and the decay factor is negatively correlated with the particle diversity index; A non-dominated sorting strategy is used to screen the Pareto optimal solution set, and the optimal fusion weight matrix is selected from the solution set by the entropy weight method.

5. The image medical multi-modality data fusion processing system of claim 1, wherein, The global optimization layer performs lesion region segmentation based on the multi-modal fusion image, including: A U-Net++ segmentation network is constructed, which contains an encoder-decoder structure and dense skip connection, the encoder uses ResNet-50 to extract multi-scale features, and the decoder gradually restores the resolution through transposed convolution; A multi-task loss function is designed, including Dice loss, edge perception loss and lesion shape constraint term, the edge perception loss is calculated by the Sobel operator to calculate the gradient difference of the segmentation boundary, and the lesion shape constraint term is calculated by the compactness of the lesion region; A curriculum learning strategy is used to train the segmentation network, which focuses on global feature learning in the initial stage and gradually increases the weight of complex samples.

6. The image medical multi-modality data fusion processing system of claim 1, wherein, The local correction layer performs image edge enhancement based on the segmentation result, including: An edge-oriented filtering network is constructed, which uses a deformable convolution to adaptively adjust the filter kernel shape guided by the segmentation mask; An edge enhancement loss function is designed, including a high frequency component preservation term and a smoothing term, the high frequency component preservation term is calculated by the image Laplacian operator response, and the smoothing term is calculated by the gray difference of adjacent pixels; An iterative optimization algorithm is used to alternately update the filtering network parameters and the edge enhancement weight until the convergence condition is met.

7. The image medical multi-modality data fusion processing system of claim 1, wherein, The execution layer adjusts the image reconstruction parameters through the model predictive control algorithm, including: A state space model of the image reconstruction parameters is established, the state variables include the iteration number, the regularization coefficient and the filter kernel size, and the control variable is the parameter adjustment amount; A prediction cost function is constructed, including a reconstruction error term, a parameter stability term and a computational efficiency term, and the optimal control sequence is solved by quadratic programming; A rolling horizon strategy is used to dynamically update the parameter adjustment amount, and only the first step control instruction is executed at each optimization.

8. The image medical multi-modality data fusion processing system of claim 2, wherein, The drug concentration change rate is calculated by a differential equation, including: The plasma compartment concentration equation is defined as the convolution integral of the input function and the clearance rate, and the tissue compartment concentration equation is the product of the blood flow rate and the extraction fraction; A fourth-order Runge-Kutta method is used to numerically solve the differential equation set, and the time step is adaptively adjusted according to the curve slope; A Bayesian optimization algorithm is used to estimate the model parameters, a posterior probability distribution of the parameters is constructed, and parameter estimation values are generated by Markov chain Monte Carlo sampling.

9. The image medical multi-modality data fusion processing system of claim 3, wherein, The generator structure of the generative adversarial network includes: The input layer receives the sinogram of the low-dose PET projection data; The encoder part uses 3 layers of convolution layers to compress the feature dimension, and each layer of convolution is followed by instance normalization and LeakyReLU activation; The decoder part adopts 3 layers of transpose convolution layers to recover the spatial resolution, and the last layer uses Tanh activation function to constrain the output range; Spectral normalization technique is introduced to stabilize the generator training process and limit the spectral norm of the weight matrix of the convolution layer.

10. The image medical multi-modality data fusion processing system of claim 4, wherein, The non-dominated sorting strategy for screening the Pareto optimal solution set includes: Calculate the dominance relationship of each particle, and mark the non-dominated individuals as the first front layer; Use the crowding distance sorting algorithm to sort the solutions in the same front layer, and the crowding distance is calculated by the difference in hyper volume of adjacent solutions in the objective space; Design an elite reservation strategy, and reserve the top 20% non-dominated solutions of each generation into the next generation population.

Images