Imaging medicine multi-modal data fusion processing system

Abstract

The invention relates to the technical field of imaging medicine processing, and discloses an imaging medicine multi-modal data fusion processing system. The system comprises a multi-modal data acquisition module, a dynamic PET parameterization analysis module, a dose-image quality optimization module, a multi-target fusion model and a hierarchical processing control module. The multi-modal data acquisition module is used for acquiring a dynamic PET time-activity curve, and the dynamic PET parameterization analysis module is used for modeling to generate a dynamic parameterization image; the dose-image quality optimization module constructs a correlation model and outputs an optimal dose distribution scheme; performing feature level fusion on the multi-target fusion model to generate a multi-modal fusion image; and the hierarchical processing control module realizes focus region segmentation, image edge enhancement and image reconstruction parameter adjustment through a global optimization layer, a local correction layer and an execution layer. The system effectively fuses the multi-modal image information, optimizes the image quality, and improves the accuracy and efficiency of disease diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image medical processing, in particular to an image medical multi-modal data fusion processing system. BACKGROUND

[0002] In the modern medical field, image diagnosis has become one of the key means for disease diagnosis and treatment plan formulation. With the rapid development of technology, various medical imaging modalities have emerged, such as positron emission tomography (PET), computed tomography (CT), magnetic resonance imaging (MRI), etc. Each modality has its unique imaging principle and advantages. However, the image of a single modality often cannot provide sufficient and accurate information, limiting the early diagnosis and precise treatment of diseases.

[0003] PET can reflect the metabolic function of the human body from the molecular level, and has important value in early detection of tumors, efficacy evaluation, etc. However, the spatial resolution of PET images is relatively low, and the anatomical structure is not clear enough, which can easily cause errors in judging the location and range of lesions. For example, when diagnosing brain tumors, it is difficult to accurately distinguish the boundary between tumors and surrounding normal tissues relying on PET images alone, which may lead to inaccurate surgical resection range and affect treatment effect.

[0004] CT has high spatial resolution and can clearly present the anatomical structures of human bones, organs, etc. However, CT mainly images based on tissue density differences, and has limited ability to detect some functional lesions. In the diagnosis of cardiovascular diseases, CT cannot directly reflect the metabolic status of myocardium and accurately judge whether there is functional abnormality such as myocardial ischemia.

[0005] MRI performs well in soft tissue imaging and can provide rich soft tissue contrast information. However, MRI examination time is relatively long, and the cooperation of patients is required. The imaging process is easily disturbed by motion artifacts. When examining children or patients who cannot maintain stillness for a long time, motion artifacts may seriously affect image quality, leading to misdiagnosis or missed diagnosis.

[0006] In addition, some problems in the image acquisition process also need to be solved. For example, PET imaging requires the use of radioactive drugs, and the selection of drug dosage has always been a difficult problem. If the dosage is too high, it will increase the risk of radiation exposure for patients and harm their health; if the dosage is too low, it will lead to decreased image quality, increased noise, reduced lesion contrast, and affect diagnostic accuracy. Currently, the determination of radioactive drug dosage in clinical practice often lacks scientific and precise methods, and mostly relies on the experience of doctors, which makes it difficult to ensure that each examination can obtain the best image quality and diagnostic effect.

[0007] In the aspect of image processing and analysis, the traditional method is mostly for processing single modality image, which cannot fully utilize the complementary information of multi-modality image. Moreover, in the key links of lesion region segmentation, image edge enhancement and image reconstruction parameter optimization, the existing technical means also have many deficiencies. The traditional lesion segmentation algorithm has low accuracy and is prone to segmentation errors or omissions; the image edge enhancement method may introduce noise, resulting in loss of image details; the adjustment of image reconstruction parameters lacks effective strategies, which cannot guarantee the reconstruction quality while taking into account the calculation efficiency. SUMMARY

[0008] The purpose of the present application is to provide an image medical multi-modality data fusion processing system to solve the problems raised in the background art.

[0009] To achieve the above purpose, the present application provides the following technical scheme: an image medical multi-modality data fusion processing system, the system comprising: a multi-modality data acquisition module for acquiring dynamic PET time-activity curves through a dynamic PET scanner; a dynamic PET parameterization analysis module for parameterizing modeling of the dynamic PET time-activity curves based on a hemodynamic model, extracting blood flow features using a convolutional neural network, and generating dynamic parameterization images by fusing hemodynamic parameters at multiple time points through a time series attention mechanism; a dose-image quality optimization module for constructing a correlation model of radioactive drug dose and image quality, generating simulated PET images under low dose using a generative adversarial network, optimizing a dose-image quality trade-off function through a reinforcement learning strategy, and outputting an optimal dose allocation scheme; a multi-objective fusion model for feature-level fusion of multi-modality image data based on an improved multi-objective particle swarm optimization algorithm, which introduces adaptive inertia weight and dynamic learning factor to optimize the fusion weight matrix and generate multi-modality fusion images; a hierarchical processing control module using a hierarchical adaptive control strategy, including a global optimization layer, a local correction layer and an execution layer, the global optimization layer performing lesion region segmentation based on multi-modality fusion images, the local correction layer performing image edge enhancement based on the segmentation results, and the execution layer adjusting image reconstruction parameters through a model predictive control algorithm.

[0010] Preferably, the parameterization modeling of the dynamic PET time-activity curves based on the hemodynamic model comprises: a two-compartment model is constructed to describe the metabolism of radioactive drugs in tissues, the two-compartment model including a plasma compartment and a tissue compartment, and the rate of change of drug concentration is calculated by a differential equation; The time-activity curve is extracted by a convolutional neural network, the convolutional neural network comprising a time convolution layer, a max pooling layer and a full connection layer, the time convolution layer having a kernel size of 3x1 and a step size of 1; The blood flow parameters at different time points are fused by a multi-head attention mechanism, the importance weight of each time point parameter is calculated, and a fused dynamic parameterized image is generated. A post-processing network of the parameterized image is constructed, residual structure is used to suppress image noise, and the weight of the blood flow feature channel is enhanced through the channel attention mechanism.

[0011] Preferably, the correlation model of the radio pharmaceutical dose and the image quality comprises: A dose-image quality trade-off function is defined, the trade-off function comprising a dose penalty term, a noise level term and a lesion contrast term, wherein the dose penalty term is calculated by the square sum of the injection dose, the noise level term is calculated by the standard deviation of the PET image, and the lesion contrast term is calculated by the SUV ratio of the lesion region to the background; A generative adversarial network is constructed, the generator takes low-dose PET projection data as input and outputs high-dose simulation images, and the discriminator uses a multi-scale convolution structure to distinguish the authenticity of the images; A deep deterministic policy gradient algorithm is used to optimize the dose allocation strategy, the state space is the current dose distribution and the image quality index, the action space is the dose adjustment amount, and the reward function is based on the weighted combination of the trade-off function.

[0012] Preferably, the feature-level fusion of multi-modal image data based on the improved multi-objective particle swarm algorithm comprises: A multi-objective optimization function is defined, comprising a feature similarity term, an information entropy term and a modality complementarity term, the feature similarity term is calculated by the structural similarity of the inter-modal feature map, the information entropy term is calculated by the information entropy value of the fusion image, and the modality complementarity term is calculated by the covariance matrix of the inter-modal feature difference; An adaptive inertia weight update rule is designed, the adaptive inertia weight decays exponentially with the iteration number, and the decay factor is negatively correlated with the particle diversity index; A non-dominated sorting strategy is used to screen the Pareto optimal solution set, and an optimal fusion weight matrix is selected from the solution set by an entropy weight method.

[0013] Preferably, the lesion region segmentation based on the multi-modal fusion image in the global optimization layer comprises: A U-Net++ segmentation network is constructed, the U-Net++ segmentation network comprising an encoder-decoder structure and a dense skip connection, the encoder uses ResNet-50 to extract multi-scale features, and the decoder gradually restores the resolution by transposed convolution; A multi-task loss function is designed, including a Dice loss, an edge-aware loss and a lesion shape constraint term, the edge-aware loss is calculated by the Sobel operator to calculate the gradient difference of the segmentation boundary, and the lesion shape constraint term is measured by the compactness of the lesion region; A curriculum learning strategy is used to train the segmentation network, and the initial stage focuses on global feature learning, and the weight of complex samples is gradually increased.

[0014] Preferably, the local correction layer performs image edge enhancement based on the segmentation result, including: An edge-oriented filtering network is constructed, which uses a deformable convolution to adaptively adjust the shape of the filter kernel guided by the segmentation mask; An edge enhancement loss function is designed, including a high-frequency component preservation term and a smoothing term, the high-frequency component preservation term is calculated by the image Laplacian operator response, and the smoothing term is calculated by the gray difference of adjacent pixels; An iterative optimization algorithm is used to alternately update the filter network parameters and the edge enhancement weight until the convergence condition is met.

[0015] Preferably, the execution layer adjusts the image reconstruction parameters through a model predictive control algorithm, including: A state space model of the image reconstruction parameters is established, the state variables include the number of iterations, the regularization coefficient and the filter kernel size, and the control variable is the parameter adjustment amount; A prediction cost function is constructed, including a reconstruction error term, a parameter stability term and a computational efficiency term, and the optimal control sequence is solved by quadratic programming; A rolling horizon strategy is used to dynamically update the parameter adjustment amount, and only the first step control instruction is executed each time.

[0016] Preferably, the drug concentration rate of change is calculated by a differential equation, including: The plasma compartment concentration equation is defined as the convolution integral of the input function and the clearance rate, and the tissue compartment concentration equation is the product of the blood flow rate and the extraction fraction; A fourth-order Runge-Kutta method is used to numerically solve the differential equation set, and the time step is adaptively adjusted according to the curve slope; A Bayesian optimization algorithm is used to estimate the model parameters, a posterior probability distribution of the parameters is constructed, and a Markov chain Monte Carlo sampling is used to generate parameter estimates.

[0017] Preferably, the generator structure of the generative adversarial network includes: The input layer receives the sinogram of the low-dose PET projection data; The encoder part uses 3 layers of convolution layers to compress the feature dimension, and each layer of convolution is followed by instance normalization and LeakyReLU activation; The decoder part recovers the spatial resolution by using three layers of transpose convolutional layers, and the last layer uses a Tanh activation function to constrain the output range; The spectral normalization technique is introduced to stabilize the generator training process, and the spectral norm of the weight matrix of the convolutional layer is limited.

[0018] Preferably, the non-dominated sorting strategy for screening the Pareto optimal solution set comprises: The dominance relationship of each particle is calculated, and the non-dominated individual is marked as the first front layer; The solutions in the same front layer are sorted by using the crowding degree sorting algorithm, and the crowding degree is calculated by the difference in the hyper volume of adjacent solutions in the target space; An elite reservation strategy is designed, and the top 20% of non-dominated solutions in each generation are reserved into the next generation population.

[0019] Compared with the prior art, the beneficial effects of the present application are: At the data acquisition and analysis level, the multi-modal data acquisition module cooperates with the dynamic PET parameterization analysis module to deeply mine the value of PET data. By constructing a double-chamber model to describe the metabolism of radiopharmaceuticals and using the fourth-order Runge-Kutta method to solve the differential equation, the rate of change of drug concentration can be accurately obtained. Combined with the Bayesian optimization algorithm to estimate the model parameters, the accuracy of modeling is ensured. Convolutional neural networks and temporal attention mechanisms work together to effectively extract blood flow features and fuse multiple time point parameters, generating high-quality dynamic parameterization images that provide doctors with more rich metabolic information, which helps to early detect small lesions and improve the sensitivity of disease diagnosis.

[0020] The dose-image quality optimization module plays a key role in ensuring patient safety and image quality. The constructed radiopharmaceutical dose and image quality correlation model comprehensively considers factors such as dose penalty, noise level, and lesion contrast by defining a scientific trade-off function. The generative adversarial network can generate simulated PET images at low dose, and the deep deterministic policy gradient algorithm optimizes the dose allocation strategy, reducing the radiation dose to patients while maintaining or even improving image quality, reducing the potential harm of radiation to the patient's body, and making PET examination safer and more reliable.

[0021] The multi-objective fusion model is based on an improved multi-objective particle swarm optimization algorithm to realize multi-modal image feature-level fusion. The adaptive inertia weight and dynamic learning factor optimize the fusion weight matrix, and the multi-objective optimization function considers feature similarity, information entropy, and modality complementarity to generate multi-modal fusion images that can fully integrate the advantages of different modalities. This not only improves the clarity and information content of the image, but also enhances the doctor's ability to identify complex lesions, which helps to more accurately determine the nature, location, and extent of the lesion, reducing the occurrence of misdiagnosis and missed diagnosis.

[0022] The global optimization layer of the hierarchical processing control module utilizes a U-Net++ segmentation network and a multi-task loss function, combined with a curriculum learning strategy, to accurately segment the lesion area. The local correction layer enhances the image edges and highlights the lesion details without introducing noise by using an edge-oriented filtering network and an iterative optimization algorithm. The execution layer establishes a reasonable state space model and a prediction cost function by means of a model predictive control algorithm, dynamically updates the image reconstruction parameters, improves the reconstruction quality, and takes into account the calculation efficiency, thereby providing strong support for clinical rapid diagnosis.

[0023] Overall, the system realizes efficient fusion and processing of multi-modal data through the cooperative work of various modules, provides doctors with more comprehensive, accurate, and clear image information, improves the accuracy and efficiency of disease diagnosis, and helps the development of precision medicine. It has broad prospects in clinical applications and can effectively improve the medical experience and treatment effect of patients. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The working principle diagram of the image medical multi-modal data fusion processing system described in the present application; Figure 2 The working principle diagram of the dynamic PET parameterization analysis module; Figure 3 The flowchart of the global optimization layer; Figure 4 The working principle diagram of the dynamic adjustment of the image reconstruction parameter execution layer. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0026] Please refer to Figures 1-4 The present application provides an image medical multi-modal data fusion processing system, and the overall implementation scheme is as follows: The system first works through the multi-modal data acquisition module, and uses a dynamic PET scanner to acquire a dynamic PET time-activity curve. This curve contains the metabolic information of the radiopharmaceutical in the body over time, which is important basic data for subsequent analysis.

[0027] After collecting the dynamic PET time-activity curves, the dynamic PET parameterization analysis module models the collected curves based on the hemodynamic model, extracts blood flow features using convolutional neural networks, and fuses multiple time-point hemodynamic parameters with the help of a time-series attention mechanism to generate dynamic parameterization images. These images can more intuitively show the hemodynamic changes in the tissue and provide doctors with more rich diagnostic information.

[0028] The dose-image quality optimization module constructs a correlation model between the dose of radiopharmaceuticals and the quality of images, generates simulated PET images at low doses through a generative adversarial network, and optimizes the dose-image quality trade-off function using reinforcement learning strategies to output the optimal dose allocation scheme. This not only ensures that the image quality meets the diagnostic requirements, but also reduces the dose of radiopharmaceuticals received by the patient and reduces radiation hazards.

[0029] The multi-objective fusion model processes multi-modal image data. Based on the improved multi-objective particle swarm optimization algorithm, the model performs feature-level fusion by introducing adaptive inertia weights and dynamic learning factors to optimize the fusion weight matrix, and then generates multi-modal fusion images. This kind of fusion image integrates the advantages of multiple modal images and can provide more comprehensive and accurate medical information.

[0030] Finally, the hierarchical processing control module further processes the multi-modal fusion image. This module adopts a hierarchical adaptive control strategy, including a global optimization layer, a local correction layer, and an execution layer. The global optimization layer performs lesion region segmentation based on the multi-modal fusion image to determine the approximate location and range of the lesion. The local correction layer enhances the image edge based on the segmentation results to make the lesion edge clearer. The execution layer adjusts the image reconstruction parameters through model predictive control algorithms to optimize the reconstruction effect of the image and improve the image quality.

[0031] The following embodiments 1 to 5 will further illustrate the implementation of the present application.

[0032] Embodiment 1: This embodiment mainly focuses on the parameterization modeling of dynamic PET time-activity curves based on the hemodynamic model in the dynamic PET parameterization analysis module. First, a two-compartment model is constructed to describe the metabolism of radiopharmaceuticals in tissues, which consists of a plasma compartment and a tissue compartment. In the plasma compartment, the concentration of the drug is not only affected by the input function, but also related to the clearance rate. Assuming that the plasma compartment concentration is , the input function is , and the clearance rate is , then the plasma compartment concentration equation is: , where the integral represents the time from the start to The cumulative effect of the product of the input function and the clearance-dependent exponential function. In the tissue compartment, assuming the tissue compartment concentration is , the blood flow rate is , the extraction fraction is , and the tissue compartment concentration equation is , the formula embodies the relationship between the tissue compartment concentration and the plasma compartment concentration, as well as the blood flow rate and the extraction fraction. The fourth-order Runge-Kutta method is used to numerically solve the above differential equation set. In the solving process, the time step will be adaptively adjusted according to the curve slope. If the curve slope changes greatly, it means that the drug concentration changes rapidly, at this time the time step is reduced to improve the solving accuracy; otherwise, the time step is appropriately increased to improve the calculation efficiency. At the same time, the model parameters are estimated by the Bayesian optimization algorithm, the parameter posterior probability distribution is constructed, and the parameter estimate value is generated by Markov chain Monte Carlo sampling. After completing the model construction, the convolutional neural network is used to extract the features of the time-activity curve. The convolutional neural network includes a time convolution layer, a max pooling layer and a fully connected layer, where the size of the convolution kernel of the time convolution layer is , and the step is . The time convolution layer performs convolution operation on the time-activity curve in the time dimension through a convolution kernel of a specific size to extract features on the time series. The max pooling layer filters the features after convolution to retain the main features and reduce the data volume. The fully connected layer integrates the processed features to output a feature vector. Then, the blood flow parameters at different time points are fused through the multi-head attention mechanism. The multi-head attention mechanism calculates the importance weight of each time point parameter, assuming that the weight of the th time point is , the fused dynamic parameterized image is generated by weighted sum of parameters at different time points, i.e. . Finally, the post-processing network of the parameterized image is constructed, the residual structure is used to suppress image noise, and the channel attention mechanism is used to enhance the weight of the blood flow feature channel to further optimize the image quality.

[0033] Example 2: This embodiment focuses on the construction of the correlation model between the dose of radiopharmaceuticals and the image quality in the dose-image quality optimization module. First, define the dose-image quality trade-off function, which includes the dose penalty term, the noise level term and the lesion contrast term. Let the injection dose be , the dose penalty term is calculated by the square sum of the injection dose, i.e. the dose penalty term , where represents the dose of each injection, and the square sum of all injection doses is used to punish high dose. Assuming that the standard deviation of the PET image is , the noise level term is calculated by the standard deviation of the PET image, i.e. the noise level term The greater the standard deviation, the more image noise. Let the SUV value of the lesion region be , the SUV value of the background be , and the lesion contrast term be calculated by the SUV ratio of the lesion region to the background, that is, the lesion contrast term The greater the ratio, the higher the distinction between the lesion and the background. The trade-off function is a combination of the three terms, such as wherein , , is a weight coefficient for adjusting the importance of each term in the trade-off function. Then, a generative adversarial network is constructed, the generator of which takes low-dose PET projection data as input and outputs a high-dose simulated image. The structure of the generator includes: an input layer that receives a sinogram of low-dose PET projection data; an encoder part that uses 3 layers of convolutional layers to compress the feature dimension, and after each convolution, an instance normalization and a LeakyReLU activation; a decoder part that uses 3 layers of transposed convolutional layers to restore the spatial resolution, and the last layer uses a Tanh activation function to constrain the output range. The spectral normalization technique is introduced to stabilize the generator training process, and the spectral norm of the weight matrix of the convolutional layer is limited. The discriminator adopts a multi-scale convolutional structure to distinguish the authenticity of the image. The deep deterministic policy gradient algorithm is used to optimize the dose allocation strategy, the state space is constructed as the current dose distribution and the image quality index, the action space is the dose adjustment amount, and the reward function is based on the weighted combination of the trade-off function. Through continuous iteration and optimization, the optimal dose allocation scheme is found.

[0034] Embodiment 3: This embodiment details the process of feature-level fusion of multi-modal image data based on an improved multi-objective particle swarm algorithm in a multi-objective fusion model. First, a multi-objective optimization function is defined, including a feature similarity term, an information entropy term, and a modality complementarity term. Let the feature maps of modality and modality be and respectively, and the feature similarity term be calculated by the structural similarity of the feature maps between modalities, the structural similarity index is calculated as wherein , are the mean values of the feature maps , respectively, , are the standard deviations of the feature maps , respectively, is the covariance of and , , is a constant used to maintain the stability of the calculation. The information entropy term is calculated by fusing the information entropy value of the image. The greater the information entropy, the more information the image contains. Let the feature difference of modalities and modalities be , and the modality complementarity term is calculated by the covariance matrix of the feature difference between modalities. Assuming that the covariance matrix is , the modality complementarity term is related to . An adaptive inertia weight update rule is designed, and the inertia weight decays exponentially with the number of iterations. The decay factor is negatively related to the particle diversity index. The higher the particle diversity index, , the smaller the inertia weight decay, which can maintain the global search ability of particles in the early stage of the algorithm and gradually enhance the local search ability in the later stage. A non-dominated sorting strategy is used to select the Pareto optimal solution set. The dominance relationship of each particle is calculated, and the non-dominated individuals are marked as the first front layer. A crowding degree sorting algorithm is used to sort the solutions in the same front layer. The crowding degree is calculated by the difference in hyper volume between adjacent solutions in the objective space. An elite reservation strategy is designed, and the top 20% of non-dominated solutions in each generation are reserved for the next generation population. Finally, the optimal fusion weight matrix is selected from the solution set by the entropy weight method.

[0035] Example 4 This example is used to describe the implementation of the global optimization layer in the hierarchical processing control module based on multi-modal fusion images for lesion region segmentation. Specifically, it includes: A U-Net++ segmentation network is constructed, which includes an encoder-decoder structure and dense skip connections. The encoder uses ResNet-50 to extract multi-scale features. ResNet-50 extracts features from the input multi-modal fusion image through a series of convolutional layers and residual blocks, which can obtain feature information at different scales of the image. The decoder gradually restores the resolution by transposed convolution, upsamples the features extracted by the encoder, and finally outputs the segmentation result. A multi-task loss function is designed, including Dice loss, edge perception loss, and lesion shape constraint term. Let the segmentation result be , the true label be , the Dice loss be , and the value be closer to 1, indicating better segmentation effect. The Sobel operator is used to calculate the gradient difference of the segmentation boundary, and the edge perception loss is designed accordingly. Assuming that the boundary gradient of the segmentation result calculated by the Sobel operator is , the boundary gradient of the true label is , and the edge perception loss is , this loss function can make the segmentation boundary closer to the true boundary. The shape constraint term is measured by the compactness of the lesion region. Assuming that the area of the lesion region is , the perimeter is , shape constraint term The closer the value is to 1, the closer the lesion region is to a circle, i.e. the shape is more regular. The segmentation network is trained using a curriculum learning strategy. In the initial stage, global feature learning is focused on, and simple samples are given greater weight. As the training progresses, the weight of complex samples is gradually increased, so that the network can learn more complex features and improve the segmentation accuracy.

[0036] Embodiment 5: This embodiment mainly introduces the process of image edge enhancement based on the segmentation result in the local correction layer of the hierarchical processing control module and the adjustment of the image reconstruction parameters by the model predictive control algorithm in the execution layer. Specifically, it includes: The local correction layer constructs an edge-oriented filtering network. The network is guided by the segmentation mask and uses deformable convolution to adaptively adjust the shape of the filter kernel. Deformable convolution can dynamically adjust the sampling position of the convolution kernel according to the local features of the image, better capturing edge information. An edge enhancement loss function is designed, including a high-frequency component preservation term and a smoothing term. Let the image be The high-frequency component preservation term is calculated by the image Laplacian operator response. The Laplacian operator is sensitive to the high-frequency components of the image. The high-frequency component preservation term The larger the value, the better the preservation of the high-frequency components of the image. The smoothing term is calculated by the gray level difference of adjacent pixels. Assuming that the gray level values of adjacent pixels are and , the smoothing term is The smaller the value, the smoother the image. An iterative optimization algorithm is used to alternately update the filter network parameters and the edge enhancement weight until the convergence condition is met, so that the image edge is enhanced. The execution layer establishes a state space model of the image reconstruction parameters. The state variables include the iteration number, the regularization coefficient and the filter kernel size, and the control variable is the parameter adjustment amount. A prediction cost function is constructed, including a reconstruction error term, a parameter stability term and a computational efficiency term. Let the difference between the reconstructed image and the original image be The reconstruction error term is ; let the change amount before and after the parameter adjustment be , and the parameter stability term is The computational efficiency term is related to the calculation time, and the optimal control sequence is solved by quadratic programming. A rolling horizon strategy is used to dynamically update the parameter adjustment amount. Only the first step control instruction is executed in each optimization, and the image reconstruction parameters are constantly adjusted to optimize the image reconstruction effect.

[0037] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and illustrative figures, it should be apparent that the scope of the present application is not limited to these specific embodiments.

[0038] While the embodiments of the application have been shown and described herein, it will be understood by those skilled in the art that many changes, modifications, substitutions and alterations to these embodiments can be made without departing from the principles and spirits of the application, and it is intended that the scope of the application be limited solely by the scope of the appended claims and the equivalents thereof.

Claims

1. An image medical multi-modality data fusion processing system, characterized by, include: The multimodal data acquisition module is used to acquire dynamic PET time-activity curves using a dynamic PET scanner; The dynamic PET parametric analysis module parametrically models the dynamic PET time-activity curve based on a hemodynamic model, extracts blood flow features using a convolutional neural network, and fuses hemodynamic parameters from multiple time points through a temporal attention mechanism to generate a dynamic parametric image. The dose-image quality optimization module constructs a correlation model between radiopharmaceutical dose and image quality, uses generative adversarial networks to generate simulated PET images under low doses, optimizes the dose-image quality trade-off function through reinforcement learning strategies, and outputs the optimal dose allocation scheme. A multi-objective fusion model is based on an improved multi-objective particle swarm optimization algorithm to perform feature-level fusion of multimodal image data. The improved multi-objective particle swarm optimization algorithm introduces adaptive inertial weights and dynamic learning factors to optimize the fusion weight matrix and generate a multimodal fused image. The hierarchical processing control module adopts a hierarchical adaptive control strategy, including a global optimization layer, a local correction layer, and an execution layer. The global optimization layer performs lesion region segmentation based on multimodal fusion images, the local correction layer enhances image edges based on the segmentation results, and the execution layer adjusts image reconstruction parameters through a model prediction control algorithm.

2. The image medical multi-modality data fusion processing system of claim 1, wherein, The parametric modeling of the dynamic PET time-activity curve based on the hemodynamic model includes: A two-compartment model was constructed to describe the metabolic process of radiopharmaceuticals in tissues. The two-compartment model includes a plasma compartment and a tissue compartment. The rate of change of drug concentration was calculated using differential equations. A convolutional neural network is used to extract features from the time-activity curve. The convolutional neural network includes a temporal convolutional layer, a max pooling layer, and a fully connected layer. The kernel size of the temporal convolutional layer is 3×1, and the stride is 1. By fusing blood flow parameters at different time points through a multi-head attention mechanism, the importance weight of each parameter at each time point is calculated, and a fused dynamic parameterized image is generated. A post-processing network for parameterized images is constructed, and residual structures are used to suppress image noise. The weights of blood flow feature channels are enhanced through a channel attention mechanism.

3. The image medical multi-modality data fusion processing system of claim 1, wherein, The construction of the correlation model between radiopharmaceutical dosage and image quality includes: Define a dose-image quality tradeoff function, which includes a dose penalty term, a noise level term, and a lesion contrast term. The dose penalty term is calculated by the sum of squares of the injection dose, the noise level term is calculated by the standard deviation of the PET image, and the lesion contrast term is calculated by the SUV ratio of the lesion area to the background. A generative adversarial network is constructed. The generator takes low-dose PET projection data as input and outputs high-dose simulated images. The discriminator uses a multi-scale convolutional structure to determine the authenticity of the images. A deep deterministic gradient algorithm is used to optimize the dose allocation strategy. The state space is constructed as the current dose distribution and image quality index, the action space is the dose adjustment amount, and the reward function is based on a weighted combination of trade-off functions.

4. The image medical multi-modality data fusion processing system of claim 1, wherein, Feature-level fusion of multimodal image data based on an improved multi-objective particle swarm optimization algorithm includes: A multi-objective optimization function is defined, including a feature similarity term, an information entropy term and a modality complementarity term, the feature similarity term is calculated by the structural similarity of inter-modal feature maps, the information entropy term is calculated by the information entropy value of the fused image, and the modality complementarity term is calculated by the covariance matrix of inter-modal feature difference; An adaptive inertia weight update rule is designed, the adaptive inertia weight exponentially decays with the iteration number, and the decay factor is negatively correlated with the particle diversity index; A non-dominated sorting strategy is used to screen the Pareto optimal solution set, and the optimal fusion weight matrix is selected from the solution set by the entropy weight method.

5. The image medical multi-modality data fusion processing system of claim 1, wherein, The global optimization layer performs lesion region segmentation based on the multi-modal fusion image, including: A U-Net++ segmentation network is constructed, which contains an encoder-decoder structure and dense skip connection, the encoder uses ResNet-50 to extract multi-scale features, and the decoder gradually restores the resolution through transposed convolution; A multi-task loss function is designed, including Dice loss, edge perception loss and lesion shape constraint term, the edge perception loss is calculated by the Sobel operator to calculate the gradient difference of the segmentation boundary, and the lesion shape constraint term is calculated by the compactness of the lesion region; A curriculum learning strategy is used to train the segmentation network, which focuses on global feature learning in the initial stage and gradually increases the weight of complex samples.

6. The image medical multi-modality data fusion processing system of claim 1, wherein, The local correction layer performs image edge enhancement based on the segmentation result, including: An edge-oriented filtering network is constructed, which uses a deformable convolution to adaptively adjust the filter kernel shape guided by the segmentation mask; An edge enhancement loss function is designed, including a high frequency component preservation term and a smoothing term, the high frequency component preservation term is calculated by the image Laplacian operator response, and the smoothing term is calculated by the gray difference of adjacent pixels; An iterative optimization algorithm is used to alternately update the filtering network parameters and the edge enhancement weight until the convergence condition is met.

7. The image medical multi-modality data fusion processing system of claim 1, wherein, The execution layer adjusts the image reconstruction parameters through the model predictive control algorithm, including: A state space model of the image reconstruction parameters is established, the state variables include the iteration number, the regularization coefficient and the filter kernel size, and the control variable is the parameter adjustment amount; A prediction cost function is constructed, including a reconstruction error term, a parameter stability term and a computational efficiency term, and the optimal control sequence is solved by quadratic programming; A rolling horizon strategy is used to dynamically update the parameter adjustment amount, and only the first step control instruction is executed at each optimization.

8. The image medical multi-modality data fusion processing system of claim 2, wherein, The drug concentration change rate is calculated by a differential equation, including: The plasma compartment concentration equation is defined as the convolution integral of the input function and the clearance rate, and the tissue compartment concentration equation is the product of the blood flow rate and the extraction fraction; A fourth-order Runge-Kutta method is used to numerically solve the differential equation set, and the time step is adaptively adjusted according to the curve slope; A Bayesian optimization algorithm is used to estimate the model parameters, a posterior probability distribution of the parameters is constructed, and parameter estimation values are generated by Markov chain Monte Carlo sampling.

9. The image medical multi-modality data fusion processing system of claim 3, wherein, The generator structure of the generative adversarial network includes: The input layer receives the sinogram of the low-dose PET projection data; The encoder part uses 3 layers of convolution layers to compress the feature dimension, and each layer of convolution is followed by instance normalization and LeakyReLU activation; The decoder part adopts 3 layers of transpose convolution layers to recover the spatial resolution, and the last layer uses Tanh activation function to constrain the output range; Spectral normalization technique is introduced to stabilize the generator training process and limit the spectral norm of the weight matrix of the convolution layer.

10. The image medical multi-modality data fusion processing system of claim 4, wherein, The non-dominated sorting strategy for screening the Pareto optimal solution set includes: Calculate the dominance relationship of each particle, and mark the non-dominated individuals as the first front layer; Use the crowding distance sorting algorithm to sort the solutions in the same front layer, and the crowding distance is calculated by the difference in hyper volume of adjacent solutions in the objective space; Design an elite reservation strategy, and reserve the top 20% non-dominated solutions of each generation into the next generation population.

Images