Medical image recognition processing system and method based on multi-modal image fusion

Abstract

The invention discloses a medical image recognition processing system and method based on multi-modal image fusion, and belongs to the technical field of image processing, and the method comprises the steps: obtaining a medical image, and correcting the medical image to obtain an image data set; inputting the image data set into a parallel neural network, extracting to obtain tracheal tissue structure features and biological metabolism features, and fusing to generate a comprehensive image feature map; and comparing the fusion features of different organs in the comprehensive image feature map with features in a preset typical case feature library to generate risk assessment values of corresponding organs, and associating the risk assessment values with corresponding organs in a preset AR visual image. According to the method, the trachea tissue structure features and the biological metabolism features are extracted through the parallel neural network and fused to generate the comprehensive image feature map, then the comprehensive image feature map is compared with the typical case feature library to generate the risk assessment value, finally the risk assessment value is associated with the AR visual image, and effective fusion and utilization of multi-modal image data are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a medical image recognition and processing system and method based on multimodal image fusion. Background Art

[0002] With the rapid development of medical imaging technology, doctors can obtain more and more types and modalities of medical imaging data, such as CT (computed tomography), MRI (magnetic resonance imaging), PET (positron emission tomography), etc. These different modalities of imaging data provide rich information about the internal structure and function of the human body, which is crucial for the early diagnosis of diseases, the formulation of treatment plans, and the evaluation of treatment effects.

[0003] With the development of computer technology and artificial intelligence, multimodal image fusion technology has gradually become a hot research direction in the field of medical image analysis. Existing multimodal image fusion methods often require complex preprocessing steps, which not only increase the complexity of the calculation, but also may introduce additional errors. In addition, existing fusion methods often focus on simple feature splicing or weighted averaging, ignoring the deep correlation and complementarity between different modal features, resulting in limited accuracy of the fusion results. At the same time, how to effectively perform risk assessment using the fused features is also an urgent problem to be solved. Summary of the invention

[0004] To solve the above problems, the present invention provides a medical image recognition and processing system and method based on multimodal image fusion, which extracts tracheal tissue structure characteristics and biological metabolic characteristics through parallel neural networks, and fuses them to generate a comprehensive image feature map, which is then compared with a typical case feature library to generate a risk assessment value, and finally associated with an AR visualization image, thereby realizing the effective fusion and utilization of multimodal image data.

[0005] The above objectives can be achieved through the following solutions: A medical image recognition and processing method based on multimodal image fusion, comprising: acquiring at least two types of medical images from different devices and correcting them to obtain an image data set; inputting the image data set into a preset parallel neural network to extract tracheal tissue structure characteristics and biological metabolism characteristics; fusing the tracheal tissue structure characteristics and the biological metabolism characteristics to obtain fusion characteristics, and using the fusion characteristics of each organ to generate a comprehensive image feature map; comparing the fusion characteristics of different organs in the comprehensive image feature map with the characteristics in a preset typical case feature library to generate a risk assessment value for the corresponding organ; and associating the risk assessment value with the corresponding organ in a preset AR visualization image to obtain the corresponding risk assessment value through the corresponding organ in the AR visualization image.

[0006] Optionally, acquiring at least two types of medical images from different devices and correcting them to obtain an image data set includes: time-series aligning medical images from different devices to obtain an initialized image data set; mapping the image coordinate system in the initialized image data set to a unified human anatomical coordinate system, and outputting spatially aligned multimodal image data; performing denoising on the multimodal image data using a preset nonlinear filtering algorithm, and outputting denoised image data; and normalizing and calibrating the grayscale values ​​of the denoised image data based on device parameter differences to generate an image data set with a unified quantization range.

[0007] Optionally, the image data set is input into a preset parallel neural network to extract tracheal tissue structure characteristics and biological metabolic characteristics, including: taking any image data in the image data set as a reference, performing voxel interpolation processing on each image data in the image data set; for the image data after voxel interpolation processing, performing bidirectional linear interpolation compensation on the mechanical distortion area generated in the mapping process to obtain the final image data set.

[0008] Optionally, the step of inputting the image data set into a preset parallel neural network to extract tracheal tissue structure features and biological metabolism features includes: inputting the final image data set into a 3D U-Net network with dilated convolution to output tracheal tissue structure features; inputting the final image data set into a multi-scale feature pyramid network to output biological metabolism features.

[0009] Optionally, the fusion of the tracheal tissue structural feature and the biological metabolism feature to obtain the fusion feature includes: obtaining the structural segmentation accuracy score of the current region, calculating the structural weight value of the tracheal tissue structural feature, and for the structural weight value ,have , In the formula, is an exponential function, Score the accuracy of the structure segmentation in the current region, is the adjustment coefficient; obtain the activity standard deviation of the biological metabolic characteristics of the current area and the activity standard deviation of the biological metabolic characteristics in the entire map, calculate the metabolic weight value of the biological metabolic characteristics, and for the metabolic weight value ,have , In the formula, is the activity standard deviation of the biological metabolic characteristics of the current area, The standard deviation of the activity of the biological metabolic features in the whole map, is a maximum value function; using the structure weight value and the metabolic weight value, a fusion feature is calculated, and for the fusion feature ,have , In the formula, The tracheal tissue structure characteristics, is the biological metabolic characteristic, is the outer product of the tensor.

[0010] Optionally, the use of the fusion features of each organ to generate a comprehensive image feature map includes: spatially registering the fusion features with standard organ boundaries in a preset human anatomical atlas database; obtaining the metabolic activity values ​​of each organ region after registration and sorting them from large to small, selecting the metabolic activity value ranked first, and extracting a metabolic activity quantitative index; obtaining the average structural density of each organ region after registration, and extracting a structural density quantitative index; using the metabolic activity quantitative index and the structural density quantitative index, generating a feature index table with three-dimensional spatial coding to obtain a comprehensive image feature map.

[0011] Optionally, the comparing the fusion features of different organs in the comprehensive image feature map with the features in a preset typical case feature library includes: searching for a pathological template matching the corresponding organ in the typical case feature library according to the organ code in the feature index table of the comprehensive image feature map; obtaining the metabolic activity quantitative index and the structural density quantitative index of the corresponding organ, calculating a similarity index with the pathological template, and calculating the similarity index. ,have , In the formula, For the A quantitative indicator of the structural density of each organ. For the A quantitative indicator of the metabolic activity of an organ.

[0012] Optionally, generating a risk assessment value for the corresponding organ includes: determining whether the similarity index is greater than a preset threshold; if the similarity index is greater than the preset threshold, outputting a risk assessment value of 0; if the similarity index is less than the preset threshold, marking the pathology template as a similar case; and outputting the clinical manifestation probability of the similar case through a preset Bayesian network to obtain a risk assessment value for the corresponding organ.

[0013] Optionally, the method also includes: obtaining the true pathological results of the corresponding organ, and performing an evaluation based on the true pathological results to obtain a true evaluation value; calculating the difference between the true evaluation value of the corresponding organ and the risk evaluation value; if the difference is greater than a preset warning value, reversely correcting the weight parameters of the corresponding pathological template in the typical case feature library.

[0014] Based on the same inventive concept, the present invention also provides a medical image recognition and processing system based on multimodal image fusion, the system comprising: an image data acquisition module, used to acquire at least two types of medical images from different devices and perform corrections to obtain an image data set; a feature extraction module, used to input the image data set into a preset parallel neural network to extract tracheal tissue structure characteristics and biological metabolic characteristics; a feature map generation module, used to fuse the tracheal tissue structure characteristics and the biological metabolic characteristics to obtain fusion characteristics, and use the fusion characteristics of each organ to generate a comprehensive image feature map; a risk assessment module, used to compare the fusion characteristics of different organs in the comprehensive image feature map with the characteristics in a preset typical case feature library to generate a risk assessment value for the corresponding organ; a visualization module, used to associate the risk assessment value with the corresponding organ in a preset AR visualization image.

[0015] Compared with the prior art, the present invention has the following advantages: 1. The present invention extracts tracheal tissue structural features and biological metabolic features through parallel neural networks and performs deep fusion, fully considering the complementarity between different modal images, thereby improving the accuracy of image analysis; traditional multimodal image fusion methods often focus on simple feature splicing or weighted averaging, ignoring the deep-level association between features, while the present invention achieves the optimal coordination of multimodal image features through an adaptive weighting mechanism; 2. The present invention compares the comprehensive image feature map with the typical case feature library to generate the risk assessment value of the corresponding organ; this method not only takes into account the image features, but also combines clinical pathological data, which can more comprehensively assess the patient's condition and potential risks; in addition, through the reverse correction mechanism, the system can continuously learn and optimize to improve the accuracy of risk assessment; 3. The present invention realizes the automated processing and analysis of multimodal imaging data, greatly shortening the diagnosis time; through parallel processing and deep learning technology, the system can quickly and accurately extract imaging features, generate risk assessment values, and provide timely feedback to doctors, thereby improving the efficiency of the entire diagnostic process; Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of a medical image recognition and processing method based on multimodal image fusion according to an embodiment of the present invention.

[0018] Figure 2 It is a schematic diagram of the process of medical image correction according to an embodiment of the present invention.

[0019] Figure 3 Schematic diagram of the structure of a parallel neural network according to an embodiment of the present invention.

[0020] Figure 4 It is a structural schematic diagram of a medical image recognition and processing system based on multimodal image fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Reference Figure 1 An embodiment of the present invention proposes a medical image recognition and processing method based on multimodal image fusion, which extracts tracheal tissue structure characteristics and biological metabolic characteristics through parallel neural networks, and fuses them to generate a comprehensive image feature map, which is then compared with a typical case feature library to generate a risk assessment value, and finally associated with an AR visualization image, thereby achieving effective fusion and utilization of multimodal image data.

[0023] The method of this embodiment specifically includes: Acquire at least two types of medical images from different devices and perform corrections to obtain an image dataset; Specifically, at least two types of medical images are obtained from different devices, which may include but are not limited to CT (computed tomography), MRI (magnetic resonance imaging), PET (positron emission tomography), etc. These image data reflect the structure and function information of different tissues and organs of the human body and are the basis for subsequent processing and analysis.

[0024] Inputting the image data set into a preset parallel neural network to extract tracheal tissue structure characteristics and biological metabolism characteristics; Specifically, after acquiring the image data, the data will be corrected to obtain an image data set. The correction process includes steps such as timing alignment, spatial alignment, noise reduction processing and grayscale value normalization calibration. Timing alignment is to solve the problem of asynchronous scanning time of different devices. By extracting the scanning timestamp in the DICOM file header, a multimodal image timing sequence is established, and the timing misalignment caused by respiratory movement is compensated by the time interpolation method based on the sliding window. Spatial alignment maps the coordinate systems of different images to a unified human anatomical coordinate system to ensure the spatial consistency of the image data. Noise reduction processing uses a preset nonlinear filtering algorithm to reduce the noise of the image data to improve the image quality. Grayscale value normalization calibration calibrates the grayscale value of the image data based on the difference in device parameters to ensure that the image data scanned by different devices are comparable in grayscale value.

[0025] Specifically, the corrected image data set is then input into a preset parallel neural network to extract the tracheal tissue structure features and biological metabolic features. This parallel neural network can include a 3D U-Net network with dilated convolution and a multi-scale feature pyramid network. The 3D U-Net network with dilated convolution is used to extract tracheal tissue structure features, which can accurately capture structural information such as the four-level bifurcation morphology of the trachea. The multi-scale feature pyramid network is used to extract biological metabolic features, which can effectively analyze metabolic activity information in image data.

[0026] The tracheal tissue structure feature and the biological metabolism feature are fused to obtain a fusion feature, and a comprehensive image feature map is generated using the fusion feature of each organ; Comparing the fusion features of different organs in the comprehensive image feature atlas with the features in a preset typical case feature library to generate a risk assessment value for the corresponding organ; Specifically, the extracted tracheal tissue structural features and biological metabolic features are then fused to obtain fused features. The fusion process takes into account the weights of different features, and the tracheal tissue structural features and biological metabolic features are weightedly fused by calculating the structural weight value and the metabolic weight value. The structural weight value is calculated based on the segmentation accuracy score of the tracheal tissue structure, and the metabolic weight value is calculated based on the activity standard deviation of the biological metabolic feature. The fused features of each organ are used to generate a comprehensive image feature map, and the fused features of different organs in the comprehensive image feature map are compared with the features in the preset typical case feature library to generate risk assessment values ​​for the corresponding organs. These risk assessment values ​​can help doctors better understand the patient's condition and potential risks.

[0027] The risk assessment value is associated with a corresponding organ in a preset AR visualization image, so as to obtain a corresponding risk assessment value through the corresponding organ in the AR visualization image.

[0028] AR ‌ refers to the technology of superimposing virtual images on the real world through augmented reality technology (AR for short). AR technology captures the position and angle of organ images through a camera, performs stereoscopic image processing, and puts the virtual image of the organ into the real world for interaction. This technology combines the virtual world with the real world, which can improve the visual stereoscopic experience and interactivity of the organ. For example, suppose the system obtains a CT image and a PET image of a patient. First, the system will align the two images in time and space to ensure their consistency in time and space. Then, the corrected image data set is input into the parallel neural network to extract the tissue structure characteristics of the trachea and the biological metabolic characteristics in the PET image. Then, these features are fused to obtain fused features. Finally, the fused features are used to generate a comprehensive image feature map, which is compared with the features in the typical case feature library to generate the risk assessment value of the corresponding organ. Assuming that the patient has abnormal metabolic activity in the lungs, the system may give a higher risk assessment value, prompting the doctor to further examine and treat.

[0029] Alternatively, if Figure 2 As shown, the obtaining of at least two types of medical images from different devices and performing correction to obtain an image data set includes: Align the medical images from different devices in time sequence to obtain an initialized image dataset; Specifically, a multimodal image timing sequence is established by extracting the scanning timestamp in the DICOM file header. When the time interval between CT images and PET images exceeds the device sampling period, a sliding window-based time interpolation method is used to compensate for the timing misalignment caused by respiratory movement.

[0030] For example, to address the synchronization problem of chest CT and dynamic PET, the synchronization rate for chest CT is 0.5 seconds per layer and for dynamic PET is 2 seconds per frame. The R wave of the electrocardiogram can be used as a reference point to perform gated reconstruction on the PET projection data within the same cardiac cycle, ensuring that the CT thin-layer scan and the PET metabolic image are in the same respiratory phase.

[0031] Mapping the image coordinate system in the initialized image data set to a unified human anatomical coordinate system, and outputting spatially aligned multimodal image data; Specifically, the initial coordinate system was established with the three points of anterior superior iliac spine, sternal notch, and external occipital protuberance, and the 3D affine transformation matrix was used to map each modality image to the standard MNI152 space template, and the feature point registration error was controlled within 1.2 mm.

[0032] Exemplarily, the PET / CT fusion system uses an iterative closest point (ICP) algorithm based on feature points. First, the SIFT feature descriptors of each modality image are extracted; then the stable matching point pairs are selected through the RANSAC algorithm; finally, the optimal rigid body transformation parameters are calculated to make the Hausdorff distance of the liver contour after registration ≤3mm.

[0033] Performing noise reduction processing on the multimodal image data using a preset nonlinear filtering algorithm, and outputting noise-reduced image data; Specifically, an improved filter based on anisotropic diffusion is constructed, and the diffusion coefficient is given by the formula Determine, where is an exponential function, is the gradient threshold, It is the gray value of a pixel in the image, where the gradient threshold is dynamically adjusted based on the image signal-to-noise ratio to suppress random noise while retaining the edge features of the organ.

[0034] The grayscale values ​​of the noise-reduced image data are normalized and calibrated based on the differences in device parameters to generate an image data set with a unified quantization range.

[0035] Specifically, a mapping relationship between Hounsfield units and standardized uptake values ​​(SUV) was established, and calibration curves under different scanning protocols were generated by bicubic spline interpolation to ensure that the bone tissue CT value was stable at 100-300HU and the tumor SUV value was scaled to the range of [0,5].

[0036] This implementation method achieves precise multimodal image registration through collaborative optimization in four dimensions: the time dimension solves the phase difference caused by physiological movement, the spatial dimension ensures the consistency of anatomical structure, the signal dimension eliminates device-related noise, and the intensity dimension unifies the quantification standard.

[0037] Alternatively, if Figure 3As shown, the image data set is input into a preset parallel neural network to extract tracheal tissue structure characteristics and biological metabolism characteristics, including: Taking any image data in the image data set as a reference, performing voxel interpolation processing on each image data in the image data set; Specifically, CT thin-layer scanning is used as the reference coordinate system with a CT thin-layer thickness of 0.625 mm. Asymmetric interpolation is performed for MRI T2WI and PET with a MRI T2WI layer thickness of 3 mm and a PET layer thickness of 4.25 mm. The PixelSpacing and ImagePositionPatient parameters in the DICOM metadata are parsed, and the Z-axis voxel resampling position is calculated based on the Euclidean space. When the interval between consecutive sagittal plane sections exceeds 5 mm, the transition layer is automatically inserted.

[0038] For the image data after voxel interpolation processing, the mechanical distortion area generated in the mapping process is compensated by bidirectional linear interpolation to obtain the final image data set.

[0039] Specifically, the geometric distortion area caused by the nonlinearity of the magnetic resonance gradient is identified, and the field strength mapping file provided by the equipment manufacturer is pre-loaded, such as the B0 field topology map of 3T MRI; the deviation matrix between the theoretical gradient field and the actual measured value is calculated; and piecewise linear compensation is applied to the phase encoding direction, where the correction coefficient of the central FOV (within 30cm in diameter) is 0.95-1.05, and the edge FOV (outside 50cm in diameter) is fitted using a high-order polynomial.

[0040] For example, in the AX T1 sequence of GE Signa 7T MRI, a 3.8mm butterfly-shaped distortion was detected in the anterior cranial fossa region; the convolution kernel was applied in the frequency domain space. Anisotropic smoothing was performed; the anatomical contours were restored by bicubic interpolation, which reduced the Hausdorff distance of the brainstem edge from 4.2 mm to 0.7 mm.

[0041] Specifically, the present invention can also perform multimodal interpolation compensation, such as dynamic compensation for bidirectional deformation caused by respiratory movement. The deformation vector field is constructed using the lung field surface reconstructed by 4D-CT, and the deformation field driven interpolation is performed on the PET metabolic activity map. The diaphragm vertex displacement error is confirmed to be ≤2mm through feature matching. For example, in the 4D-PET / CT fusion of lung cancer, the end-expiratory CT is used as the reference coordinate system; the back-projection algorithm calculates the phase signal distribution of each projection angle of PET; and the PET data of 10 respiratory phases are mapped to the CT reference system through non-rigid registration, eliminating 75% of respiratory artifacts.

[0042] This implementation method can ensure data consistency through a three-level interpolation system. The first-level geometric interpolation solves the layer thickness difference, the second-level nonlinear correction eliminates the inherent distortion of the equipment, and the third-level spatiotemporal compensation copes with physiological movement interference, which can significantly improve the accuracy of subsequent pathological analysis.

[0043] Alternatively, if Figure 3 As shown, the image data set is input into a preset parallel neural network to extract tracheal tissue structure characteristics and biological metabolism characteristics, including: The final image dataset is input into a 3D U-Net network with dilated convolution to output the tracheal tissue structure features; Specifically, in the 3rd to 5th layers of the encoder, the void rate configuration of d=2 / 4 / 6 is used respectively to achieve multi-scale receptive field coverage of 3.5cm³ to 8.7cm³, which can accurately capture the four-level bifurcation morphology of the trachea; the channel attention mechanism is introduced in the skip connection, and the feature map weights are dynamically adjusted according to the preset formula; the ternary composite loss of tracheal main bronchus segmentation plus terminal bronchioles segmentation plus boundary constraint term is adopted.

[0044] The final image dataset is input into a multi-scale feature pyramid network to output biological metabolic features.

[0045] Specifically, the basic skeleton network adopts an improved 3D ResNet-50 architecture, adjusts the stride of stage4 to 1, and retains high-resolution feature maps; the pyramid construction generates feature maps with four atrous rates of 1 / 4 / 6 / 24 through atrous spatial pyramid pooling (ASPP), fuses multi-scale features by element-by-element addition rather than splicing, and embeds a feature calibration module after the pooling layer; a dual-channel output head is set at the end of the network to predict the SUVmax distribution map and metabolic heterogeneity index (HI) respectively, and obtain the biological metabolic characteristics.

[0046] Specifically, this embodiment realizes decoupled feature extraction through a dual deep network, 3D U-Net focuses on millimeter-level anatomical structure characterization, and feature pyramid focuses on metabolic heterogeneity analysis. The two networks share the preprocessing layer but are trained independently.

[0047] Optionally, the fusion of the tracheal tissue structure feature and the biological metabolism feature to obtain a fusion feature includes: Obtain the structural segmentation accuracy score of the current region, calculate the structural weight value of the tracheal tissue structural feature, and ,have , In the formula, is an exponential function, Score the accuracy of the structure segmentation in the current region, is the adjustment factor; Specifically, the tracheal structure recognition accuracy of the current region is mapped by the S-shaped function, and the value of the adjustment coefficient in the formula is derived from the statistical results of clinical trials. The weight change slope is steepest in the interval [0.6, 0.9], which can sensitively capture the topological errors of bronchi below level 4; for extremely low segmentation accuracy, such as In the area <0.4, soft threshold truncation is used to avoid noise amplification; 3×3×3 voxel neighborhood smoothing filtering is applied during weight calculation to prevent local mutations in weight values.

[0048] For example, assuming that the segmentation result of a certain bronchus on CT is excellent for the main bronchus segmentation, =0.93, then the calculated weight is 0.977, close to the full score of 1; small branches are generally segmented due to the limited CT resolution. =0.57, then the weight is 0.801, which is still usable but has reduced influence.

[0049] Obtain the activity standard deviation of the biological metabolic characteristics of the current area and the activity standard deviation of the biological metabolic characteristics in the entire map, calculate the metabolic weight value of the biological metabolic characteristics, and ,have , In the formula, is the activity standard deviation of the biological metabolic characteristics of the current area, The standard deviation of the activity of the biological metabolic features in the whole map, is the maximum value function; Specifically, the weight is automatically adjusted according to the abnormal degree of metabolic activity in the PET image, and the more abnormal the metabolism, the higher the attention will be; the fluctuation amplitude of the metabolic value of the local area is calculated, and the larger the value, the more uneven the metabolism; the fluctuation of the background metabolism of the whole body is used as the benchmark, and a high weight is given when the local value is significantly higher than the whole body; when the whole body metabolism is extremely uniform, such as in a low metabolic state, the denominator is retained at least 0.1 to prevent the calculation result from being too large.

[0050] Specifically, the 3σ principle was used to define the ROI metabolic abnormality area, and only voxels that exceeded the whole image mean ±3σ were involved in the activity standard deviation calculation; the denominator took the maximum value function to ensure the stability of weight calculation in low metabolic scenarios, such as gray matter background, to avoid the risk of division by zero due to uniform metabolism of the whole image; stratified sampling was used to verify the Pearson correlation between the metabolic standard deviation and the degree of malignancy.

[0051] For example, the metabolism of the tumor area of ​​lung cancer lesions fluctuates dramatically, with a local standard deviation of 2.1, and the whole body metabolism is relatively stable with an overall standard deviation of 0.65. The calculated weight is 3.23, which highlights the lesion; the metabolism of calcified nodules is almost unchanged, with a standard deviation of 0.08 and an overall standard deviation of 0.52. The calculated weight is 0.15, which is almost negligible.

[0052] The structural weight value and the metabolic weight value are used to calculate the fusion feature. ,have , In the formula, The tracheal tissue structure characteristics, is the biological metabolic characteristic, is the outer product of the tensor.

[0053] Specifically, the two features of anatomical structure and metabolic activity are deeply combined to discover hidden associations; the structural features are regarded as a number bar with a length of 32, and the metabolic features are regarded as a number bar with a width of 16; the two numbers are arranged into a table of 32 rows and 16 columns at each position, which is equivalent to a feature relationship table; each grid of the table reflects the combination relationship of the two features, such as the combination of sharp structural edges and metabolic surges; a smart filter (1×1×1 convolution) is used to compress the 32×16 table into 256 key features.

[0054] Specifically, this implementation method realizes an adaptive weighting mechanism based on anatomical segmentation credibility, statistically driven modeling of metabolic heterogeneity, high-order tensor operations to capture cross-modal coupling effects, and deep adaptation of clinical parameters and mathematical models, ultimately achieving optimal synergy of multimodal imaging features.

[0055] Optionally, generating a comprehensive image feature map using the fusion features of each organ includes: spatially registering the fused features with standard organ boundaries in a preset human anatomical atlas database; Specifically, the fused features are spatially registered with the standard organ boundaries in the preset human anatomical atlas database to ensure that each fused feature can accurately correspond to a specific organ in the human anatomical structure. This step is critical because it establishes the correspondence between the fused features and the anatomical structure, providing a basis for subsequent risk assessment.

[0056] Obtain the metabolic activity values ​​of each organ region after registration and sort them from large to small, select the metabolic activity value with the highest ranking, and extract the metabolic activity quantitative index; Specifically, the metabolic activity values ​​of each organ region after registration are obtained. These values ​​reflect the biological metabolic state of the organ and are important indicators for assessing the health status of the organ. These metabolic activity values ​​are sorted from large to small, and the metabolic activity value ranked first is selected to extract the metabolic activity quantitative index. This step highlights the organ region with the most active metabolism, which helps to quickly identify organs that may have abnormalities.

[0057] Obtain the average structural density of each organ region after registration, and extract the quantitative index of structural density; Specifically, the average structural density of each organ region after registration is obtained. This index reflects the compactness of the organ's tissue structure and is also an important basis for evaluating the health status of the organ. Based on these average structural densities, the quantitative index of structural density is extracted.

[0058] The metabolic activity quantification index and the structural density quantification index are used to generate a feature index table with three-dimensional space coding to obtain a comprehensive image feature map.

[0059] Specifically, the metabolic activity quantitative index and structural density quantitative index are used to generate a feature index table with three-dimensional spatial coding. This feature index table integrates the fusion characteristics, metabolic activity quantitative index and structural density quantitative index of each organ to form a comprehensive imaging feature map. This map not only contains rich organ information, but also has three-dimensional spatial coding, which is convenient for subsequent risk assessment and visualization processing.

[0060] For example, suppose that a patient's multimodal imaging data has gone through the previous processing steps and obtained fusion features. Next, these fusion features are spatially registered with the preset human anatomical atlas database. After registration, it was found that the metabolic activity value of the liver region was the highest, so the metabolic activity value of the liver was selected as the metabolic activity quantification index. At the same time, we also calculated the average structural density of the liver region as a structural density quantification index. Finally, these indicators were integrated into the feature index table to obtain the patient's comprehensive imaging feature atlas. This atlas clearly shows the metabolic activity and tissue structure characteristics of the liver, providing strong support for subsequent risk assessment.

[0061] Optionally, comparing the fusion features of different organs in the comprehensive image feature atlas with features in a preset typical case feature library includes: According to the organ code in the feature index table of the comprehensive image feature atlas, searching for a pathological template matching the corresponding organ in the typical case feature library; Specifically, a unique code is assigned to each organ, such as LUNG001 for the lung; typical cases of the same type of organ in the database are quickly retrieved based on the code, such as templates for benign tumors / malignant tumors / inflammation; the target location is determined through the coordinate information in the code, such as the specific area of ​​the tumor in the right upper lobe of the lung.

[0062] Obtain the metabolic activity quantitative index and the structural density quantitative index of the corresponding organ, calculate the similarity index with the pathological template, and ,have , In the formula, For the A quantitative indicator of the structural density of each organ. For the A quantitative indicator of the metabolic activity of an organ.

[0063] Specifically, the density value of the current organ is compared with the reference value recorded in the case database to evaluate the difference in metabolic activity; the score is high only when both are close to the reference value at the same time to avoid misjudgment of a single indicator; the smaller the similarity index, the greater the gap with the pathological template, and the lower the possibility of abnormality.

[0064] Optionally, generating a risk assessment value corresponding to the organ includes: Determine whether the similarity index is greater than a preset threshold; If the similarity index is greater than a preset threshold, the output risk assessment value is 0; Specifically, it is determined whether the similarity index of the corresponding organ in the comprehensive image feature map is greater than the preset threshold. This step is the key to risk assessment, which determines whether further risk calculation is required. If the similarity index is greater than the preset threshold, it means that the health status of the organ is less similar to the pathological template in the typical case feature library and is in a normal or low-risk state, so the risk assessment value is directly output as 0.

[0065] If the similarity index is less than a preset threshold, marking the pathology template as a similar case; Through the preset Bayesian network, the clinical manifestation probability of the similar cases is output to obtain the risk assessment value of the corresponding organ.

[0066] Specifically, the Bayesian network combines medical prior knowledge with current data to calculate probability. The input layer is the metabolic activity and structural density of the current patient's organs; the hidden layer is the pathological results of each similar case, such as 1 for lung cancer and 0 for tuberculosis; the output layer is the posterior probability that the current case is malignant; every time a true diagnosis is confirmed, the conditional probability table between nodes is automatically adjusted.

[0067] For example, assume that the patient data matches similar case 01 and similar case 28, where the malignancy probability of similar case 01 is 85%, the CT density is 145, and the PET metabolism is 22.3; the malignancy probability of similar case 28 is 72%, the CT density is 130, and the PET metabolism is 18.7; after weighted calculation, the density of the current patient's organ is 138, and the metabolism is 19.5, resulting in a comprehensive risk probability of 79.3%.

[0068] Optionally, the method further comprises: Obtaining the true pathological result of the corresponding organ, and performing evaluation according to the true pathological result to obtain a true evaluation value; Specifically, a feedback loop between artificial intelligence output and final diagnosis is established; surgical specimen pathology reports, puncture biopsy results, etc. are used as data calibration anchors; the warning value is set to 15%, and the database is updated when the difference is greater than 15%; templates related to misjudged cases are corrected first, such as increasing the weight of corresponding templates when malignant cases are misjudged as benign; a sandbox testing mechanism is adopted, such as the corrected parameters must be verified by historical cases before they take effect.

[0069] Calculating a difference between the actual assessment value and the risk assessment value of the corresponding organ; If the difference value is greater than the preset warning value, the weight parameter of the corresponding pathological template in the typical case feature library is reversely corrected.

[0070] Specifically, label each organ to be corrected according to the pathology report, such as the real label: benign / malignant / other; use the Manhattan distance calculation system to predict the absolute deviation between the risk value and the real value; reversely track the Top 5 similar templates called when comparing the case; apply a penalty factor to the weight of the wrong template; add annotations to the affected templates, such as "V2.1_Reason for correction: optimization after mucinous adenocarcinoma was misdiagnosed as pneumonia".

[0071] For example, suppose the system performs a risk assessment on a patient's lungs and gives a risk assessment value. However, through actual diagnosis, it is found that the patient's lungs have more serious lesions, which are significantly different from the risk assessment value. At this time, the difference between the true assessment value and the risk assessment value will be calculated, and it will be determined whether the difference is greater than the preset warning value. If the difference is greater than the warning value, the weight parameters of the pathological template related to lung lesions in the typical case feature library will be reversed to more accurately reflect the actual pathological conditions in future risk assessments. Through such a self-optimization mechanism, the system can continuously improve the accuracy and reliability of its risk assessment and provide doctors with more accurate and useful diagnostic auxiliary information.

[0072] Based on the same inventive concept, Figure 4As shown, the present invention also provides a medical image recognition and processing system based on multimodal image fusion, the system comprising: An image data acquisition module, used to acquire at least two types of medical images from different devices and perform corrections to obtain an image data set; A feature extraction module, used to input the image data set into a preset parallel neural network to extract tracheal tissue structure features and biological metabolism features; A feature map generation module, used to fuse the tracheal tissue structure feature and the biological metabolism feature to obtain a fusion feature, and generate a comprehensive image feature map using the fusion feature of each organ; A risk assessment module, used to compare the fusion features of different organs in the comprehensive image feature map with the features in a preset typical case feature library to generate a risk assessment value for the corresponding organ; The visualization module is used to associate the risk assessment value with the corresponding organ in the preset AR visualization image.

[0073] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean direct connection of the lines, and the indirect connection mode can be applied to the embodiments of the present invention as long as the purpose of the present invention is achieved. The above is only an exemplary embodiment of the present invention and cannot be used to limit the scope of the present invention.

[0074] That is, any equivalent changes and modifications made according to the teachings of the present invention are still within the scope of the present invention. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present invention. This application is intended to cover any variation, use or adaptive change of the present invention, which follows the general principles of the present invention and includes common knowledge or customary technical means in the art that are not described in the present invention.

Claims

1. A medical image recognition and processing method based on multimodal image fusion, characterized in that: The method comprises: Acquire at least two types of medical images from different devices and perform corrections to obtain an image dataset; Inputting the image data set into a preset parallel neural network to extract tracheal tissue structure characteristics and biological metabolism characteristics; The tracheal tissue structure feature and the biological metabolism feature are fused to obtain a fusion feature, and a comprehensive image feature map is generated using the fusion feature of each organ; Comparing the fusion features of different organs in the comprehensive image feature atlas with the features in a preset typical case feature library to generate a risk assessment value for the corresponding organ; The risk assessment value is associated with a corresponding organ in a preset AR visualization image, so as to obtain a corresponding risk assessment value through the corresponding organ in the AR visualization image.

2. The medical image recognition and processing method based on multimodal image fusion according to claim 1, characterized in that: The obtaining of at least two types of medical images from different devices and performing correction to obtain an image data set includes: Align the medical images from different devices in time sequence to obtain an initialized image dataset; Mapping the image coordinate system in the initialized image data set to a unified human anatomical coordinate system, and outputting spatially aligned multimodal image data; Performing noise reduction processing on the multimodal image data using a preset nonlinear filtering algorithm, and outputting noise-reduced image data; The grayscale values ​​of the noise-reduced image data are normalized and calibrated based on the differences in device parameters to generate an image data set with a unified quantization range.

3. The medical image recognition and processing method based on multimodal image fusion according to claim 1, characterized in that: The step of inputting the image data set into a preset parallel neural network to extract tracheal tissue structure characteristics and biological metabolism characteristics includes: Taking any image data in the image data set as a reference, performing voxel interpolation processing on each image data in the image data set; For the image data after voxel interpolation processing, the mechanical distortion area generated in the mapping process is compensated by bidirectional linear interpolation to obtain the final image data set.

4. The medical image recognition and processing method based on multimodal image fusion according to claim 1, characterized in that: The step of inputting the image data set into a preset parallel neural network to extract tracheal tissue structure characteristics and biological metabolism characteristics includes: The final image dataset is input into a 3D U-Net network with dilated convolution to output the tracheal tissue structure features; The final image dataset is input into a multi-scale feature pyramid network to output biological metabolic features.

5. The medical image recognition and processing method based on multimodal image fusion according to claim 1, characterized in that: The fusion of the tracheal tissue structure feature and the biological metabolism feature to obtain the fusion feature includes: Obtain the structural segmentation accuracy score of the current region, calculate the structural weight value of the tracheal tissue structural feature, and ,have ， In the formula, is an exponential function, Score the accuracy of the structure segmentation in the current region, is the adjustment factor; Obtain the activity standard deviation of the biological metabolic characteristics of the current area and the activity standard deviation of the biological metabolic characteristics in the entire map, calculate the metabolic weight value of the biological metabolic characteristics, and ,have ， In the formula, is the activity standard deviation of the biological metabolic characteristics of the current area, The standard deviation of the activity of the biological metabolic features in the whole map, is the maximum value function; The structural weight value and the metabolic weight value are used to calculate the fusion feature. ,have ， In the formula, The tracheal tissue structure characteristics, is the biological metabolic characteristic, is the outer product of the tensor.

6. The medical image recognition and processing method based on multimodal image fusion according to claim 1, characterized in that: The step of generating a comprehensive image feature map using the fusion features of each organ includes: spatially registering the fused features with standard organ boundaries in a preset human anatomical atlas database; Obtain the metabolic activity values ​​of each organ region after registration and sort them from large to small, select the metabolic activity value with the highest ranking, and extract the metabolic activity quantitative index; Obtain the average structural density of each organ region after registration, and extract the quantitative index of structural density; The metabolic activity quantification index and the structural density quantification index are used to generate a feature index table with three-dimensional space coding to obtain a comprehensive image feature map.

7. The medical image recognition and processing method based on multimodal image fusion according to claim 6, characterized in that: The comparing the fusion features of different organs in the comprehensive image feature atlas with the features in a preset typical case feature library includes: According to the organ code in the feature index table of the comprehensive image feature atlas, searching for a pathological template matching the corresponding organ in the typical case feature library; Obtain the metabolic activity quantitative index and the structural density quantitative index of the corresponding organ, calculate the similarity index with the pathological template, and ,have ， In the formula, For the A quantitative indicator of the structural density of each organ. For the A quantitative indicator of the metabolic activity of an organ.

8. The medical image recognition and processing method based on multimodal image fusion according to claim 7, characterized in that: Generating the risk assessment value of the corresponding organ includes: Determine whether the similarity index is greater than a preset threshold; If the similarity index is greater than a preset threshold, the output risk assessment value is 0; If the similarity index is less than a preset threshold, marking the pathology template as a similar case; Through the preset Bayesian network, the clinical manifestation probability of the similar cases is output to obtain the risk assessment value of the corresponding organ.

9. The medical image recognition and processing method based on multimodal image fusion according to claim 1, characterized in that: The method further comprises: Obtaining the true pathological result of the corresponding organ, and performing evaluation according to the true pathological result to obtain a true evaluation value; Calculating a difference between the actual assessment value and the risk assessment value of the corresponding organ; If the difference value is greater than the preset warning value, the weight parameter of the corresponding pathological template in the typical case feature library is reversely corrected.

10. A medical image recognition and processing system based on multimodal image fusion, applied to a medical image recognition and processing method based on multimodal image fusion as claimed in any one of claims 1 to 9, characterized in that: The system comprises: An image data acquisition module, used to acquire at least two types of medical images from different devices and perform corrections to obtain an image data set; A feature extraction module, used to input the image data set into a preset parallel neural network to extract tracheal tissue structure features and biological metabolism features; A feature map generation module, used to fuse the tracheal tissue structure feature and the biological metabolism feature to obtain a fusion feature, and generate a comprehensive image feature map using the fusion feature of each organ; A risk assessment module, used to compare the fusion features of different organs in the comprehensive image feature map with the features in a preset typical case feature library to generate a risk assessment value for the corresponding organ; The visualization module is used to associate the risk assessment value with the corresponding organ in the preset AR visualization image.

Images