Multi-modal medical image data fusion analysis system based on deep learning

Abstract

The invention discloses a multi-modal medical image data fusion analysis system based on deep learning, and relates to the field of medical images, and the system comprises a medical image module which is used for obtaining medical image data of different modals and multiple angles; the image recognition module is used for recognizing part contours in the medical image data of all the modalities, and the image recognition module aligns all the medical image data based on the part contours in all the medical image data so as to obtain fusion data of all the modalities at different angles; and the mapping module is used for extracting part contours of the medical image data of each modal from the image recognition module and generating a medical model of a corresponding dimension based on each part contour. Through efficient fusion of multi-modal image data, limitation of a single-modal image is effectively overcome, advantages of different modals are combined through introduction of multi-modal data fusion, and more comprehensive information support is provided for accurate identification of a focus area.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging, and specifically to a multi-modal medical imaging data fusion and analysis system based on deep learning. Background Art

[0002] With the progress of medical imaging technology, multi-modal image data fusion has become an effective method to solve the limitations of single modalities. By fusing data from different imaging modalities (such as CT, MRI, PET, etc.), the advantages of each modality can be fully utilized to provide more comprehensive lesion feature information. This fusion technology helps to make up for the deficiencies of single-modal data and improve the accuracy of lesion recognition and disease diagnosis; After retrieval, a Chinese patent (Publication No.: CN111476772B) discloses a method and device for analyzing lesions in medical images. The patent includes: extracting lesion characterization data based on medical image data; extracting lesion feature information based on the lesion characterization data; inputting the lesion feature information into a first machine learning model to obtain a first lesion feature vector; inputting the medical image data and the lesion characterization data into a second machine learning model to obtain a second lesion feature vector; combining the first lesion feature vector and the second lesion feature vector to obtain a fusion feature vector corresponding to the lesion; and obtaining an analysis result of the lesion according to the fusion feature vector.

[0003] In the prior art, how to make full use of information from different modalities, improve the accuracy of lesion recognition, and achieve efficient fusion of lesion regions is an important issue in multi-modal image analysis. Therefore, the present invention proposes a multi-modal medical imaging data fusion and analysis system based on deep learning. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-modal medical imaging data fusion and analysis system based on deep learning to solve the problems mentioned in the above background art.

[0005] The present invention can be realized by the following technical solutions: A multi-modal medical imaging data fusion and analysis system based on deep learning includes a medical imaging module, an image recognition module, a deep learning module, a mapping module, and a visualization module; The medical imaging module is used to obtain medical imaging data at multiple angles of each modality. The medical imaging data of each modality includes CT (Computed Tomography), MRI (Magnetic Resonance Imaging), PET (Positron Emission Tomography), ultrasound images, etc. The medical imaging data of each modality provides pathological and physiological information in different dimensions; The image recognition module is used to recognize the part contours in various modal medical image data, and based on the part contours in each medical image data, aligns each medical image data, that is, stretches or shrinks the patient contour part in each medical image data to the same size, so as to obtain fusion data of different angles in each modality; The mapping module is used to extract the part contours of the medical image data in each modality from the image recognition module and generate corresponding medical models; The deep learning module is designed and trained based on a deep neural network (such as a convolutional neural network CNN). By learning a large amount of labeled data, the deep learning model can automatically identify the features in the image data and extract the lesion regions in the fusion data of each modality; The deep learning module fuses the lesion regions in the fusion data of different angles in the same modality to obtain the fusion lesion region corresponding to the modality; The mapping module collects each fusion lesion region and projects each fusion lesion region onto the corresponding medical model, and generates a lesion matrix in the medical model based on the coincidence degree of each fusion lesion region; The higher the coincidence degree of the fusion lesion regions, the larger the corresponding matrix value; The visualization module visually displays the medical model and, based on the matrix value of the lesion matrix, displays the lesion region with corresponding marks in the medical model.

[0006] A further technical improvement of the present invention lies in that the method for the deep learning module to obtain the fusion lesion region includes the following steps: S1. Align the fusion data of different angles in the same modality to the same first coordinate system; S2. Extract the lesion regions from the fusion data of each angle and project these regions into a unified space; S3. Calculate the clarity of the lesion regions in the fusion data of each angle , and the formula used is ; In the formula, is the contrast of the lesion region in angle i, which is obtained by calculating the gray level difference within the region; is the clarity of the lesion region in angle i, which is obtained by calculating the gradient of the boundary of the lesion region; S4. Normalize the clarity of the lesion regions in the fusion data of each angle so that the weights of all the fusion data of different angles sum to 1, and the formula used is ; In the formula, is the weight corresponding to the fusion data in angle i, and N is the number of the fusion data of different angles; S5. Perform weighted fusion based on the clarity of lesions in the data fused from different angles. For the lesion area of each angle weight it according to its weight to obtain the fused lesion area , and its calculation formula is: ; S6. Finally, use boundary smoothing or refinement methods to further optimize the edge of the fused lesion area and output it.

[0007] A further technical improvement of the present invention is that: the mapping module divides the fused lesion areas of each modality and the medical model into regional units, that is, each regional unit represents a spatial area for subsequent calculations; For the regional units corresponding to the fused lesion areas of each modality, the mapping module sets a non-zero initial matrix value to avoid the problem that in the subsequent calculation of the matrix values of the fused lesion areas of each modality, if there is no overlapping lesion part between the fused lesion area of a certain modality and the fused lesion areas of the remaining modalities, the lesion area cannot be reflected in the lesion matrix.

[0008] A further technical improvement of the present invention is that: the mapping unit represents each regional unit in the medical model with a second coordinate system, that is, by assigning coordinates on the second coordinate system to each regional unit in the lesion area.

[0009] A further technical improvement of the present invention is that: the mapping module calculates whether there is a closed area with a closed matrix value in the lesion matrix (that is, whether there is a non-zero matrix value in the adjacent areas of the lesion matrix). If so, the corresponding closed area in the corresponding lesion matrix is also marked as a lesion area, and the visualization module displays it using a preset mark.

[0010] A further technical improvement of the present invention is that: the method for the mapping module to calculate the closed area includes: Z1. The mapping module identifies the lesion matrix M. Each regional unit in the lesion matrix M represents whether there is a non-zero matrix value at the position ( ). If so, it is marked as a lesion unit, and the lesion area is composed of multiple adjacent lesion units; Z2. The mapping module uses the connected component analysis method to traverse the lesion matrix to find adjacent lesion units. For each regional unit , if its matrix value is non-zero, check whether the matrix values of the regional units in its neighborhood are also non-zero; Z3. Calculate the closed region. When there are similar matrix values in the neighborhoods of two or more lesion units, they are considered to be "closed". Specifically, if the matrix values of adjacent regions are all non-zero, and non-zero regions are found in the neighborhood, these regions are merged into one lesion region. Z4. Mark the closed region. Set the same ID for the closed region, indicating that the closed region is a complete lesion region. Z5. Output the marked result of the lesion matrix. Each lesion region has a unique ID to correspond to different lesion regions.

[0011] A further technical improvement of the present invention is that: the mapping module presets a center point in the second coordinate system, and the mapping unit calculates the distance and the corresponding angle from the center point to each lesion region, and the mapping unit counts the size of each lesion region in the second coordinate system, so as to generate a lesion relationship set of the medical model at the corresponding time point. The specific steps include: Q1. Calculate the distance and angle between the center point of the coordinate system and each lesion region. When calculating the distance, the Euclidean distance formula is used to calculate the distance from each lesion region to the preset coordinate point. When calculating the angle, the trigonometric function is used to calculate the angle of the lesion region relative to the center point. Q2. For each lesion region, the mapping module calculates its proportion in the lesion matrix. Q3. The mapping module saves the proportion size, distance and angle of each lesion region to a data structure. Q4. The mapping module integrates each data structure to obtain a lesion relationship set of the medical model at the corresponding time point.

[0012] A further technical improvement of the present invention is that: the mapping module establishes multiple medical models along the time axis, and the mapping module uses the medical model at the initial time node as the basis to align the medical models and the corresponding lesion matrices at the remaining time points.

[0013] A further technical improvement of the present invention is that: the mapping module takes the center point in each medical model as the reference point to calculate the change trend of each lesion relationship set along the time axis, including: Lesion proportion analysis: Evaluate the change of the proportion of the lesion region at each time point to judge the expansion or contraction of the lesion. Lesion distance analysis: Calculate the change of the distance of the lesion region at different time points, and whether there are new lesions generated, to assist in judging whether the lesion has metastasized or spread. Lesion angle analysis: Calculate the change of the angle of the lesion region relative to the center point to assist in judging the rotation or directional change of the lesion region.

[0014] Compared with the prior art, the present invention has the following beneficial effects: Through the efficient fusion of multi-modal image data, the present invention effectively overcomes the limitations of single-modal images. By introducing multi-modal data fusion, the advantages of different modalities are combined, providing more comprehensive information support for the accurate identification of lesion regions. Moreover, through the automatic extraction and fusion of lesion features by a deep learning model, on the basis of high-precision registration, the accuracy of lesion detection can be improved, making up for the deficiencies of traditional single-modal methods in diagnosis; The present invention uses deep learning algorithms to automatically identify and extract lesion regions in different-modal images, which can accurately capture the spatial distribution, morphology, and features of lesions. By combining the advantages of multi-modal images, the system realizes a comprehensive analysis of lesion regions, thereby improving the accuracy and robustness of lesion detection. Additionally, through the generation of lesion matrices and the calculation of the coincidence degree of lesion regions, the lesion regions of different modalities can be quantified in the same coordinate system, assisting users in making diagnoses based on information such as the spatial distribution and morphological changes of lesions, and reducing diagnostic errors caused by insufficient single-modal data; On the other hand, by introducing the analysis of lesions at multiple time points and combining deep learning and lesion matrix generation technologies, the present invention provides tools for the dynamic monitoring of diseases and the evaluation of treatment effects; by analyzing the set of lesion relationships at different time points, it can help users track the changes of lesions in real time, evaluate the treatment effects, and predict the progression of diseases, not only improving the accuracy of early disease diagnosis, but also providing strong support for the formulation of personalized treatment plans and long-term disease management. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For the convenience of those skilled in the art to understand, the present invention will be further described below with reference to the accompanying drawings.

[0016] Figure 1 is the system block diagram of the present invention; Figure 2 is the system mind map of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] To further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following will, with reference to the accompanying drawings and preferred embodiments, describe in detail the specific embodiments, structures, features, and effects according to the present invention.

[0018] Embodiment 1 Please refer to Figure 1-2 As shown, the present invention provides a multi-modal medical image data fusion and analysis system based on deep learning, including a medical image module, an image recognition module, a deep learning module, a mapping module, and a visualization module; The medical imaging module is used to acquire medical imaging data at multiple angles of each modality. The medical imaging data of each modality includes CT (Computed Tomography), MRI (Magnetic Resonance Imaging), PET (Positron Emission Tomography), ultrasound images, etc. The medical imaging data of each modality is used to provide pathological and physiological information in different dimensions; The image recognition module is used to recognize the part contours in the medical imaging data of each modality. And based on the part contours in each medical imaging data, the image recognition module aligns each medical imaging data, that is, stretches or shrinks the patient contour part in each medical imaging data to the same size, so as to obtain the fusion data of different angles of each modality; The mapping module is used to extract the part contours of the medical imaging data of each modality from the image recognition module and generate corresponding medical models; The deep learning module is designed and trained based on a deep neural network (such as a convolutional neural network CNN). By learning a large amount of labeled data, the deep learning model can automatically identify the features in the imaging data and extract the lesion areas in the fusion data of each modality. And the deep learning module includes: Convolutional layer: Through convolution operations, the network can automatically learn local features in the image (such as edges, textures, shapes, etc.; CNN scans the image through a set of filters (convolution kernels) and extracts features; each convolutional layer extracts feature information at different scales and levels; Activation layer: A activation layer follows the convolutional layer, which introduces non-linearity, enabling the model to learn more complex feature relationships; Pooling layer: It is used for downsampling, reducing the spatial dimension of the data, extracting more abstract information, and enhancing the robustness of the model; the pooling layer can reduce the impact of position changes in the image on the model output; Fully connected layer: In the later stage of CNN, the extracted features will be input into the fully connected layer for higher-level analysis and decision-making; the fully connected layer is used to synthesize the features extracted by the previous layers and finally output the position, size or category of the lesion area; The steps for the deep learning module to extract the lesion areas in the fusion data of each modality include: A1. Use the imaging data of different modalities to train the model, and preprocess each modality of imaging data before use, including normalization, denoising, and data augmentation; A2. Use CNN or other networks to extract the features in the fusion data of each modality, learn the local features in the fusion data (including textures, shapes, boundaries, etc.) to obtain the deep learning model; A3. Identify the fusion data of each modality through the trained deep learning model in A2 to obtain the lesion areas in the fusion data of each modality; A4. Output the lesion regions in the fused data corresponding to each modality; The deep learning module fuses the lesion regions in the fused data at different angles in the same modality to obtain the fused lesion region corresponding to the modality; The method by which the deep learning module obtains the fused lesion region includes the following steps: S1. Align the fused data at different angles in the same modality to the same first coordinate system; In this process, a spatial registration method is adopted, that is, for two fused data and , the deep learning module generates a transformation matrix T. After transformation, the content of the fused data and the fused data are aligned; S2. Extract the lesion regions from the fused data at each angle and project these regions into a unified space; S3. Calculate the clarity of the lesion regions in the fused data at each angle , and the formula used is ; in the formula, is the contrast of the lesion region at angle i, which is obtained by calculating the gray level difference within the region; is the clarity of the lesion region at angle i, which is obtained by calculating the gradient of the boundary of the lesion region; S4. Normalize the clarity of the lesion regions in the fused data at each angle so that the weights of all the fused data at each angle sum to 1, and the formula used is ; in the formula, is the weight corresponding to the fused data at angle i, and N is the number of fused data at different angles; S5. Perform weighted fusion based on the clarity of the lesions in the fused data at different angles, and weight the lesion region at each angle according to its weight to obtain the fused lesion region , and its calculation formula is: ; S6. Finally, use a boundary smoothing or thinning method to further optimize the edge of the fused lesion region and output it.

[0019] The mapping module collects each fused lesion region, projects each fused lesion region onto the corresponding medical model, and generates a lesion matrix in the medical model based on the overlap degree of each fused lesion region; The higher the coincidence degree of the fused lesion area, the larger the corresponding matrix value. Through the generated lesion matrix, the correlation and consistency between different modalities can be further analyzed. For example, the row and column patterns of the matrix can be used to evaluate which modalities have the most consistent lesion areas and which modalities may deviate in certain specific types of lesion areas. The mapping module divides the fused lesion areas of each modality and the medical model into regional units, that is, each regional unit represents a spatial area for subsequent calculations. The mapping module sets a non-zero initial matrix value for each regional unit corresponding to the fused lesion area of each modality to avoid the problem that in the subsequent calculation of the matrix value of the fused lesion area of each modality, if there is no lesion overlap between the fused lesion area of a certain modality and the fused lesion areas of the other modalities, the lesion area cannot be reflected in the lesion matrix.

[0020] The mapping unit represents each regional unit in the medical model with a second coordinate system, that is, by assigning coordinates on the second coordinate system to each regional unit in the lesion area. In a two-dimensional space, the coordinates of each regional unit in the lesion matrix are (x, y).

[0021] The mapping module calculates whether there is a closed area with a closed matrix value in the lesion matrix (that is, whether there is a non-zero matrix value in the adjacent areas of the lesion matrix). If so, the corresponding closed area in the corresponding lesion matrix is also marked as a lesion area, and the visualization module uses the color preset in the matrix value-color mapping table for marking.

[0022] The method for the mapping module to calculate the closed area includes: Z1. The mapping module identifies the lesion matrix M, and each regional unit in the lesion matrix M represents whether there is a non-zero matrix value at the position ( ). If so, it is marked as a lesion unit, and the lesion area consists of multiple adjacent lesion units. In this embodiment, =1 represents that there is a non-zero matrix value at the position ( ), that is, this position is a lesion unit. =0 indicates that there is no non-zero matrix value at ( ), which is a background unit. Z2. The mapping module uses the connected component analysis method to traverse the lesion matrix to find adjacent lesion units. For each unit , if its value is 1, check whether its neighboring pixels (neighborhoods such as up, down, left, right, diagonal, etc.) also have a matrix value of 1. Z3. Calculate the closed region. When there are similar matrix values in the neighborhoods of two or more lesion units, they are considered "closed"; specifically, if the matrix values of adjacent regions are all non-zero and the same non-zero regions are found within the neighborhood, these regions are merged into one lesion region. Z4. Mark the closed region. Set the same ID for the closed region, indicating that this closed region is a complete lesion region. Z5. Output the marked result of the lesion matrix. Each lesion region has a unique ID to correspond to different lesion regions.

[0023] The visualization module visually displays the medical model and, based on the matrix values of the lesion matrix, marks the lesion regions with corresponding markings in the medical model. In this embodiment, the visualization module is provided with a matrix value-color mapping table. The visualization module marks the specific parts of the medical model with corresponding colors based on the specific matrix values of the lesion matrix in the medical model.

[0024] Embodiment 2 A multi-modal medical image data fusion and analysis system based on deep learning includes a medical image module, an image recognition module, a deep learning module, a mapping module, and a visualization module. The medical image module is used to obtain medical image data of multiple angles of each modality. The medical image data of each modality includes CT (Computed Tomography), MRI (Magnetic Resonance Imaging), PET (Positron Emission Tomography), ultrasound images, etc. The medical image data of each modality provides different-dimensional pathological and physiological information. The image recognition module is used to recognize the part contours in the medical image data of each modality, and based on the part contours in each medical image data, align each medical image data, that is, stretch or shrink the patient contour part in each medical image data to the same size, so as to obtain the fusion data of different angles of each modality. The mapping module is used to extract the part contours of the medical image data of each modality from the image recognition module and generate a corresponding medical model. The deep learning module recognizes the features in the image data and extracts the lesion regions from the fusion data of each modality. The deep learning module fuses the lesion regions extracted from the fusion data of different angles in the same modality to obtain the fusion lesion region corresponding to the modality. The method for the deep learning module to obtain the fusion lesion region includes the following steps: S1. Align the fusion data of different angles in the same modality to the same first coordinate system. S2. Extract the lesion regions from the fusion data of each angle and project these regions into a unified space. S3. Calculate the clarity of the lesion area in the fused data for each angle , and the formula used is ; In the formula, is the contrast of the lesion area at angle i, which is obtained by calculating the gray-scale difference within the area; is the clarity of the lesion area at angle i, which is obtained by calculating the gradient of the boundary of the lesion area; S4. Normalize the clarity of the lesion area in the fused data for each angle so that the sum of the weights of the fused data for all angles is 1, and the formula used is ; In the formula, is the weight corresponding to the fused data at angle i, and N is the number of fused data for different angles; S5. Perform weighted fusion based on the clarity of the lesions in the fused data for different angles. The lesion area for each angle is weighted according to its weight to obtain the fused lesion area , and its calculation formula is: ; S6. Finally, use boundary smoothing or thinning methods to further optimize the edge of the fused lesion area and output it.

[0025] The mapping module collects each fused lesion area and projects each fused lesion area onto the corresponding medical model, and based on the overlap degree of each fused lesion area, generates a lesion matrix in the medical model; For the fused lesion area with a higher overlap degree, the corresponding matrix value is larger; The mapping module divides the fused lesion area of each modality and the medical model into regional units; The mapping module sets a non-zero initial matrix value for each regional unit corresponding to the fused lesion area of each modality.

[0026] The mapping unit represents each regional unit in the medical model with a second coordinate system, that is, by assigning coordinates on the second coordinate system to each regional unit in the lesion area. In three-dimensional space, the coordinates of each regional unit in the lesion matrix are (x, y, z).

[0027] The mapping module calculates whether there is a closed area with a closed matrix value in the lesion matrix (that is, whether there is a non-zero matrix value in the adjacent areas of the lesion matrix). If so, the corresponding closed area in the corresponding lesion matrix is also marked as a lesion area, and the visualization module uses the color preset in the matrix value-color mapping table for marking.

[0028] Method for the mapping module to calculate the closed region, including: Z1. The mapping module identifies the lesion matrix M, and each regional unit in the lesion matrix M represents whether there is a non-zero matrix value at the position . If there is, it is marked as a lesion unit, and the lesion area consists of multiple adjacent lesion units; The mapping module uses the connected component analysis method to traverse the lesion matrix and find adjacent lesion units. For each regional unit , if its matrix value is non-zero, check whether the matrix value of the regional unit in its neighborhood is also non-zero; Z2. The mapping module uses the connected component analysis method to traverse the lesion matrix and find adjacent lesion units. For each unit , if its value is 1, check whether the pixels in its neighborhood (such as the upper, lower, left, right, diagonal neighborhoods) also have a matrix value of 1; Z3. Calculate the closed region. When the neighborhoods of two or more lesion units have similar matrix values, they are considered "closed"; specifically, if the matrix values of adjacent regions are all non-zero and the same non-zero regions are found in the neighborhood, these regions are merged into one lesion region; Z4. Mark the closed region, set the same ID for the closed region, indicating that the closed region is a complete lesion region; Z5. Output the marked result of the lesion matrix. Each lesion region has a unique ID to correspond to different lesion regions.

[0029] The visualization module visually displays the medical model, and based on the matrix values of the lesion matrix, the lesion regions are displayed with corresponding marks in the medical model; In this embodiment, the visualization module is provided with a matrix value-color mapping table. The visualization module marks with corresponding colors at specific parts of the medical model based on the specific matrix values of the lesion matrix in the medical model.

[0030] The mapping module presets a center point in the second coordinate system, and the mapping unit calculates the distance and corresponding angle from the center point to each lesion region, and the mapping unit statistics the size of each lesion region in the second coordinate system, so as to generate the lesion relationship set of the medical model at the corresponding time point. The specific steps include: Q1. Calculate the distance and angle between the center point of the coordinate system and each lesion region; When calculating the distance, the Euclidean distance formula is used to calculate the distance from each lesion region to the preset coordinate point; In three-dimensional space, the formula used is: ; In the formula, ( , , are the coordinates of the center point; are the coordinates of the geometric center of the corresponding lesion area, and its acquisition formula is: , , ; , , are the coordinates of the lesion area in the coordinate system; When calculating the angle, the trigonometric function is used to calculate the angle of the lesion area relative to the center point; In three-dimensional space, calculate the pitch angle and azimuth angle ; ; ; Q2. For each lesion area, the mapping module calculates its proportion in the lesion matrix; Q3. The mapping module saves the proportion size, distance, and angle of each lesion area to a data structure; Q4. The mapping module integrates each data structure to obtain the lesion relationship set of the medical model at the corresponding time point.

[0031] The mapping module establishes multiple medical models along the time axis, and the mapping module uses the medical model at the initial time node as the basis to align the medical models at the remaining time points and the corresponding lesion matrices.

[0032] The mapping module takes the center point in each medical model as the reference point and calculates the change trend of each lesion relationship set along the time axis, including: Lesion proportion analysis: Evaluate the change in the proportion of the lesion area at each time point to judge the expansion or contraction of the lesion; Lesion distance analysis: Calculate the change in the distance of the lesion area at different time points and whether there are new lesions generated to assist in judging whether the lesion has metastasized or spread; Lesion angle analysis: Calculate the change in the angle of the lesion area relative to the center point to assist in judging the rotation or directional change of the lesion area; In this embodiment, the system uses statistical methods to analyze the long-term change trend of the lesion area to facilitate tracking the change of the lesion area.

[0033] The above are only the preferred embodiments of the present invention and do not impose any formal limitations on the present invention. Although the present invention has been disclosed above with the preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments with equivalent changes within the scope of the technical solution of the present invention. However, as long as it does not depart from the content of the technical solution of the present invention, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A multi-modal medical image data fusion and analysis system based on deep learning, characterized in that , including: A medical imaging module for acquiring medical imaging data of different modalities and multiple angles; An image recognition module for identifying the part contours in the medical imaging data of each modality, and based on the part contours in each medical imaging data, aligning each medical imaging data to obtain fused data of different angles of each modality; A mapping module for extracting the part contours of the medical imaging data of each modality from the image recognition module and generating a medical model of the corresponding dimension based on each part contour; A deep learning module for extracting the lesion regions from the fused data of each modality, and fusing the lesion data from different angles in the same modality to form a fused lesion region of the corresponding modality; The mapping module collects each fused lesion region and projects each fused lesion region onto the corresponding medical model, and based on the coincidence degree of each fused lesion region, the mapping module generates a matrix value in the corresponding part of the medical model, and the mapping module integrates each matrix value to generate a lesion matrix of the corresponding medical model; A visualization module for visually displaying the medical model, and based on the matrix values of the lesion matrix, the visualization module displays the lesion regions with corresponding marks in the medical model.

2. The multi-modal medical image data fusion and analysis system based on deep learning according to claim 1, wherein The method for the deep learning module to obtain the fused lesion region includes the following steps: S1. Align the fused data of the same modality and different angles to the same first coordinate system; S2. Extract the lesion regions from the fused data of each angle and project these regions into a unified space; S3. Calculate the clarity of the lesion area in the fused data for each angle , and the formula used is ; in the formula, is the contrast of the lesion area in angle i; is the clarity of the lesion area in angle i; S4. Normalize the clarity of the lesion area in the fused data at each angle so that the weights of all the fused data at different angles sum up to 1. The formula used is ; in the formula, is the weight corresponding to the fused data at angle i, and N is the number of fused data at different angles; S5. Perform weighted fusion based on the clarity of lesions in the data fused from different angles, and weight the lesion areas at each angle according to their weights to obtain a fused lesion area and ; S6. Finally, use the boundary smoothing or refinement method to further optimize the edge of the fused lesion region and output it.

3. The multi-modal medical image data fusion and analysis system based on deep learning according to claim 1, characterized in that, The mapping module divides the fused lesion regions and the medical models of each modality into regional units; And the mapping unit represents each regional unit in the medical model with a second coordinate system; The mapping module sets a non-zero initial matrix value for each regional unit corresponding to the fused lesion regions of each modality.

4. The multimodal medical image data fusion and analysis system based on deep learning according to claim 3, wherein The mapping module calculates whether there is a closed region with closed matrix values in the lesion matrix; If it exists, mark the corresponding closed region in the corresponding lesion matrix as a lesion region, and the visualization module uses a preset mark for display.

5. The multi-modal medical image data fusion and analysis system based on deep learning according to claim 4, wherein The method for the mapping module to calculate the closed region includes: Z1. The mapping module identifies the lesion matrix M, and each regional unit in the lesion matrix M represents whether there is a non-zero matrix value at the position ( ). If it exists, mark it as a lesion unit; Z2. The mapping module uses the connected component analysis method to traverse the lesion matrix and find adjacent lesion units. For each regional unit , if its matrix value is non-zero, check whether the matrix values of the regional units in its neighborhood are also non-zero; Z3. Calculate the closed region. When there are non-zero matrix values in the neighborhood of two or more lesion units, the mapping module considers it as a "closed" lesion region; Z4. Mark the closed region and set the same ID for the closed region; Z5. Output the marked result of the lesion matrix.

6. The multi-modal medical image data fusion and analysis system based on deep learning according to claim 5, characterized in that The mapping module presets a center point in the second coordinate system, and the mapping unit calculates the distance and corresponding angle from the center point to each lesion region, and the mapping unit statistics the size of each lesion region in the second coordinate system, so as to generate a lesion relationship set of the medical model at the corresponding time point.

7. The multi-modal medical image data fusion and analysis system based on deep learning according to claim 6, characterized in that, The steps for the mapping module to generate the lesion relationship set include: Q1. Calculate the distance and angle between the center point of the coordinate system and each lesion region; Q2. For each lesion region, the mapping module calculates its proportion in the lesion matrix; Q3. The mapping module saves the proportion size, distance, and angle of each lesion area to a data structure; Q4. The mapping module integrates each data structure to obtain the lesion relationship set of the medical model at the corresponding time point.

8. The multi-modal medical image data fusion and analysis system based on deep learning according to claim 7, characterized in that The mapping module establishes multiple medical models along the time axis, and based on the medical model at the initial time node, the mapping module aligns the medical models and the corresponding lesion matrices at the remaining time points; The mapping module calculates the change trend of each lesion relationship set along the time axis with the center point in each medical model as the reference point.

Images