Visual pathway imaging method and device based on multi-modal fusion

Abstract

The invention provides a visual path imaging method and equipment based on multi-modal fusion. The method comprises the following steps: acquiring image data of different modals; performing alignment operation on the image data of different modalities in space by using a nonlinear matching algorithm; key features in the image data of different modalities are extracted respectively; fusing different types of key features based on the image data; the multi-modal feature image is preprocessed; and constructing a three-dimensional model of the visual path. According to the invention, by integrating various imaging technologies of image data of different modalities, the complex structure and function of a visual pathway can be revealed from different angles and levels. According to the multi-modal fusion visual pathway imaging method, the trend and integrity of white matter fibers in the brain and a high-resolution image of a retina structure can be revealed, and detailed information of a functional activity area in the brain and the brain structure can be detected, so that the multi-modal fusion visual pathway imaging method improves the accuracy and comprehensiveness of visual pathway imaging.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional imaging of visual pathways, and in particular to a method and device for visual pathway imaging based on multimodal fusion. Background Art

[0002] In recent years, biomedical imaging technology has developed rapidly and can depict the structure and function of living organisms at different scales, such as living organisms, tissues, and molecules. Among them, the visual pathway is an important way for humans to perceive external information, and the integrity of its structure and function is crucial to maintaining normal visual experience. However, early diagnosis and effective treatment of visual pathway diseases (such as glaucoma, optic neuropathy, etc.) have always been a difficult problem faced by the medical community. Traditional single-modality imaging methods, such as optical imaging and magnetic resonance imaging (MRI), can provide information on a certain aspect of the visual pathway, but often have problems such as incomplete information, insufficient resolution, or susceptibility to interference. Summary of the invention

[0003] The present invention aims to at least solve the technical problem of incomplete information in the prior art, and in particular innovatively proposes a visual pathway imaging method and device based on multimodal fusion.

[0004] In order to achieve the above-mentioned object of the present invention, the present invention provides a visual pathway imaging method based on multimodal fusion, the method comprising:

[0005] S1. Acquisition of diffusion tensor images, optical coherence tomography images, functional magnetic resonance images and structural magnetic resonance images;

[0006] S2. spatially aligning the diffusion tensor image, the optical coherence tomography image, the functional magnetic resonance image, and the structural magnetic resonance image using a nonlinear matching algorithm to obtain aligned image data;

[0007] S3, respectively extracting key features from the diffusion tensor image, the optical coherence tomography image, the functional magnetic resonance image, and the structural magnetic resonance image;

[0008] S4, fusing the key features of different types based on the aligned image data to obtain a multimodal feature image;

[0009] S5, preprocessing and image enhancement of the multimodal feature image;

[0010] S6. Segment the preprocessed multimodal feature image according to the functional areas of the visual pathway using a segmentation model, and construct a three-dimensional model of the visual pathway based on the segmentation result.

[0011] As an optional embodiment of the present invention, optionally, in step S6, segmenting the preprocessed multimodal feature image according to the functional areas of the visual pathway using the segmentation model and constructing the three-dimensional model of the visual pathway includes:

[0012] S601, obtaining a medical image with annotated visual pathway functional areas, and inputting the medical image into an initial model, training the initial model, and obtaining a segmentation model;

[0013] S602, using the segmentation model to identify the visual pathway functional area in the preprocessed multimodal feature image, and perform segmentation;

[0014] S603, converting the two-dimensional image data into a three-dimensional model using a three-dimensional reconstruction method based on the segmentation result;

[0015] S604: Optimize the three-dimensional model to obtain a three-dimensional model of the visual pathway.

[0016] As an optional embodiment of the present invention, optionally, the alignment operation in step S2 includes:

[0017] S201, performing preliminary spatial position correction on the diffusion tensor image, optical coherence tomography image, functional magnetic resonance image and structural magnetic resonance image;

[0018] S202, using a nonlinear matching algorithm, and randomly selecting one modality image as a reference image, and performing nonlinear deformation on the remaining modality images, so that the modality images are spatially aligned;

[0019] S203, using an optimization algorithm to adjust nonlinear deformation parameters so that the alignment accuracy of each modality image reaches a preset threshold, and finally generating aligned image data.

[0020] As an optional embodiment of the present invention, optionally, in step S3, the feature extraction of the diffusion tensor into an image includes:

[0021] S301, using a fiber tracking algorithm to obtain brain white matter fiber bundles in the diffusion tensor image;

[0022] S302: Calculate partial anisotropy parameters and average diffusion rate parameters based on the diffusion tensor image.

[0023] As an optional embodiment of the present invention, optionally, in step S3, the feature extraction of the optical coherence tomography image includes:

[0024] S303, identifying and segmenting each layer structure of the retina using an image segmentation algorithm based on the optical coherence tomography image;

[0025] S304, measuring the thickness of each layer of the retina structure.

[0026] As an optional embodiment of the present invention, optionally, the feature extraction of the functional magnetic resonance imaging in step S3 includes:

[0027] S305, identifying functionally activated brain regions based on the functional magnetic resonance imaging;

[0028] S306, obtaining the connection relationship between the functionally activated brain regions.

[0029] As an optional embodiment of the present invention, optionally, the feature extraction of the structural magnetic resonance imaging in step S3 includes:

[0030] S307, measuring the volume data of each brain region, the thickness of the cerebral cortex, and the surface area of ​​the cerebral cortex based on the structural magnetic resonance imaging;

[0031] S308. Extracting morphological features of the brain based on the structural magnetic resonance imaging.

[0032] As an optional embodiment of the present invention, optionally, fusing the key features of different types in step S4 includes:

[0033] S401, performing splicing operations on key features of different types respectively to obtain a multimodal feature set;

[0034] S402: Optimize the multimodal feature set using principal component analysis to obtain a multimodal feature image.

[0035] As an optional embodiment of the present invention, optionally, preprocessing the multimodal feature image in step S5 includes:

[0036] S501, removing noise from the multimodal feature image;

[0037] S502, performing a re-alignment operation on the multimodal feature image;

[0038] S503, standardizing the multimodal feature image;

[0039] S504: Use an interpolation method to improve the resolution of the multimodal feature image.

[0040] The beneficial effect of the present invention is that the present invention can reveal the complex structure and function of the visual pathway from different angles and levels by integrating multiple imaging technologies such as diffusion tensor images, optical coherence tomography images, functional magnetic resonance images and structural magnetic resonance images. Diffusion tensor imaging technology can reveal the direction and integrity of white matter fibers in the brain, optical coherence tomography technology can provide high-resolution images of retinal structure, functional magnetic resonance imaging technology can detect functional activity areas in the brain, and structural magnetic resonance imaging technology can provide detailed information on brain structure. Therefore, the multimodal fusion visual pathway imaging method of the present invention improves the accuracy and comprehensiveness of visual pathway imaging.

[0041] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0043] Figure 1 It is a flow chart of a visual pathway imaging method based on multimodal fusion of the present invention. DETAILED DESCRIPTION

[0044] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0045] Example 1

[0046] like Figure 1 As shown, a visual pathway imaging method based on multimodal fusion, the method comprising:

[0047] S1. Acquisition of diffusion tensor images, optical coherence tomography images, functional magnetic resonance images and structural magnetic resonance images;

[0048] It should be noted that the diffusion tensor image in step S1 is obtained by diffusion tensor imaging technology, which infers the microstructure of the tissue by measuring the diffusion of water molecules in the tissue. Diffusion tensor imaging can provide detailed information about the direction and integrity of white matter fibers in the brain. Optical coherence tomography images are obtained by optical coherence tomography technology, which uses the coherence of light to perform high-resolution imaging of biological tissues such as the retina. Functional magnetic resonance images are obtained by functional magnetic resonance imaging technology, which can detect changes in blood oxygen levels in functionally active areas of the brain, thereby revealing brain functional activity. Structural magnetic resonance images are obtained by structural magnetic resonance imaging technology, which can provide detailed information about brain structure, including volume data of various brain regions, thickness and surface area of ​​the cerebral cortex, etc.

[0049] S2. spatially aligning the diffusion tensor image, the optical coherence tomography image, the functional magnetic resonance image, and the structural magnetic resonance image using a nonlinear matching algorithm to obtain aligned image data;

[0050] It should be noted that the alignment operation in step S2 is crucial because different imaging modalities have different spatial resolutions and imaging directions. Through accurate spatial alignment, the correspondence between the images of each modality in subsequent analysis can be ensured to be accurate, thereby improving the reliability of multimodal fusion imaging.

[0051] S3, respectively extracting key features from the diffusion tensor image, the optical coherence tomography image, the functional magnetic resonance image, and the structural magnetic resonance image;

[0052] It should be noted that extracting key features in step S3 is one of the core steps of the multimodal fusion imaging method. By analyzing the key features provided by different imaging techniques, the structure and function of the visual pathway can be more comprehensively understood. For example, diffusion tensor imaging technology can reveal the direction and integrity of white matter fibers in the brain, which is crucial for understanding the neural conduction pathways in the visual pathway. Optical coherence tomography technology can provide high-resolution images of retinal structure, which is of great significance for studying the peripheral part of the visual pathway. Functional magnetic resonance imaging technology can detect functional activity areas in the brain, which is crucial for understanding the functional localization and network connection of the visual pathway in the brain. Structural magnetic resonance imaging technology can provide detailed information on brain structure, which is crucial for understanding the anatomical location and morphological characteristics of the visual pathway in the brain.

[0053] S4, fusing the key features of different types based on the aligned image data to obtain a multimodal feature image;

[0054] It should be noted that fusing different types of features in step S4 is a key step to realize the advantages of multimodal imaging technology. By combining the white matter fiber orientation revealed by diffusion tensor imaging technology, the high-resolution retinal images provided by optical coherence tomography technology, the functional activity areas in the brain detected by functional magnetic resonance imaging technology, and the detailed information of brain structure provided by structural magnetic resonance imaging technology, a more comprehensive and accurate visual pathway imaging model can be constructed. This fusion can not only provide richer visual pathway structural information, but also help reveal the dynamic changes of the visual pathway under different functional states.

[0055] S5, preprocessing and image enhancement of the multimodal feature image;

[0056] It should be noted that preprocessing and image enhancement in step S5 are important links to ensure the final imaging quality. The preprocessing step includes removing noise from the image, which helps to reduce interference factors in the image and improve the accuracy of subsequent analysis. The re-alignment operation ensures the consistency of the image at different processing stages and avoids image dislocation due to improper operation. Standardization processing makes images under different imaging modalities comparable, which facilitates subsequent analysis and comparison. Interpolation is used to improve image resolution, making image details clearer and helping to more accurately analyze the structure and function of the visual pathway. Image enhancement mainly includes contrast enhancement and edge enhancement, which can highlight important features in the image, making the details of the visual pathway more obvious, and facilitating doctors and researchers to diagnose and study.

[0057] S6. Segment the preprocessed multimodal feature image according to the functional areas of the visual pathway using a segmentation model, and construct a three-dimensional model of the visual pathway based on the segmentation result.

[0058] It should be noted that in step S6, the use of the segmentation model is a key step in converting a complex multimodal feature image into an intuitive three-dimensional structure. Through accurate segmentation, different functional areas of the visual pathway can be clearly divided, thereby providing a more intuitive and accurate visual pathway model for clinical diagnosis and scientific research. The segmentation model can be based on a variety of algorithms, such as threshold-based segmentation, region-based segmentation, edge-based segmentation, and machine learning-based segmentation. These algorithms each have their own advantages and can be selected and optimized according to actual needs and image characteristics.

[0059] When constructing a three-dimensional model of the visual pathway, this embodiment uses a variety of technical means, such as volume rendering, surface rendering, etc., to achieve three-dimensional visualization of the structure of the visual pathway. Volume rendering technology can display the information in the entire three-dimensional data set, while surface rendering technology focuses on displaying the details of the surface of the data set. Through these technologies, the segmented multimodal feature image can be converted into an intuitive three-dimensional model, thereby more intuitively displaying the structure and function of the visual pathway.

[0060] In summary, this embodiment can reveal the complex structure and function of the visual pathway from different angles and levels by integrating multiple imaging technologies such as diffusion tensor images, optical coherence tomography images, functional magnetic resonance images, and structural magnetic resonance images. Diffusion tensor imaging technology can reveal the direction and integrity of white matter fibers in the brain, optical coherence tomography technology can provide high-resolution images of retinal structure, functional magnetic resonance imaging technology can detect functional activity areas in the brain, and structural magnetic resonance imaging technology can provide detailed information on brain structure. Therefore, the multimodal fusion visual pathway imaging method of this embodiment improves the accuracy and comprehensiveness of visual pathway imaging.

[0061] As an optional embodiment of the present invention, optionally, in step S6, segmenting the preprocessed multimodal feature image according to the functional areas of the visual pathway using the segmentation model and constructing the three-dimensional model of the visual pathway includes:

[0062] S601, obtaining a medical image with annotated visual pathway functional areas, and inputting the medical image into an initial model, training the initial model, and obtaining a segmentation model;

[0063] It should be noted that in step S601, the acquisition of medical images is a crucial step, because these images provide accurate annotation information of the functional areas of the visual pathway. These annotation information are the basis for training the segmentation model, ensuring that the model can accurately identify and segment the different functional areas in the visual pathway. By using the annotated images for training, the segmentation model can learn the characteristics of different areas and accurately apply this knowledge in the subsequent segmentation process, thereby achieving accurate division of the functional areas of the visual pathway. The initial model is a pre-set neural network model, which includes basic structure and parameter settings for processing medical image data. During the training process, the initial model will be adjusted and optimized according to the input annotated medical image to improve its segmentation performance, and the attention mechanism is also integrated into the segmentation model. After the training is completed, the segmentation model can use the attention mechanism to accurately segment the new multimodal feature image, thereby constructing a three-dimensional model of the visual pathway.

[0064] S602, using the segmentation model to identify the visual pathway functional area in the preprocessed multimodal feature image, and perform segmentation;

[0065] It should be noted that in step S602, the segmentation model introduces the attention mechanism to enhance the model's ability to capture key features. The attention mechanism allows the model to automatically focus on the most critical parts for identifying the functional areas of the visual pathway when processing the image, thereby improving the accuracy and efficiency of the segmentation. This mechanism not only helps the model better understand the contextual information in the image, but also makes the segmentation results more consistent with the actual structure of the visual pathway.

[0066] S603, converting the two-dimensional image data into a three-dimensional model using a three-dimensional reconstruction method based on the segmentation result;

[0067] It should be noted that the three-dimensional reconstruction method in step S603 is a key step in converting the two-dimensional image data into a three-dimensional model. Through this process, the two-dimensional image data of the visual pathway functional area obtained by segmentation can be converted into a specific structure in a three-dimensional space, thereby providing a more intuitive and three-dimensional visual pathway model. When performing three-dimensional reconstruction, a variety of algorithms can be used, such as a voxel-based method, a surface-based method, and an image registration-based method. These methods have their own characteristics and can be selected according to the actual image data and the required reconstruction accuracy.

[0068] S604: Optimize the three-dimensional model to obtain a three-dimensional model of the visual pathway.

[0069] It should be noted that in step S604, the optimization process is an important step in ensuring the quality of the three-dimensional model. Through optimization, the noise and unnecessary details generated during the three-dimensional reconstruction process can be removed to make the model smoother and more accurate. Optimization methods include but are not limited to filtering, surface smoothing, and topology optimization. Filtering can reduce noise in the image and improve the clarity of the model; surface smoothing can eliminate jagged edges generated during the reconstruction process and make the model look more natural; topology optimization focuses on the structural integrity of the model to ensure that the three-dimensional model has good visual effects and analytical value while maintaining structural accuracy.

[0070] After completing the above steps, the final three-dimensional model of the visual pathway can not only provide doctors and researchers with an intuitive visual pathway structure, but can also be used for further quantitative analysis and simulation. For example, the model can be used for hemodynamic simulation to evaluate the blood supply to different visual pathway regions; or for biomechanical analysis to study the response of the visual pathway under different external forces. In addition, the three-dimensional model can also be used as a teaching tool to help medical students and related professionals better understand the complex structure and function of the visual pathway.

[0071] As an optional embodiment of the present invention, optionally, the alignment operation in step S2 includes:

[0072] S201, performing preliminary spatial position correction on the diffusion tensor image, optical coherence tomography image, functional magnetic resonance image and structural magnetic resonance image;

[0073] It should be noted that the preliminary spatial position correction in step S201 is mainly to ensure that the images obtained by different imaging technologies can be aligned in space, so as to facilitate the subsequent multimodal fusion processing. This process usually includes operations such as rotating, scaling and translating the image to eliminate the spatial deviation between images caused by imaging equipment, patient position changes or scanning parameter differences. The accuracy of spatial position correction directly affects the quality of multimodal image fusion, and then affects the accuracy of the final three-dimensional model of the visual pathway. When performing preliminary spatial position correction, a variety of methods can be used, such as registration based on markers, registration based on image features, and registration based on mutual information. The registration method based on markers introduces markers of known positions during the imaging process, and then realizes the alignment between different images by identifying these markers in the image processing stage. The registration method based on image features uses natural features in the image, such as edges, corners, etc., to find the corresponding relationship between different images. The registration method based on mutual information evaluates the similarity between different images by calculating the mutual information between them, and aligns them based on this.

[0074] S202, using a nonlinear matching algorithm, and randomly selecting one modality image as a reference image, and performing nonlinear deformation on the remaining modality images, so that the modality images are spatially aligned;

[0075] It should be noted that in step S202, the use of nonlinear matching algorithms is to further improve the accuracy of multimodal image alignment. Since different imaging technologies may have different spatial deformation characteristics, simple linear transformations often cannot completely eliminate the spatial deviations between images. Nonlinear matching algorithms can more accurately describe and correct these deformations by introducing more complex transformation models, thereby achieving more accurate image alignment. In actual operation, a variety of nonlinear transformation methods can be used, such as registration based on elastic deformation, registration based on manifold learning, and registration based on deep learning. The registration method based on elastic deformation adjusts the image by simulating the deformation behavior of a physical elastic body, while the registration method based on manifold learning uses the intrinsic geometric structure of the image data for registration. The registration method based on deep learning learns the nonlinear transformation relationship between images by training a neural network model.

[0076] S203, using an optimization algorithm to adjust nonlinear deformation parameters so that the alignment accuracy of each modality image reaches a preset threshold, and finally generating aligned image data.

[0077] It should be noted that the use of the optimization algorithm in step S203 is to ensure that the accuracy of multimodal image alignment reaches the optimal state. The optimization algorithm iteratively adjusts the nonlinear deformation parameters to minimize the differences between images, thereby achieving a preset alignment accuracy threshold. In this process, a variety of optimization strategies can be used, such as gradient descent, genetic algorithm, and simulated annealing algorithm. The gradient descent method guides the adjustment direction of the parameters by calculating the gradient of the loss function, while the genetic algorithm simulates the process of natural selection and optimizes the parameters through crossover, mutation, and selection operations. The simulated annealing algorithm draws on the physical annealing process and reduces the energy of the system by gradually lowering the "temperature" to find the global optimal solution.

[0078] As an optional embodiment of the present invention, optionally, in step S3, the feature extraction of the diffusion tensor into an image includes:

[0079] S301, using a fiber tracking algorithm to obtain brain white matter fiber bundles in the diffusion tensor image;

[0080] It should be noted that in step S301, the fiber tracking algorithm is a technology used to analyze diffusion tensor imaging (DTI) data, which can reconstruct the three-dimensional path of white matter fibers in the brain. By tracking the diffusion tensor in various parts of the brain, the fiber tracking algorithm can reveal the direction and connection pattern of white matter fiber bundles. This process is of great significance for understanding brain structure and function, especially when studying visual pathways, it can better understand how visual information is transmitted and processed in the brain.

[0081] S302: Calculate partial anisotropy parameters and average diffusion rate parameters based on the diffusion tensor image.

[0082] It should be noted that in step S302, the anisotropy parameter reflects the arrangement direction and structural integrity of the white matter fiber bundles, while the mean diffusivity parameter provides information on the average speed of water molecules diffusing in the brain. These parameters are crucial for evaluating the health and functional status of white matter. For example, a decrease in the anisotropy value may indicate structural damage to the white matter fiber bundles, while an increase in the mean diffusivity may indicate abnormal diffusion of water molecules in brain tissue, which may be associated with certain neurological diseases. By analyzing these parameters, doctors and researchers can gain a deeper understanding of the microstructural changes in the visual pathway, providing an important basis for early diagnosis and treatment of diseases.

[0083] As an optional embodiment of the present invention, optionally, in step S3, the feature extraction of the optical coherence tomography image includes:

[0084] S303, identifying and segmenting each layer structure of the retina using an image segmentation algorithm based on the optical coherence tomography image;

[0085] It should be noted that the image segmentation algorithm in step S303 is a technique for separating the region of interest from the image. In retinal imaging, the image segmentation algorithm can accurately identify the different layers of the retina, including the nerve fiber layer, inner nuclear layer, outer nuclear layer, etc. By accurately segmenting these layers, the health of the retina can be evaluated, which is of great significance for diagnosing retinal-related diseases such as diabetic retinopathy and macular degeneration. In addition, the accurate segmentation of the retinal layers also provides important anatomical information for studying the structure and function of the visual pathway.

[0086] S304, measuring the thickness of each layer of the retina structure.

[0087] It should be noted that measuring the thickness of each layer of the retina structure in step S304 is a key step in assessing the health of the retina. Changes in retinal thickness can reflect the development of a variety of eye diseases, such as diabetic retinopathy and macular degeneration. By accurately measuring the thickness of each layer of the retina, doctors can detect abnormalities earlier and take timely treatment measures. In addition, the measurement results of retinal thickness also provide important quantitative data for studying the structure and function of the visual pathway, which helps to deeply understand the processing mechanism of visual information at the retinal level. In order to achieve accurate retinal thickness measurement, a variety of image processing techniques can be used. For example, an edge detection algorithm can be used to identify the boundaries between the layers of the retina, and then the thickness of each layer is obtained by calculating the distance between these boundaries. In addition, machine learning methods can also be used to identify the characteristics of each layer of the retina by training models, thereby realizing automated thickness measurement. The application of these technologies not only improves the accuracy of measurement, but also greatly reduces the workload of doctors, making the measurement of retinal thickness more efficient and reliable.

[0088] As an optional embodiment of the present invention, optionally, the feature extraction of the functional magnetic resonance imaging in step S3 includes:

[0089] S305, identifying functionally activated brain regions based on the functional magnetic resonance imaging;

[0090] It should be noted that in step S305, functional magnetic resonance imaging (fMRI) is a technology that uses blood oxygen level dependent (BOLD) signals to detect active areas in the brain. In visual pathway studies, functional magnetic resonance imaging can reveal activation patterns of various brain regions under visual stimulation. By analyzing these activation patterns, researchers can understand how visual information is processed and transmitted in the brain. In step S305, the process of identifying functionally activated brain areas usually involves statistical analysis of fMRI data to determine which brain areas show significant signal changes during visual tasks. This is usually achieved by comparing BOLD signals during visual tasks and in a resting state. The identified functionally activated brain areas provide important functional information for studying the visual pathway and help reveal the neural mechanisms of visual information processing.

[0091] S306, obtaining the connection relationship between the functionally activated brain regions.

[0092] It should be noted that, in step S306, the connection relationship between each functionally activated brain region can be obtained by using a variety of methods to analyze the connection relationship between functionally activated brain regions. A commonly used method is to calculate a functional connectivity matrix, which is based on time series data and evaluates the functional connection strength between different brain region signals by analyzing the synchronization of the signals. In addition, graph theory analysis methods can also be used to quantify the topological properties of the brain network, such as the centrality of the nodes, the clustering coefficient of the network, and the path length. These properties help to reveal the role of key brain regions in the visual pathway and the organizational structure of the entire network. In the specific implementation process, a variety of statistical analysis methods can be used to determine the significance of the functional connection relationship. For example, voxel-based statistical analysis can be used to identify pairs of brain regions that show significant functional connections in visual tasks. In addition, machine learning algorithms, such as support vector machines (SVM) or random forests, can also be applied to predict the connection patterns between different brain regions.

[0093] As an optional embodiment of the present invention, optionally, the feature extraction of the structural magnetic resonance imaging in step S3 includes:

[0094] S307, measuring the volume data of each brain region, the thickness of the cerebral cortex, and the surface area of ​​the cerebral cortex based on the structural magnetic resonance imaging;

[0095] It should be noted that in step S307, measuring the volume data of each brain region, the thickness of the cerebral cortex, and the surface area of ​​the cerebral cortex are important indicators for assessing the health of the brain structure. These parameters can not only reflect the development of the brain, but also have important significance for diagnosing neurodegenerative diseases, brain damage, etc. For example, changes in the thickness of the cerebral cortex may be related to neurodegenerative diseases such as Alzheimer's disease, while abnormal changes in the volume of various brain regions may indicate brain tumors or other structural lesions. By accurately measuring these parameters, doctors and researchers can more accurately assess the health of the brain and provide an important basis for early diagnosis and treatment of diseases.

[0096] S308. Extracting morphological features of the brain based on the structural magnetic resonance imaging.

[0097] It should be noted that, in step S308, extracting the morphological features of the brain is a key step to further understand the structure and function of the brain. Morphological features include but are not limited to the morphology of sulci, gyri, the size and shape of ventricles, etc. These features are of great significance for studying the development and aging of the brain and the relationship with various neuropsychiatric diseases. For example, by analyzing the morphology of sulci and gyri, the complexity and information processing ability of the cerebral cortex can be understood. In addition, changes in the size and shape of the ventricles may be related to cerebrospinal fluid circulation disorders, which is of guiding significance for the diagnosis and treatment of related diseases. In order to achieve accurate morphological feature extraction, a variety of image processing techniques can be used, such as three-dimensional reconstruction, morphological analysis, etc. The application of these technologies not only improves the accuracy of feature extraction, but also provides rich data support for studying brain structure and function.

[0098] As an optional embodiment of the present invention, optionally, fusing the key features of different types in step S4 includes:

[0099] S401, performing splicing operations on key features of different types respectively to obtain a multimodal feature set;

[0100] It should be noted that in step S401, stitching different types of key features is an important step in achieving multimodal data fusion. Through the stitching operation, data from different imaging technologies can be integrated into a unified feature set, thereby providing more comprehensive retinal and brain structural information. For example, retinal thickness data can be combined with functional magnetic resonance imaging (fMRI) data to obtain a comprehensive view of the function and structure of the visual pathway. This fusion not only helps to improve the accuracy of disease diagnosis, but also provides richer data support for personalized medicine.

[0101] S402: Optimize the multimodal feature set using principal component analysis to obtain a multimodal feature image.

[0102] It should be noted that the purpose of optimizing the multimodal feature set using principal component analysis (PCA) in step S402 is to reduce the dimension of the data while retaining the most important information. Through principal component analysis, multiple highly correlated variables can be converted into a few unrelated principal components, which can capture most of the variation in the original data set. In the process of optimizing the multimodal feature set, the data must first be standardized to eliminate the effects of different dimensions and magnitudes. Then, the covariance matrix of the data is calculated, and its eigenvalues ​​and eigenvectors are solved. Several eigenvectors with the largest eigenvalues ​​are selected as principal components, and the original data are projected onto these principal components to obtain an optimized multimodal feature image. This optimization method not only helps to reduce the computational complexity of subsequent processing, but also improves the generalization ability of the model.

[0103] As an optional embodiment of the present invention, optionally, preprocessing the multimodal feature image in step S5 includes:

[0104] S501, removing noise from the multimodal feature image;

[0105] S502, performing a re-alignment operation on the multimodal feature image;

[0106] S503, standardizing the multimodal feature image;

[0107] S504: Use an interpolation method to improve the resolution of the multimodal feature image.

[0108] It should be noted that the interpolation method is used to improve the resolution of the multimodal feature image in step S504 in order to ensure that more accurate image details can be obtained in the subsequent analysis and diagnosis process. Interpolation is a mathematical method that increases the pixel density of the image by estimating new data points between known data points. In the multimodal feature image, the high-resolution image can provide clearer visual information, which helps to more accurately identify and analyze the structural and functional characteristics of the brain region. For example, bilinear interpolation, cubic convolution interpolation or spline interpolation can be used to improve the resolution of the image. These methods generate new pixels by calculating the weighted average of adjacent pixel values, thereby increasing the details of the image without losing image quality. After interpolation processing, the multimodal feature image will be more suitable for further analysis, such as pattern recognition, anomaly detection and disease diagnosis. In addition, improving image resolution can also help improve the visual experience of doctors and researchers in clinical applications, thereby improving the accuracy and efficiency of diagnosis.

[0109] Example 2

[0110] A computer device comprising:

[0111] processor;

[0112] a memory for storing processor-executable instructions;

[0113] Wherein, the processor is configured to implement a visual pathway imaging method based on multimodal fusion in Example 1 when executing the executable instructions.

[0114] It should be noted that the computer device includes: a processor, a memory, and the computer device may also include one or more of a multimedia component, an input / output (I / O) interface, and a communication component.

[0115] The processor is used to control the overall operation of the computer device to complete all or part of the steps in the above-mentioned visual pathway imaging method based on multimodal fusion.

[0116] The memory is used to store various types of data to support operations on the computer device, which data may include, for example, instructions for any application or method operating on the computer device, as well as application-related data; the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0117] The multimedia component may include a screen and an audio component, wherein the screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals; for example, the audio component may include a microphone for receiving external audio signals, and the received audio signals may be further stored in a memory or sent via a communication component; the audio component also includes at least one speaker for outputting audio signals.

[0118] The I / O interface provides an interface between the processor and other interface modules, which may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons.

[0119] The communication component is used for wired or wireless communication between the computer device and other devices; wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G or 5G, or a combination of one or more of them, so the corresponding communication component may include: Wi-Fi module, Bluetooth module, NFC module, mobile phone communication module.

[0120] As a preferred solution of this embodiment, the computer device can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned visual pathway imaging method based on multimodal fusion.

[0121] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A visual pathway imaging method based on multimodal fusion, characterized in that , the method comprises: S1. Acquisition of diffusion tensor images, optical coherence tomography images, functional magnetic resonance images and structural magnetic resonance images; S2. spatially aligning the diffusion tensor image, the optical coherence tomography image, the functional magnetic resonance image, and the structural magnetic resonance image using a nonlinear matching algorithm to obtain aligned image data; S3, respectively extracting key features from the diffusion tensor image, the optical coherence tomography image, the functional magnetic resonance image, and the structural magnetic resonance image; S4, fusing the key features of different types based on the aligned image data to obtain a multimodal feature image; S5, preprocessing and image enhancement of the multimodal feature image; S6. Segment the preprocessed multimodal feature image according to the functional areas of the visual pathway using a segmentation model, and construct a three-dimensional model of the visual pathway based on the segmentation result.

2. A visual pathway imaging method based on multimodal fusion as claimed in claim 1, characterized in that: In step S6, using the segmentation model to segment the preprocessed multimodal feature image according to the functional areas of the visual pathway and constructing a three-dimensional model of the visual pathway includes: S601, obtaining a medical image with annotated functional areas of a visual pathway, and inputting the medical image into an initial model, training the initial model, and obtaining a segmentation model; S602, using the segmentation model to identify the visual pathway functional area in the preprocessed multimodal feature image, and perform segmentation; S603, converting the two-dimensional image data into a three-dimensional model using a three-dimensional reconstruction method based on the segmentation result; S604: Optimize the three-dimensional model to obtain a three-dimensional model of the visual pathway.

3. The visual pathway imaging method based on multimodal fusion according to claim 1, characterized in that: In step S2, the alignment operation includes: S201, performing preliminary spatial position correction on the diffusion tensor image, optical coherence tomography image, functional magnetic resonance image and structural magnetic resonance image; S202, using a nonlinear matching algorithm, and randomly selecting one modality image as a reference image, and performing nonlinear deformation on the remaining modality images, so that the modality images are spatially aligned; S203, using an optimization algorithm to adjust nonlinear deformation parameters so that the alignment accuracy of each modality image reaches a preset threshold, and finally generating aligned image data.

4. The visual pathway imaging method based on multimodal fusion according to claim 1, characterized in that: In step S3, the feature extraction of the diffusion tensor into an image includes: S301, using a fiber tracking algorithm to obtain brain white matter fiber bundles in the diffusion tensor image; S302: Calculate partial anisotropy parameters and average diffusion rate parameters based on the diffusion tensor image.

5. The visual pathway imaging method based on multimodal fusion according to claim 1, characterized in that: In step S3, the feature extraction of the optical coherence tomography image includes: S303, identifying and segmenting each layer structure of the retina using an image segmentation algorithm based on the optical coherence tomography image; S304, measuring the thickness of each layer of the retina structure.

6. The visual pathway imaging method based on multimodal fusion according to claim 1, characterized in that: In step S3, the feature extraction of the functional magnetic resonance imaging includes: S305, identifying functionally activated brain regions based on the functional magnetic resonance imaging; S306, obtaining the connection relationship between the functionally activated brain regions.

7. The visual pathway imaging method based on multimodal fusion according to claim 1, characterized in that: In step S3, the feature extraction of the structural magnetic resonance imaging includes: S307, measuring the volume data of each brain region, the thickness of the cerebral cortex, and the surface area of ​​the cerebral cortex based on the structural magnetic resonance imaging; S308. Extracting morphological features of the brain based on the structural magnetic resonance imaging.

8. The visual pathway imaging method based on multimodal fusion according to claim 1, characterized in that: In step S4, fusing the key features of different types includes: S401, performing splicing operations on key features of different types respectively to obtain a multimodal feature set; S402: Optimize the multimodal feature set using principal component analysis to obtain a multimodal feature image.

9. The visual pathway imaging method based on multimodal fusion according to claim 1, characterized in that: Preprocessing the multimodal feature image in step S5 includes: S501, removing noise from the multimodal feature image; S502, performing a re-alignment operation on the multimodal feature image; S503, standardizing the multimodal feature image; S504: Use an interpolation method to improve the resolution of the multimodal feature image.

10. A computer device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement a visual pathway imaging method based on multimodal fusion as described in any one of claims 1 to 9 when executing the executable instructions.

Images