Intelligent storing and filing system and method based on multi-mode brain data fusion

Abstract

The invention belongs to the technical field of multi-modal brain data, and particularly relates to an intelligent storage and archiving system and method based on multi-modal brain data fusion, and the system comprises a data collection module which is used for carrying out the butt joint with various devices, and achieves the automatic collection and transmission of data; through a standardized interface protocol, it can be ensured that data of different devices can be smoothly accessed to the system, and metadata information of data acquisition is recorded at the same time; and the data preprocessing module is used for carrying out format conversion, noise removal and data calibration on the data. Through the data acquisition module, the data acquisition problem caused by equipment interface difference is solved, it is ensured that data of different equipment can be rapidly and stably accessed to the system, and the efficiency and accuracy of data acquisition are improved. And meanwhile, metadata information such as data acquisition time, equipment parameters and the like is recorded, so that rich background information is provided for subsequent data management, tracing and analysis, and the availability and the understandability of the data are enhanced.

Description

Technical Field

[0001] This invention relates to the field of multimodal brain data technology, specifically to an intelligent storage and archiving system and method based on multimodal brain data fusion. Background Technology

[0002] With the rapid development of medical technology, especially in the field of neuroscience, the acquisition of multimodal brain data has become increasingly common and convenient. Magnetic resonance imaging (MRI) can clearly show the structural information of the brain, functional magnetic resonance imaging (fMRI) can reflect hemodynamic changes during brain neural activity, near-infrared functional brain imaging (fNIRS) can monitor the oxygenation status of brain tissue in real time, and electroencephalography (EEG) records the electrical activity of the brain, etc. These different modalities of data provide rich information from multiple perspectives for studying the structure and function of the brain, diagnosing brain diseases, and evaluating treatment effects.

[0003] In clinical practice, multimodal brain data fusion helps doctors gain a more comprehensive and accurate understanding of a patient's brain condition, improving the accuracy of disease diagnosis and the effectiveness of treatment plans. For example, in diagnosing brain tumors, combining structural information from MRI with functional information from fMRI can more clearly define the tumor's boundaries and assess its impact on surrounding brain tissue function, thus providing a more reliable basis for surgical planning. In the scientific research field, multimodal data fusion provides a powerful tool for in-depth exploration of the brain's mysteries, such as cognitive neuroscience studying cognitive processes and neurodevelopmental science studying brain developmental mechanisms, helping to reveal the complex mechanisms of the brain in both healthy and diseased states.

[0004] However, the current storage and archiving of multimodal brain data has the following problems: 1. Diverse data formats and compatibility challenges: 1.1 Different brain data acquisition devices and technologies produce data in vastly different formats. MRI data is typically stored in DICOM format, containing rich image information but with a complex file structure; FINR data may use custom formats, focusing on recording time-series physiological parameters; EEG data is saved in a specific binary format, emphasizing the precise recording of electrical signals. This necessitates significant time and effort to parse and convert these different data formats to ensure compatibility when performing multimodal data fusion. For example, when researchers attempt to fuse MRI structural data with EEG electrophysiological data, the first challenge is addressing data format incompatibility. Improper format conversion may result in the loss of crucial information, affecting the accuracy of subsequent analysis results.

[0005] 2. Data storage and management challenges: 2.1 High Storage Capacity Requirements: Multimodal brain data is extremely massive. High-resolution MRI images can reach hundreds of MB or even GB per sequence, and long-term monitoring FNIRS data and continuously recorded EEG data also rapidly accumulate. With the increasing clinical applications and research projects, data storage demands are growing exponentially, placing extremely high demands on storage capacity. Many medical institutions and research units' existing storage systems are insufficient to meet such large-scale data storage needs, facing the dilemma of frequently upgrading storage devices or migrating data.

[0006] 2.2 Complex Data Management: The diverse sources, collection times, and purposes of multimodal data make data management exceptionally complex. How to effectively classify, index, and retrieve this data to ensure quick and accurate access when needed is a pressing issue. For example, in a database containing a large amount of multimodal brain data from patients, if a doctor wants to find combined MRI and FNIRS data for a specific patient within a specific time period, without a proper data management system, they may need to search through massive amounts of data file one by one, resulting in extremely low efficiency.

[0007] 3. Bottlenecks in data fusion technology: 3.1 The accuracy and efficiency of fusion algorithms need improvement: Existing multimodal data fusion algorithms struggle to balance accuracy and efficiency when processing brain data. Some algorithms, in pursuit of high accuracy, involve complex computational processes, consuming significant computational resources and time, failing to meet the needs of real-time clinical diagnosis or large-scale data analysis. Conversely, some efficiency-oriented algorithms often compromise on fusion accuracy, failing to fully exploit the complementary information between different modalities. For example, in machine learning-based fusion algorithms, some require multiple iterative calculations on large amounts of data during model training, leading to excessive computation time and delaying optimal disease diagnosis.

[0008] 3.2 Insufficient Mining of Complex Relationships: Complex nonlinear relationships exist among multimodal brain data; however, current fusion techniques often fail to fully uncover these relationships. The brain information reflected by different modalities is interconnected on both spatial and temporal scales. For example, there is a correspondence between functional activation areas on fMRI and anatomical structures on MRI; changes in oxygenation in fNIRS and changes in electrical activity on EEG may have a sequential or synchronous relationship in time. However, most existing fusion methods simply splice the data or perform shallow feature fusion, failing to reveal these complex relationships and limiting a deeper understanding of brain function and disease mechanisms.

[0009] Based on the above, an intelligent storage and archiving system and method based on multimodal brain data fusion is invented. Summary of the Invention

[0010] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution: An intelligent storage and archiving system based on multimodal brain data fusion, comprising: The data acquisition module is used to interface with various devices to achieve automatic data acquisition and transmission; it can also ensure that data from different devices can be successfully accessed into the system through standardized interface protocols, while recording metadata information of the acquired data. The data preprocessing module is used to perform data format conversion, noise removal, and data calibration; The data quality assessment module is used to assess the quality of preprocessed data and generate an assessment report. The intelligent semantic annotation module is used to semantically annotate data, so as to give the data a semantic understanding; The data fusion module is used to first extract features, then perform data fusion, and finally mine the complex nonlinear relationships between different modalities of data. The data storage module is used to distribute data across multiple storage nodes using a distributed storage system to improve storage capacity and data read / write performance. Simultaneously, it utilizes data compression technology to compress image data, reducing storage space usage. Furthermore, it establishes a data backup and recovery mechanism to regularly back up important data, ensuring data security and integrity. The data management and application module provides functions such as data indexing and retrieval, data analysis and visualization, access control and data prediction.

[0011] As a preferred embodiment of the intelligent storage and archiving system based on multimodal brain data fusion described in this invention, the specific steps of the data preprocessing module are as follows: Step 1, Format Conversion: Based on the built-in multi-data format parser, brain data in different formats is converted into a unified intermediate format; Step 2, noise removal: Based on the noise characteristics of different modal data, corresponding filtering algorithms are used for noise removal; Step 3, Data Calibration: Perform time synchronization calibration on the data to ensure consistency of different modal data in the time dimension.

[0012] As a preferred embodiment of the intelligent storage and archiving system based on multimodal brain data fusion described in this invention, the specific steps of the data quality assessment module are as follows: Step 1, Quality Indicator Setting: Set multiple quality assessment indicators for different modalities of brain data; Step 2, Automated Evaluation: Utilize algorithms to automatically perform quality checks on the data; Step 3, Report Generation: Generate a detailed data quality assessment report to visually display the assessment results of each indicator and mark the data parts with quality problems and the types of problems.

[0013] As a preferred embodiment of the intelligent storage and archiving system based on multimodal brain data fusion described in this invention, the specific steps of the intelligent semantic annotation module are as follows: Step 1, Data Reception and Preliminary Processing: First, the multimodal brain data that has passed the data quality assessment module is transmitted to the intelligent semantic annotation module; then, preliminary format adaptation and content parsing are performed according to the characteristics of different modalities of data. Step 2, Semantic Extraction: Semantic extraction is performed on image data, time series data, and text data; Step 3, Knowledge Graph Association: The extracted semantic information is associated with the pre-constructed brain medical knowledge graph; first, the labeled brain structure semantics are connected with the anatomical structure nodes in the knowledge graph through a semantic matching algorithm, and then the disease-related semantics are linked with the disease diagnosis nodes. Step 4, Review and Correction of Annotation Results: Review the results of automatic annotation to check the accuracy, completeness and rationality of semantic annotation, and correct any errors or ambiguities in the annotations. Step 5, Dynamically update annotations: Establish a dynamic update mechanism for annotation rules and knowledge graphs to track the latest research results in the medical field, updates to clinical guidelines, and the release of new diagnostic criteria. When new knowledge related to brain data annotations emerges, the annotation rules and knowledge graphs will be updated in a timely manner. Step Six, Annotated Data Output: The reviewed and updated semantically annotated data is integrated with the original multimodal brain data, and a data file with semantically annotated information is output. The data file is stored in a standardized format for easy subsequent data retrieval. At the same time, an index of annotated data is established to facilitate quick retrieval and query of data with specific semantic annotations, providing more valuable information support for clinical diagnosis and scientific research analysis.

[0014] As a preferred embodiment of the intelligent storage and archiving system based on multimodal brain data fusion described in this invention, the semantic extraction process of the image data is as follows: using computer vision technology, feature recognition is performed on MRI and fMRI images; firstly, the anatomical structure of the brain is automatically identified through a convolutional neural network model, and corresponding information such as location and size is labeled; then, for lesion areas, features are first identified and converted into semantic descriptions. The semantic extraction process for the time series data is as follows: For EEG and fnirs time series data, signal processing and machine learning algorithms are used to extract semantics; firstly, through spectrum analysis, the energy distribution characteristics of different frequency bands are extracted from the EEG data and converted into semantics; then, the changing trends of oxyhemoglobin and deoxyhemoglobin are analyzed from the fnirs data and labeled. The semantic extraction process of the text data is as follows: natural language processing technology is used to extract semantics from the text report; first, the key entities in the text are extracted through named entity recognition algorithm; then, dependency parsing is used to understand the semantic relationships between words in the text, and further refine the complete semantic information.

[0015] As a preferred embodiment of the intelligent storage and archiving system based on multimodal brain data fusion described in this invention, the specific steps of the data fusion module are as follows: Step 1, Feature Extraction: First, use a deep learning model to extract spatial features from MRI and fMRI image data, and then use a recurrent neural network to extract temporal features from EEG and fNIRS time series data. Step 2, Fusion Algorithm: A multimodal fusion algorithm based on an attention mechanism is adopted to assign weights according to the importance of different modal data to the target task, thereby achieving deep data fusion; Step 3, Relationship Mining: Introducing graph neural networks to mine complex nonlinear relationships between different modalities of data in order to construct a data relationship graph, providing support for in-depth analysis of brain function and disease mechanisms.

[0016] As a preferred embodiment of the intelligent storage and archiving system based on multimodal brain data fusion described in this invention, the data management and application module includes: The data indexing and retrieval module is used to build multi-dimensional data indexes to support fast retrieval based on conditions, and can use full-text search technology to achieve fuzzy search and precise matching. The data analysis and visualization module provides a variety of data analysis tools to help users conduct in-depth data analysis. At the same time, it can use data visualization technology to intuitively display the integrated data in various forms, making it easier for doctors and researchers to understand the data characteristics.

[0017] As a preferred embodiment of the intelligent storage and archiving system based on multimodal brain data fusion described in this invention, the data management and application module further includes: The access control module is used to assign different operation permissions to different users using a role-permission model, ensuring data security and privacy. The data prediction module is used to first train a disease prediction model based on historical multimodal brain data and patient medical records; then, it combines the patient's current multimodal data, prediction results, and clinical guidelines to recommend personalized treatment plans for doctors; finally, the prediction results and recommended plans are presented in a visual form to intuitively show the possible results of different decision-making paths, assisting doctors and patients in making joint decisions.

[0018] A smart storage and archiving method based on multimodal brain data fusion includes the following specific steps: S1: It can interface with various devices to achieve automatic data collection and transmission; and through standardized interface protocols, it can ensure that data from different devices can be successfully accessed into the system, while recording metadata information of the collected data. S2: First, based on the built-in multi-data format parser, brain data of different formats are converted into a unified intermediate format; then, according to the noise characteristics of different modal data, corresponding filtering algorithms are used to remove noise; then, the data is time-synchronized and calibrated to ensure the consistency of different modal data in the time dimension. S3: First, set multiple quality assessment indicators for brain data of different modalities; then use algorithms to automatically detect the quality of the data; then generate a detailed data quality assessment report to intuitively display the assessment results of each indicator and mark the data parts with quality problems and the types of problems. S4: First, the multimodal brain data, after passing the data quality assessment module, is transmitted to the intelligent semantic annotation module. Then, preliminary format adaptation and content parsing are performed based on the characteristics of different modalities. After parsing, semantic extraction is performed on image data, time-series data, and text data. Following extraction, the extracted semantic information is associated with a pre-constructed brain medical knowledge graph. A semantic matching algorithm connects the annotated brain structure semantics with anatomical structure nodes in the knowledge graph, and then links disease-related semantics with disease diagnosis nodes. After association, the automatic annotation results are reviewed to check the accuracy, completeness, and rationality of the semantic annotations. Any issues are addressed. Errors or unclear annotations are corrected. After review, a dynamic update mechanism for annotation rules and knowledge graphs is established to track the latest research findings in the medical field, updates to clinical guidelines, and the release of new diagnostic criteria. When new knowledge related to brain data annotation emerges, the annotation rules and knowledge graphs are updated promptly. Subsequently, the reviewed and updated semantically annotated data is integrated with the original multimodal brain data, and data files with semantic annotation information are output. The data files are stored in a standardized format for easy subsequent data retrieval. At the same time, an index of annotated data is established to facilitate quick retrieval and querying of data with specific semantic annotations, providing more valuable information support for clinical diagnosis and scientific research analysis. S5: First, a deep learning model is used to extract spatial features from MRI and fMRI image data. Then, a recurrent neural network is used to extract temporal features from EEG and fNIRS time series data. After extraction, a multimodal fusion algorithm based on an attention mechanism is used to assign weights according to the importance of different modal data to the target task, thereby achieving deep data fusion. After fusion, a graph neural network is introduced to mine the complex nonlinear relationships between different modal data to construct a data relationship map, providing support for in-depth analysis of brain function and disease mechanisms. S6: A distributed storage system is used to distribute data across multiple storage nodes to improve storage capacity and data read / write performance; at the same time, data compression technology is used to compress image data to reduce storage space usage; and a data backup and recovery mechanism is established to regularly back up important data to ensure data security and integrity.

[0019] Compared with existing technologies: 1. The data acquisition module effectively solves data acquisition challenges caused by differences in device interfaces, ensuring that data from different devices can be quickly and stably accessed into the system, thus improving the efficiency and accuracy of data acquisition. Simultaneously, it records metadata information such as the data acquisition time and device parameters, providing rich background information for subsequent data management, traceability, and analysis, enhancing the usability and understandability of the data.

[0020] 2. The data preprocessing module not only effectively solves the problem of data format incompatibility and reduces the risk of information loss during format conversion, laying the foundation for the fusion analysis of multimodal data; it also accurately cleans the data, improves data quality, avoids noise interference with subsequent analysis results, and makes the data more realistically reflect brain physiological and pathological information; in addition, it can solve the problem of time asynchrony of multimodal data, ensuring the spatiotemporal accuracy of data fusion and improving the overall usability of the data.

[0021] 3. Through the data fusion module, not only can the core features of the data be captured efficiently and accurately, but the features extracted by traditional methods are also more representative, providing high-quality basic data for data fusion; it can also Ensuring fusion accuracy and improving computational efficiency enables rapid and accurate data fusion in real-time clinical diagnosis and large-scale data analysis scenarios, uncovering complementary information between data. Furthermore, it can reveal the complex spatial and temporal relationships between multimodal brain data, providing a more in-depth analytical perspective for studying brain function and disease mechanisms.

[0022] 4. By distributing data across multiple storage nodes through a data storage module, storage capacity is significantly expanded, meeting the ever-growing storage demands of multimodal brain data and avoiding frequent storage device upgrades. Data compression technology reduces storage space usage and lowers storage costs. Simultaneously, a multi-dimensional data index and efficient retrieval mechanism are established, supporting rapid data retrieval based on various criteria. This changes the previous inefficient search practices in massive datasets, improving data management efficiency and data access convenience.

[0023] 5. Through the data management and application module, not only can fuzzy queries and precise matching be achieved, enabling doctors and researchers to quickly locate the data they need, greatly improving data retrieval efficiency and meeting the needs of efficient data use in clinical and scientific research; it also facilitates users' understanding of data characteristics, assists in in-depth data analysis and decision-making, and enhances the application value of data; in addition, it can strictly control data access, prevent data from being illegally obtained and disseminated, protect sensitive patient brain data from the access control level, and ensure data security and privacy. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall framework of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0026] This invention provides an intelligent storage and archiving system based on multimodal brain data fusion. Please refer to [link / reference]. Figure 1 ,include: The data acquisition module is used to interface with various devices such as MRI, fMRI, fNIRS, and EEG to achieve automatic data acquisition and transmission. It can also ensure that data from different devices can be successfully connected to the system through standardized interface protocols, while recording metadata information such as data acquisition time and device parameters. The data preprocessing module is used to perform data format conversion, noise removal, and data calibration; The specific steps of the data preprocessing module are as follows: Step 1, Format Conversion: Based on the built-in multi-data format parser, brain data in different formats are converted into a unified intermediate format, such as NIfTI format, to facilitate subsequent processing; Step 2, noise removal: Based on the noise characteristics of different modal data, corresponding filtering algorithms are used for noise removal. For example, median filtering is used to remove image noise from MRI data, and independent component analysis (ICA) is used to remove artifacts such as electrooculography and electromyography from EEG data. Step 3, Data Calibration: Perform time synchronization calibration on the data to ensure consistency of different modal data in the time dimension.

[0027] The data quality assessment module is used to assess the quality of preprocessed data and generate an assessment report. By incorporating a data quality assessment module, low-quality data can be pre-screened, preventing biases in fusion results due to data quality issues and improving the accuracy and effectiveness of data fusion. This provides a more reliable data foundation for subsequent clinical diagnosis and research analysis. Simultaneously, it helps data collectors and researchers identify problems during data collection, enabling improvements to collection methods and equipment parameter settings.

[0028] The specific steps of the data quality assessment module are as follows: Step 1, Quality Indicator Setting: Set multiple quality assessment indicators for different modalities of brain data; for example, for MRI and fMRI image data, set indicators such as image resolution, signal-to-noise ratio, and contrast; for EEG and fNIRS time series data, set indicators such as signal stability, sampling frequency accuracy, and baseline drift. Step 2, Automated Evaluation: Utilize algorithms to automatically perform quality checks on the data; for example, determine the quality of MRI images by calculating the signal-to-noise ratio, analyze the stability of EEG signals using statistical methods, and determine whether the data meets the standards based on set thresholds; Step 3, Report Generation: Generate a detailed data quality assessment report to visually display the assessment results of each indicator and mark the data parts with quality problems and the types of problems, such as artifact areas in MRI images and abnormal fluctuation ranges in EEG signals.

[0029] The intelligent semantic annotation module is used to semantically annotate data, so as to give the data a semantic understanding; By setting up an intelligent semantic annotation module, it can endow multimodal brain data with semantic understanding, so that data retrieval is no longer limited to basic conditions. Users can quickly locate relevant data through semantic keywords. During data fusion, semantic information can help the algorithm better understand the connotation of the data and optimize the fusion effect. At the same time, it helps to build a more intelligent data analysis model, providing more in-depth information for clinical and scientific research.

[0030] The specific steps of the intelligent semantic annotation module are as follows: Step 1, Data Reception and Preliminary Processing: First, the multimodal brain data, after passing the data quality assessment module, is transmitted to the intelligent semantic annotation module. Then, based on the characteristics of different modalities, preliminary format adaptation and content parsing are performed. For example, for MRI and fMRI image data, the header information of the image is parsed to obtain basic information such as scan parameters and resolution. For EEG and fNIRS time series data, key information such as sampling frequency and timestamps is organized. For related text report data, text encoding conversion and sentence segmentation are performed to prepare for subsequent semantic extraction. Step 2, Semantic Extraction: Semantic extraction is performed on image data, time series data, and text data; The semantic extraction process of the image data is as follows: using computer vision technology, feature recognition is performed on MRI and fMRI images; firstly, the anatomical structures of the brain, such as the cerebral cortex, basal ganglia, and ventricles, are automatically identified through a convolutional neural network model, and corresponding information, such as location and size, is labeled; then, for lesion areas, such as tumors and infarcts, their morphology, boundaries, and other features are first identified and converted into semantic descriptions, such as "a round or oval abnormal signal shadow is visible in the left temporal lobe, with unclear boundaries"; The semantic extraction process for the time series data is as follows: For EEG and fnirs time series data, signal processing and machine learning algorithms are used to extract semantics; firstly, through spectrum analysis, the energy distribution characteristics of different frequency bands (α, β, γ, etc.) are extracted from the EEG data and converted into semantics, such as "α band energy is significantly enhanced in the closed eye state"; then, the changing trends of oxyhemoglobin and deoxyhemoglobin are analyzed from the fnirs data and labeled, such as "the concentration of oxyhemoglobin in the prefrontal cortex region shows an upward trend after task stimulation"; The semantic extraction process of the text data is as follows: natural language processing technology is used to extract semantics from the text report; first, the named entity recognition algorithm is used to extract key entities in the text, such as patient symptoms (headache, dizziness) and diagnosis results (cerebral infarction, epilepsy), etc.; then, dependency parsing is used to understand the semantic relationships between words in the text, and further refine the complete semantic information, such as "the patient was admitted to the hospital due to a sudden headache and was diagnosed with subarachnoid hemorrhage after examination".

[0031] Step 3, Knowledge Graph Association: The extracted semantic information is associated with a pre-constructed brain medical knowledge graph, which includes rich knowledge nodes and relationships such as brain anatomy, physiological functions, disease symptoms, and diagnostic criteria. First, a semantic matching algorithm is used to connect the labeled brain structure semantics with the anatomical structure nodes in the knowledge graph. Then, disease-related semantics are linked with disease diagnosis nodes. For example, "left temporal lobe tumor" is associated with the "temporal lobe" anatomical structure node and the "tumor" disease node in the knowledge graph, and the association relationship is labeled, such as "occurred in," thereby clarifying the position and meaning of the data in the medical knowledge system. Step 4, Review and Correction of Annotation Results: Review the results of automatic annotation to check the accuracy, completeness and rationality of semantic annotation, and correct any errors or ambiguities in the annotations. Step 5, Dynamically Update Annotations: Establish a dynamic update mechanism for annotation rules and knowledge graphs to track the latest research findings in the medical field, updates to clinical guidelines, and the release of new diagnostic criteria. When new knowledge related to brain data annotation emerges, the annotation rules and knowledge graphs are updated in a timely manner. For example, if new research discovers new imaging features of a certain brain disease, the semantic annotation rules for the image data are updated, and the already annotated data is re-evaluated and corrected to ensure the timeliness and accuracy of the annotations. Step Six, Annotated Data Output: The reviewed and updated semantically annotated data is integrated with the original multimodal brain data, and a data file with semantically annotated information is output. The data file is stored in a standardized format for easy subsequent data retrieval. At the same time, an index of annotated data is established to facilitate quick retrieval and query of data with specific semantic annotations, providing more valuable information support for clinical diagnosis and scientific research analysis.

[0032] The data fusion module is used to first extract features, then perform data fusion, and finally mine the complex nonlinear relationships between different modalities of data. The specific steps of the data fusion module are as follows: Step 1, Feature Extraction: First, use deep learning models such as convolutional neural networks (CNN) to extract spatial features from MRI and fMRI image data, and then use recurrent neural networks to extract temporal features from EEG and fNIRS time series data. Step 2, Fusion Algorithm: A multimodal fusion algorithm based on attention mechanism is adopted to assign weights according to the importance of different modal data to the target task, so as to achieve deep data fusion; for example, in brain tumor diagnosis, MRI structural information and fMRI functional information are given higher weights to identify tumors more accurately. Step 3, Relationship Mining: Introducing graph neural networks to mine complex nonlinear relationships between different modalities of data in order to construct a data relationship graph, providing support for in-depth analysis of brain function and disease mechanisms.

[0033] The data storage module is used to employ a distributed storage system, such as Ceph, to distribute data across multiple storage nodes, thereby improving storage capacity and data read / write performance. Simultaneously, it utilizes data compression technologies, such as JPEG2000, to compress image data, reducing storage space usage. Furthermore, it establishes a data backup and recovery mechanism, regularly backing up important data to ensure data security and integrity. The data management and application module provides functions such as data indexing and retrieval, data analysis and visualization, access control and data prediction.

[0034] The data management and application module includes: The data indexing and retrieval module is used to build multi-dimensional data indexes to support fast retrieval based on conditions such as patient information, data modality, collection time, and disease type. It can also use full-text search technology, such as Elasticsearch, to achieve fuzzy search and precise matching. The data analysis and visualization module provides a variety of data analysis tools, such as statistical analysis and machine learning model training, to help users conduct in-depth data analysis. At the same time, it can use data visualization technology to intuitively display the fused data in various forms such as 3D images and dynamic charts, making it easier for doctors and researchers to understand the data characteristics. The access control module uses a role-permission model to assign different operation permissions to different users. For example, doctors can only view and analyze patient data, while researchers can download and perform secondary development on the data, ensuring data security and privacy. The data prediction module is used to first train a disease prediction model based on historical multimodal brain data and patient medical records, such as using deep learning algorithms to predict the postoperative recurrence probability of brain tumor patients and the seizure risk of epilepsy patients. Then, it combines the patient's current multimodal data, prediction results, and clinical guidelines to recommend personalized treatment plans for doctors. For example, for stroke patients, it recommends the priority and details of thrombolysis, surgery, or rehabilitation treatment based on their brain imaging and electrophysiological data. Finally, the prediction results and recommended plans are presented in a visual form to intuitively show the possible results of different decision-making paths and assist doctors and patients in making joint decisions.

[0035] By setting up a data prediction module, the application of multimodal brain data can be expanded from simple diagnosis and research to the fields of disease prediction and treatment decision-making, thereby enhancing the clinical value of the system; through data-driven prediction and recommendation, it provides doctors with scientific decision-making basis, which helps to improve treatment effects and patient prognosis; the visualized decision display can also enhance the efficiency of doctor-patient communication and decision-making transparency.

[0036] A smart storage and archiving method based on multimodal brain data fusion includes the following specific steps: S1: It can interface with various devices to achieve automatic data collection and transmission; and through standardized interface protocols, it can ensure that data from different devices can be successfully accessed into the system, while recording metadata information of the collected data. S2: First, based on the built-in multi-data format parser, brain data of different formats are converted into a unified intermediate format; then, according to the noise characteristics of different modal data, corresponding filtering algorithms are used to remove noise; then, the data is time-synchronized and calibrated to ensure the consistency of different modal data in the time dimension. S3: First, set multiple quality assessment indicators for brain data of different modalities; then use algorithms to automatically detect the quality of the data; then generate a detailed data quality assessment report to intuitively display the assessment results of each indicator and mark the data parts with quality problems and the types of problems. S4: First, the multimodal brain data, after passing the data quality assessment module, is transmitted to the intelligent semantic annotation module. Then, preliminary format adaptation and content parsing are performed based on the characteristics of different modalities. After parsing, semantic extraction is performed on image data, time-series data, and text data. Following extraction, the extracted semantic information is associated with a pre-constructed brain medical knowledge graph. A semantic matching algorithm connects the annotated brain structure semantics with anatomical structure nodes in the knowledge graph, and then links disease-related semantics with disease diagnosis nodes. After association, the automatic annotation results are reviewed to check the accuracy, completeness, and rationality of the semantic annotations. Any issues are addressed. Errors or unclear annotations are corrected. After review, a dynamic update mechanism for annotation rules and knowledge graphs is established to track the latest research findings in the medical field, updates to clinical guidelines, and the release of new diagnostic criteria. When new knowledge related to brain data annotation emerges, the annotation rules and knowledge graphs are updated promptly. Subsequently, the reviewed and updated semantically annotated data is integrated with the original multimodal brain data, and data files with semantic annotation information are output. The data files are stored in a standardized format for easy subsequent data retrieval. At the same time, an index of annotated data is established to facilitate quick retrieval and querying of data with specific semantic annotations, providing more valuable information support for clinical diagnosis and scientific research analysis. S5: First, a deep learning model is used to extract spatial features from MRI and fMRI image data. Then, a recurrent neural network is used to extract temporal features from EEG and fNIRS time series data. After extraction, a multimodal fusion algorithm based on an attention mechanism is used to assign weights according to the importance of different modal data to the target task, thereby achieving deep data fusion. After fusion, a graph neural network is introduced to mine the complex nonlinear relationships between different modal data to construct a data relationship map, providing support for in-depth analysis of brain function and disease mechanisms. S6: A distributed storage system is used to distribute data across multiple storage nodes to improve storage capacity and data read / write performance; at the same time, data compression technology is used to compress image data to reduce storage space usage; and a data backup and recovery mechanism is established to regularly back up important data to ensure data security and integrity.

[0037] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. An intelligent storage and archiving system based on multimodal brain data fusion, characterized in that, include: The data acquisition module is used to interface with various devices to achieve automatic data acquisition and transmission; it can also ensure that data from different devices can be successfully accessed into the system through standardized interface protocols, while recording metadata information of the acquired data. The data preprocessing module is used to perform data format conversion, noise removal, and data calibration; The data quality assessment module is used to assess the quality of preprocessed data and generate an assessment report. The intelligent semantic annotation module is used to semantically annotate data, so as to give the data a semantic understanding. The data fusion module is used to first extract features, then perform data fusion, and finally mine the complex nonlinear relationships between different modal data. The data storage module is used to distribute data across multiple storage nodes using a distributed storage system to improve storage capacity and data read / write performance; at the same time, it uses data compression technology to compress image data and reduce storage space usage. In addition, a data backup and recovery mechanism should be established to regularly back up important data and ensure data security and integrity. The data management and application module provides functions such as data indexing and retrieval, data analysis and visualization, access control and data prediction.

2. The intelligent storage and archiving system based on multimodal brain data fusion according to claim 1, characterized in that, The specific steps of the data preprocessing module are as follows: Step 1, Format Conversion: Based on the built-in multi-data format parser, brain data in different formats is converted into a unified intermediate format; Step 2, noise removal: Based on the noise characteristics of different modal data, corresponding filtering algorithms are used for noise removal; Step 3, Data Calibration: Perform time synchronization calibration on the data to ensure consistency of different modal data in the time dimension.

3. The intelligent storage and archiving system based on multimodal brain data fusion according to claim 2, characterized in that, The specific steps of the data quality assessment module are as follows: Step 1, Quality Indicator Setting: Set multiple quality assessment indicators for different modalities of brain data; Step 2, Automated Evaluation: Utilize algorithms to automatically perform quality checks on the data; Step 3, Report Generation: Generate a detailed data quality assessment report to visually display the assessment results of each indicator and mark the data parts with quality problems and the types of problems.

4. The intelligent storage and archiving system based on multimodal brain data fusion according to claim 1, characterized in that, The specific steps of the intelligent semantic annotation module are as follows: Step 1, Data Reception and Preliminary Processing: First, the multimodal brain data that has passed the data quality assessment module is transmitted to the intelligent semantic annotation module; then, preliminary format adaptation and content parsing are performed according to the characteristics of different modalities of data. Step 2, Semantic Extraction: Semantic extraction is performed on image data, time series data, and text data; Step 3, Knowledge Graph Association: The extracted semantic information is associated with the pre-constructed brain medical knowledge graph; first, the labeled brain structure semantics are connected with the anatomical structure nodes in the knowledge graph through a semantic matching algorithm, and then the disease-related semantics are linked with the disease diagnosis nodes. Step 4, Review and Correction of Annotation Results: Review the results of automatic annotation to check the accuracy, completeness and rationality of semantic annotation, and correct any errors or ambiguities in the annotations. Step 5, Dynamically update annotations: Establish a dynamic update mechanism for annotation rules and knowledge graphs to track the latest research results in the medical field, updates to clinical guidelines, and the release of new diagnostic criteria. When new knowledge related to brain data annotations emerges, the annotation rules and knowledge graphs will be updated in a timely manner. Step Six, Annotated Data Output: The reviewed and updated semantically annotated data is integrated with the original multimodal brain data, and a data file with semantically annotated information is output. The data file is stored in a standardized format for easy subsequent data retrieval. At the same time, an index of annotated data is established to facilitate quick retrieval and query of data with specific semantic annotations, providing more valuable information support for clinical diagnosis and scientific research analysis.

5. The intelligent storage and archiving system based on multimodal brain data fusion according to claim 4, characterized in that, The semantic extraction process of the image data is as follows: using computer vision technology, feature recognition is performed on MRI and fMRI images; firstly, the anatomical structure of the brain is automatically identified through a convolutional neural network model, and corresponding information such as location and size is labeled; then, for lesion areas, features are first identified and converted into semantic descriptions. The semantic extraction process for the time series data is as follows: For EEG and fnirs time series data, signal processing and machine learning algorithms are used to extract semantics; firstly, through spectrum analysis, the energy distribution characteristics of different frequency bands are extracted from the EEG data and converted into semantics; then, the changing trends of oxyhemoglobin and deoxyhemoglobin are analyzed from the fnirs data and labeled. The semantic extraction process of the text data is as follows: natural language processing technology is used to extract semantics from the text report; first, the key entities in the text are extracted through named entity recognition algorithm; then, dependency parsing is used to understand the semantic relationships between words in the text, and further refine the complete semantic information.

6. The intelligent storage and archiving system based on multimodal brain data fusion according to claim 1, characterized in that, The specific steps of the data fusion module are as follows: Step 1, Feature Extraction: First, use a deep learning model to extract spatial features from MRI and fMRI image data, and then use a recurrent neural network to extract temporal features from EEG and fNIRS time series data. Step 2, Fusion Algorithm: A multimodal fusion algorithm based on an attention mechanism is adopted to assign weights according to the importance of different modal data to the target task, thereby achieving deep data fusion; Step 3, Relationship Mining: Introducing graph neural networks to mine complex nonlinear relationships between different modalities of data in order to construct a data relationship graph, providing support for in-depth analysis of brain function and disease mechanisms.

7. The intelligent storage and archiving system based on multimodal brain data fusion according to claim 1, characterized in that, The data management and application module includes: The data indexing and retrieval module is used to build multi-dimensional data indexes to support fast retrieval based on conditions, and can use full-text search technology to achieve fuzzy search and precise matching. The data analysis and visualization module provides a variety of data analysis tools to help users conduct in-depth data analysis. At the same time, it can use data visualization technology to intuitively display the integrated data in various forms, making it easier for doctors and researchers to understand the data characteristics.

8. The intelligent storage and archiving system based on multimodal brain data fusion according to claim 7, characterized in that, The data management and application module also includes: The access control module is used to assign different operation permissions to different users using a role-permission model, ensuring data security and privacy. The data prediction module is used to first train a disease prediction model based on historical multimodal brain data and patient medical records; then, it combines the patient's current multimodal data, prediction results, and clinical guidelines to recommend personalized treatment plans for doctors; finally, the prediction results and recommended plans are presented in a visual form to intuitively show the possible results of different decision-making paths, assisting doctors and patients in making joint decisions.

9. A smart storage and archiving method based on multimodal brain data fusion, characterized in that, The specific steps are as follows: S1: It can interface with various devices to achieve automatic data collection and transmission; and through standardized interface protocols, it can ensure that data from different devices can be successfully accessed into the system, while recording metadata information of the collected data. S2: First, based on the built-in multi-data format parser, brain data of different formats are converted into a unified intermediate format; then, according to the noise characteristics of different modal data, corresponding filtering algorithms are used to remove noise; then, the data is time-synchronized and calibrated to ensure the consistency of different modal data in the time dimension. S3: First, set multiple quality assessment indicators for brain data of different modalities; then use algorithms to automatically detect the quality of the data; then generate a detailed data quality assessment report to intuitively display the assessment results of each indicator and mark the data parts with quality problems and the types of problems. S4: First, the multimodal brain data, after passing the data quality assessment module, is transmitted to the intelligent semantic annotation module. Then, preliminary format adaptation and content parsing are performed based on the characteristics of different modalities. After parsing, semantic extraction is performed on image data, time-series data, and text data. Following extraction, the extracted semantic information is associated with a pre-constructed brain medical knowledge graph. A semantic matching algorithm connects the annotated brain structure semantics with anatomical structure nodes in the knowledge graph, and then links disease-related semantics with disease diagnosis nodes. After association, the automatic annotation results are reviewed to check the accuracy, completeness, and rationality of the semantic annotations. Any issues are addressed. Errors or unclear annotations are corrected. After review, a dynamic update mechanism for annotation rules and knowledge graphs is established to track the latest research findings in the medical field, updates to clinical guidelines, and the release of new diagnostic criteria. When new knowledge related to brain data annotation emerges, the annotation rules and knowledge graphs are updated promptly. Subsequently, the reviewed and updated semantically annotated data is integrated with the original multimodal brain data, and data files with semantic annotation information are output. The data files are stored in a standardized format for easy subsequent data retrieval. At the same time, an index of annotated data is established to facilitate quick retrieval and querying of data with specific semantic annotations, providing more valuable information support for clinical diagnosis and scientific research analysis. S5: First, a deep learning model is used to extract spatial features from MRI and fMRI image data. Then, a recurrent neural network is used to extract temporal features from EEG and fNIRS time series data. After extraction, a multimodal fusion algorithm based on an attention mechanism is used to assign weights according to the importance of different modal data to the target task, thereby achieving deep data fusion. After fusion, a graph neural network is introduced to mine the complex nonlinear relationships between different modal data to construct a data relationship map, providing support for in-depth analysis of brain function and disease mechanisms. S6: A distributed storage system is used to distribute data across multiple storage nodes to improve storage capacity and data read / write performance; at the same time, data compression technology is used to compress image data to reduce storage space usage. In addition, a data backup and recovery mechanism should be established to regularly back up important data and ensure data security and integrity.

Images