Intelligent cranial nerve computing platform based on multi-modal large model
Through a brain neural intelligent computing platform based on a multimodal large model, combined with a variety of brain neural computing tools and LoRA fine-tuning strategies, the problem of complex cross-platform operation of existing tools has been solved, a low-code, low-difficulty brain neural computing process has been realized, and operational convenience and data processing efficiency have been improved.
Patent Information
- Application Number
- CN202510633549.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2025-09-05
AI Technical Summary
Existing brain neural intelligent computing tools are complex to operate and difficult to use across platforms, which makes industrial implementation difficult. In particular, people who are unfamiliar with command line code or Linux systems require extensive training, which hinders the popularization of the tools.
A brain neural intelligent computing platform based on a multimodal large model is designed. It combines multiple brain neural computing tools, performs instruction parsing and feedback through the multimodal large model, supports flexible task scheduling and resource allocation, and uses the LoRA strategy to fine-tune the model to adapt to the specific process of brain neural computing.
It realizes a low-difficulty, low-code brain neural computing process, improves the convenience and accuracy of operation, facilitates researchers to get started, and supports cross-platform calls and efficient data processing.
Smart Images
Figure CN120599449A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of artificial intelligence, multimodal large models and medical image processing, and relates to a brain neural intelligent computing platform based on multimodal large models. Background Art
[0002] With the rapid development of artificial intelligence technology, brain neural intelligent computing (including brain white matter segmentation, brain network analysis and brain neuroimaging, etc.) has achieved outstanding results. However, due to the pain points such as the multiple image data modalities involved, complex operations, and difficulty for non-professionals to quickly get started, its industrial implementation is difficult and its popularization and dissemination is low.
[0003] Previous neural network intelligent computing tools (such as MRtrix3, FreeSurfer, and FSL) were complex to operate, relied heavily on code command execution, and were difficult to use across platforms. This resulted in some novice users, especially those unfamiliar with command line code or Linux systems, requiring extensive training and practice, which greatly restricted the industrial implementation and widespread adoption of neural network computing tools. However, with the development of large-scale artificial intelligence models, especially multimodal large-scale models, their excellent intelligent dialogue and flexible response capabilities have made low-difficulty, low-code, automated neural network intelligent computing processes possible. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies and address the inherent "data island" problem faced by existing large-scale AI models, and to adapt these models to current neural computing processes while integrating multiple existing tools, this paper proposes a work platform that combines multiple neural computing tools with multimodal AI models. This platform can efficiently process multimodal data and supports flexible task scheduling and resource allocation. This platform facilitates quick and easy neural computing analysis and other operations, greatly facilitating researchers' entry into neural computing.
[0005] The present invention provides the following technical solutions:
[0006] A brain-neural intelligent computing platform based on a multimodal large model. The user first issues an instruction, and the multimodal large model parses the instruction for requirements and feedback. It then generates JSON-formatted data from the parsed results, parses these instructions through the HTTP protocol, and sends them to the brain-neural computing tool according to the functional process. After processing these instructions, the brain-neural computing tool returns the feedback results to the large model, which receives the feedback and determines whether the processing is successful. If the processing fails, the process will return to the requirements and feedback parsing link for reprocessing until successful feedback is obtained. Finally, the successful result will be output, realizing automatic parsing of user instructions, efficient interaction with the neural computing tool, and reliable output of results, ensuring the accuracy and closed-loop nature of instruction processing.
[0007] Furthermore, the brain neural intelligent computing platform includes:
[0008] Multimodal data processing module: Brain neural computing involves input images of multiple modalities, such as T1w, FA, Peaks, CT, MRA, and other multimodal and cross-scale images. Users input commands through conversational methods, and the multimodal artificial intelligence model analyzes these input images according to user needs.
[0009] The task scheduling module implements the scheduling of user instructions and builds task transmission scheduling based on the system's internal HTTP protocol. After the multimodal data processing module parses the user instructions and input images, the task scheduling module reasonably allocates corresponding tools according to user needs to perform processing such as preprocessing, denoising, alignment, or neural connection analysis;
[0010] The model training and fine-tuning module conducts secondary training or fine-tuning of multimodal large models for application scenarios of the neural computing platform to meet the customized needs of specific neural computing scenarios. Currently, most multimodal large models do not focus on the specific processes of brain neural computing and the general needs of related personnel during training. Therefore, it is necessary to fine-tune the multimodal large models according to the specific processes of brain neural computing to meet the needs of brain neural computing.
[0011] Preferably, the processing process of the model training and fine-tuning module is:
[0012] Step 1. Data Source: The data used for fine-tuning the multimodal AI model mainly comes from the brain neural processing command records of researchers at the Brain-Machine Intelligence Research Center of Zhejiang University of Technology. These mainly include operation records such as registration, denoising, segmentation, correction, white matter imaging, and brain network analysis.
[0013] Step 2, LoRA fine-tuning: The LoRA (Low-Rank Adaptation) strategy is mainly used to fine-tune the multimodal large model. This strategy can greatly reduce the number of training parameters and reduce the memory usage. When updating parameters, only two low-rank matrices are updated, reducing the computational overhead. The expression of LoRA is:
[0014] h=W0x+ΔWx=W0x+BAx
[0015] Where x is the input vector, W0 is the pre-trained weight, ΔW is the incremental parameter matrix, B and A are low-rank weight matrices. During initialization, ΔW = BA = 0. During training, the training parameters W0 are fixed and B and A are trained. After training, B and A are calculated and merged with the pre-trained weights.
[0016] In the user interaction phase, the user enters their instructions, which serves as the starting point for the entire process. The user explains their needs in a conversational manner and receives the final result after the entire process is completed. When the feedback is successful, the final processing result is output, and the process ends.
[0017] During the multimodal large model parsing and execution phase, the model first parses the user's instructions to understand their needs and analyzes previous feedback to prepare for subsequent processing. After parsing, the user's needs and related information are converted into JSON format to facilitate subsequent data interaction with external tools. The generated JSON instructions are then sent to the neural computing tool via HTTP and feedback is received from the tool set. If the feedback fails, the process returns to the "Demand and Feedback Parsing" stage for reprocessing; if the feedback is successful, the next step is continued.
[0018] The neuroscience toolkit includes a variety of classic neuroscience computational tools, capable of performing tasks such as registration, segmentation, brain connectivity analysis, and white matter tract tracing. All tools are deployed in Docker containers, enabling cross-platform access. Upon receiving instructions, each tool processes data and provides feedback according to the workflow defined by the multimodal model.
[0019] The present invention has the following benefits: it can build a neural intelligence computing platform by combining multimodal large models, allowing for convenient and rapid neural computing and analysis operations; it can efficiently process multimodal data and support flexible task scheduling and resource allocation. This convenient and rapid neural computing and analysis greatly facilitates researchers' entry into brain neural computing. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is a workflow diagram of the brain neural intelligent computing platform based on a multimodal large model. DETAILED DESCRIPTION
[0021] The present invention will be further described below with reference to the accompanying drawings.
[0022] Reference Figure 1 , a brain neural intelligent computing platform based on a multimodal large model. The user first issues an instruction, and the multimodal large model parses the instruction for demand and feedback. Then, the parsing result is generated into data in JSON format. After parsing these instructions through the HTTP protocol, it is sent to the brain neural computing tool according to the functional process. The brain neural computing tool includes many processing tools. After processing these instructions, the feedback results are returned to the large model. The large model receives the feedback and determines whether the processing is successful. If the processing fails, the process will return to the demand and feedback parsing link for reprocessing until successful feedback is obtained. Finally, the successful result will be output. The entire process realizes the automatic parsing of user instructions, efficient interaction with the neural computing tool, and reliable output of results, ensuring the accuracy and closed-loop nature of instruction processing.
[0023] The brain neural intelligence computing platform based on the multimodal large model of this embodiment includes module 1, module 2 and module 3, which are described as follows:
[0024] Module 1: Multimodal Data Processing Module
[0025] Neuro-analysis involves inputting images from multiple modalities, such as T1w, FA, Peaks, CT, and MRA, all across multiple scales. Users input commands through a conversational interface. A multimodal AI model analyzes these input images according to user needs.
[0026] Module 2: Task Scheduling Module
[0027] This module primarily implements the scheduling of user commands and builds a task transmission schedule based on the system's internal HTTP protocol. After Module 1 parses the user command and the input image, the task scheduling module rationally assigns the appropriate tools to perform tasks such as preprocessing, denoising, registration, or neural connectivity analysis based on user needs.
[0028] Module 3: Model training and fine-tuning module
[0029] This module primarily performs secondary training or fine-tuning of large multimodal models for specific application scenarios within neural computing platforms to meet the customized needs of specific neural computing scenarios. Currently, most large multimodal models are trained without considering the specific processes of neural computing and the general needs of relevant personnel. Therefore, fine-tuning of large multimodal models to meet the specific needs of neural computing requires specific processes.
[0030] The overall process of the brain neural intelligent computing platform based on the multimodal large model of this embodiment includes three main parts: (1) user interaction part; (2) multimodal large model parsing and execution part; (3) brain neural tool collection part.
[0031] In the user interaction part, the user enters their own instructions as the starting point of the entire process. The user explains their needs in a conversational way and obtains the final result after the entire process is completed. If the feedback is successful, the final processing result is output and the process ends.
[0032] In the parsing and execution part of the multimodal large model, the multimodal large model first parses the user's instructions to understand the user's needs, and analyzes the previous feedback to prepare for subsequent processing. After parsing, the user's needs and related information are converted into JSON format to facilitate subsequent data interaction with external tools. The generated JSON instructions are then sent to the brain neural computing tool via the HTTP protocol and feedback from the brain neural computing tool set is received. If the feedback fails, the process returns to the "demand and feedback parsing" link for reprocessing; if the feedback is successful, proceed to the next step.
[0033] The neurological tool set includes a variety of classic neurological computational tools, capable of performing tasks such as registration, segmentation, brain connectivity analysis, and white matter tract tracing. All tools are deployed in Docker containers, enabling cross-platform access. Upon receiving instructions, each tool processes data and provides feedback according to the workflow defined by the multimodal model.
[0034] For module 3, the model training and fine-tuning module includes the following steps:
[0035] Step 1. Data source: The data used for fine-tuning the multimodal artificial intelligence large model mainly comes from the brain neural processing command records of researchers at the Brain-Computer Intelligence Research Center of Zhejiang University of Technology, mainly including operation records such as alignment, denoising, segmentation, correction, white matter imaging, and brain network analysis.
[0036] Step 2, LoRA fine-tuning: The LoRA (Low-Rank Adaptation) strategy is mainly used to fine-tune the multimodal large model. This strategy can greatly reduce the number of training parameters and reduce the memory usage. When updating parameters, only two low-rank matrices are updated, reducing the computational overhead. The expression of LoRA is:
[0037] h=W0x+ΔWx=W0x+BAx
[0038] Where x is the input vector, W0 is the pre-trained weight, ΔW is the incremental parameter matrix, B and A are low-rank weight matrices. During initialization, ΔW = BA = 0. During training, the training parameters W0 are fixed and B and A are trained. After training, B and A are calculated and merged with the pre-trained weights.
[0039] This embodiment uses a large multimodal model as its driver. The platform can efficiently process multimodal data, implement unified multimodal task scheduling and resource management, and provide a flexible development interface to support a variety of complex application scenarios. It can efficiently process multimodal data and support flexible task scheduling and resource allocation. It can conveniently and quickly perform neural computing analysis and other operations, greatly facilitating researchers' entry into brain neural computing.
[0040] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.
Claims
1. A brain neural intelligence computing platform based on a multimodal large model, characterized by: The user first issues a command, and the multimodal big model parses the command for requirements and feedback. It then generates JSON-formatted data from the parsed results. The parsed commands are then sent to the neural computing tool via HTTP according to the functional flow. After processing the commands, the neural computing tool returns the feedback results to the big model, which receives the feedback and determines whether the processing was successful. If the processing fails, the process will return to the demand and feedback analysis stage for reprocessing until successful feedback is obtained; Ultimately, successful results will be output, enabling automatic parsing of user commands, efficient interaction with neural computing tools, and reliable output of results, ensuring the accuracy and closed-loop nature of command processing.
2. The multimodal large model-based brain neural intelligent computing platform according to claim 1, characterized in that: The brain neural intelligent computing platform includes: Multimodal data processing module: Brain neural computing involves input images of multiple modalities, including T1w, FA, Peaks, CT, or MRA. Users input commands through conversational methods, and the multimodal artificial intelligence model analyzes these input images according to user needs. The task scheduling module schedules user commands and builds task transmission scheduling based on the system's internal HTTP protocol. After the multimodal data processing module parses the user's commands and input images, the task scheduling module rationally allocates corresponding tools to perform processing based on user needs, including preprocessing, denoising, alignment, or neural connection analysis. The model training and fine-tuning module performs secondary training or fine-tuning of multimodal large models for application scenarios of the neural computing platform to meet the customized needs of specific neural computing scenarios.
3. The multimodal large model-based brain neural intelligent computing platform according to claim 2, characterized in that: The processing process of the model training and fine-tuning module is as follows: Step 1. Data Source: The data used for fine-tuning the multimodal AI model comes from the brain neural processing command records of researchers at the Brain-Machine Intelligence Research Center, including alignment, denoising, segmentation, correction, white matter imaging, and brain network analysis operation records; Step 2, LoRA fine-tuning: The LoRA strategy is used to fine-tune the multimodal large model. When updating the parameters, only two low-rank matrices are updated. The expression of LoRA is: h=W0x+ΔWx=W0x+BAx Where x is the input vector, W0 is the pre-trained weight, ΔW is the incremental parameter matrix, B and A are low-rank weight matrices. During initialization, ΔW=BA=0 is set. During training, the training parameter W0 is fixed and B and A are trained. After the training is completed, B and A are calculated and merged with the pre-trained weights.
4. The multimodal large model-based brain neural intelligence computing platform according to any one of claims 1 to 3, characterized in that: In the user interaction part, users enter their own instructions as the starting point of the entire process; The user explains their needs in a conversational manner and obtains the final result after the entire process is completed. When the feedback is successful, the final processing result is output and the process ends.
5. The multimodal large model-based brain neural intelligence computing platform according to any one of claims 1 to 3, characterized in that: During the multimodal large model parsing and execution phase, the multimodal large model first parses the user's instructions to understand their needs and analyzes previous feedback to prepare for subsequent processing. After parsing, the user's needs and related information are converted into JSON format to facilitate subsequent data interaction with external tools. The generated JSON instructions are then sent to the brain neural computing tool via the HTTP protocol and feedback from the brain neural computing tool set is received. If the feedback fails, the process returns to the "demand and feedback parsing" phase for reprocessing. If the feedback is successful, proceed to the next step.
6. The multimodal large model-based brain neural intelligence computing platform according to any one of claims 1 to 3, characterized in that: The neurological tool set includes a variety of classic neurological computing tools that perform registration, segmentation, brain connectivity analysis, and brain white matter fiber bundle tracking. All tools are configured in a Docker container to enable cross-platform calls. When receiving instructions, each tool processes data and provides feedback according to the workflow set by the multimodal large model.