Multi-modal ai data governance and support system and method for medical research collaboration

Abstract

The application discloses a kind of medical scientific research collaborative multi-modal AI data management and support system and method, it is related to medical informatization and artificial intelligence technical field.The system includes regional collaborative management platform and multiple research nodes;Research node is used to obtain medical research data and process formation local data resource, constructs project level training data set based on project authorization strategy and executes model training in local controlled environment, generates local model update information;Regional collaborative management platform is used to execute federal aggregation on local model update information, generates global model parameter or shared model parameter and issues iterative training, outputs model parameter reaching preset condition.The application can realize regional multi-agency collaborative training, improve multi-modal medical research data utilization efficiency and model training effect.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a multimodal AI data governance and support system and method for collaborative medical research. Background Technology

[0002] With the continuous improvement of regional medical collaboration, smart hospital construction, and the digitalization of medical research, more and more hospitals, regional medical centers, disease alliances, research institutions, and medical-engineering cross-platforms need to jointly utilize multi-source medical data to carry out scientific research work such as disease prediction, assisted diagnosis, risk stratification, efficacy evaluation, prognostic analysis, multimodal joint modeling, and fine-tuning of large medical models.

[0003] Existing medical research data is typically distributed across electronic medical record systems, laboratory systems, imaging systems, pathology systems, follow-up systems, biobanks, and research databases within different medical institutions. It is characterized by dispersed sources, diverse modalities, inconsistent standards, significant differences in field semantics, varying collection frequencies, and inconsistent quality. In addition to conventional data such as structured medical records, laboratory test results, and medical orders, it also includes multimodal data such as medical images, pathological slides, free-text medical records, vital sign time-series data, and omics data. Due to significant differences in data coding systems, labeling criteria, sample organization methods, and time window definitions among different institutions, it is difficult to directly establish a unified, reliable, and trainable data foundation for cross-institutional research collaboration.

[0004] On the other hand, medical data is highly sensitive, involving requirements such as patient privacy protection, data security, research ethics, and compliance with usage regulations. Traditional centralized research and training models often require raw medical data to be aggregated to a central platform for unified cleaning and training. In regional multi-institutional collaboration scenarios, this can easily lead to problems such as data leakage risks, unclear access boundaries, difficulties in usage control, insufficient audit traceability, and unclear division of responsibilities. Therefore, it is difficult to meet the actual needs of current regional medical research collaboration.

[0005] In addition, while there are existing platform-based solutions for medical AI model development, these solutions typically emphasize model development and training in a controlled environment, supporting multiple development entities through training resource scheduling, and outputting training results in a controlled manner. However, these solutions are mainly geared towards supporting medical AI development within a single platform and have not yet formed a complete solution for collaborative research among multiple institutions in a region.

[0006] Specifically, existing technologies have at least the following shortcomings: First, they lack a localized governance mechanism for multimodal medical research data, making it difficult to simultaneously handle the collaborative processing of structured data, text data, imaging data, pathological data, and time-series data; second, they lack an integrated management mechanism that links research project approval, usage control, dataset version management, and result delivery; third, they lack a federated support mechanism to enable collaborative training of regional multi-institutional models without the original data leaving the domain; and fourth, they lack training adaptation and aggregation mechanisms to address modal heterogeneity, uneven data quality, and differences in node capabilities. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal AI data governance and support system and method for collaborative medical research, in order to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a multimodal AI data governance and federated training support method for regional medical research collaboration, comprising the following steps: Receive access requests from research nodes and perform authentication management on the research nodes; Acquire medical research data from research nodes, process the research data, and form local data resources; For each research project application, a corresponding authorization strategy is generated. Based on the project authorization strategy, a project-level training dataset is constructed from the local data resources, and model training is performed in a local controlled environment to generate local model update information; The local model update information is uploaded through the research node, federated aggregation is performed on the local model update information to generate global model parameters or shared model parameters, and then distributed to the research node for iterative training. Output the global or shared model parameters that meet the preset conditions after iterative training.

[0009] In a preferred embodiment, the research data is multimodal medical research data, which includes at least one of the following: structured medical record data, laboratory test data, medical orders and treatment data, medical imaging data, pathology data, free text medical records, follow-up data, vital sign time series data, and omics data.

[0010] As a preferred embodiment, the processing of scientific research data includes at least one of the following processes: desensitization, standardization, cleaning, label verification, quality scoring, and data lineage registration.

[0011] In a preferred embodiment, the processing of scientific research data includes: A unified patient index is established for data from different business systems and different data modalities, and multimodal sample alignment is completed based on patient identification, visit events, time windows, and sample source relationships.

[0012] In a preferred embodiment, the project-level training dataset has a corresponding dataset version identifier, which includes at least one or more of the following information: data source range, sample arrangement conditions, time window, label rule version, desensitization rule version, cleaning rule version, construction time, construction node identifier, and project identifier.

[0013] As a preferred embodiment, a federated training task is generated based on the data modality configuration of the research nodes participating in the iterative training; When multiple research nodes have the same data modality, a federated training task with a unified model structure is issued. When multiple research nodes have different data modalities, a federated training task containing shared backbone parameters and modality-specific branch parameters is issued. When performing federated aggregation, only the shared backbone parameters are aggregated. The modality-specific branch parameters are retained on the local node or aggregated in groups among research nodes with the same data modality.

[0014] As a preferred embodiment, the federated aggregation of local model update information includes weighted aggregation of local model update information uploaded by multiple research nodes. The weights of the weighted aggregation are determined based on at least one of the following factors: number of local samples, data quality score, label completeness, modal completeness, node historical stability, training loss change, and node contribution score.

[0015] Secondly, this invention provides a multimodal AI data governance and federated training support system for regional medical research collaboration, including a regional collaborative management platform and multiple research nodes; The regional collaborative management platform is used to communicate and connect with scientific research user terminals, management terminals, and multiple scientific research nodes. Each of the aforementioned research nodes is used to communicate and connect with the internal business systems of this institution; The system includes: The node access authentication module is used to receive access requests from research nodes and to perform authentication management on the research nodes. The data acquisition and processing module is used to acquire medical research data from research nodes, process the research data, and form local data resources. The project application authorization module is used to respond to research project applications and generate corresponding project application authorization strategies; The dataset construction and local training module is used to construct a project-level training dataset from the local data resources based on the project authorization strategy, perform model training in a local controlled environment, and generate local model update information. The parameter upload, federated aggregation and distribution module is used to upload the local model update information through the research node, perform federated aggregation on the local model update information to generate global model parameters or shared model parameters, and distribute them to the research node for iterative training. The results output module is used to output the global model parameters or shared model parameters that have reached the preset conditions after iterative training.

[0016] In a preferred embodiment, the data acquisition and processing module is further configured to: establish a unified patient index for data from different business systems and different data modalities, and complete multimodal sample alignment based on patient identification, visit events, time windows, and sample source relationships.

[0017] In a preferred embodiment, the parameter uploading, federated aggregation, and distribution module is configured to perform federated training based on the data modality configuration of the research nodes participating in the iterative training; When multiple research nodes have the same data modality, the corresponding local model update information is uniformly aggregated. When multiple research nodes have different data modalities, the local model update information containing shared backbone parameters and modality-specific branch parameters is aggregated, and only the shared backbone parameters are aggregated. The modality-specific branch parameters are retained in the local node or aggregated in groups among research nodes with the same data modality. The parameter upload, federated aggregation, and distribution module performs weighted aggregation of local model update information uploaded by multiple research nodes. The weight of the weighted aggregation is determined based on at least one of the following factors: number of local samples, data quality score, label completeness, modal completeness, node historical stability, training loss change, and node contribution score. The result output module is used to output one or more of the following: global model parameters, shared model parameters, model files, evaluation reports, experimental records, or desensitized statistical results.

[0018] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention constructs a technical architecture that enables a regional collaborative management platform and multiple research nodes to work together. This allows each research node to complete model training in a local, controlled environment. By uploading local model update information and federated aggregation, joint modeling by multiple institutions in the region is achieved. This enables regional medical research collaborative training without the need for centralized aggregation of original medical research data, thereby improving the collaborative research capabilities of multiple institutions and reducing data security risks.

[0019] 2. This invention processes medical research data to form local data resources, and preferably organizes data from different business systems and data modalities around the same research object by establishing a unified patient index and aligning multimodal samples, thereby improving the usability of multimodal medical research data and enhancing the ability of multimodal joint analysis and joint training.

[0020] 3. This invention performs federated training based on the data modality configuration of research nodes through parameter uploading, federated aggregation, and distribution modules. It preferably uses unified aggregation, group aggregation, or weighted aggregation to process local model update information, thereby improving the training adaptability in scenarios with heterogeneous modalities and the stability and effectiveness of global model parameters or shared model parameters. Attached Figure Description

[0021] Figure 1 is a schematic diagram of a multimodal AI data governance and support system for collaborative medical research, as shown in an embodiment of the present invention. Figure 2 is a schematic diagram of the scientific research node data processing flow according to an embodiment of the present invention; Figure 3 is a flowchart illustrating a multimodal AI data governance and support method for collaborative medical research, as shown in an embodiment of the present invention. Figure 4 is a schematic diagram illustrating the connection between data and scientific research nodes in the regional collaborative management platform according to an embodiment of the present invention; Figure 5 is a framework diagram of a multimodal AI data governance and support device for collaborative medical research, as shown in an embodiment of the present invention. Figure 6 is a schematic diagram of an electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 As shown, the present invention provides a multimodal AI data governance and support system for medical research collaboration, including a regional collaborative management platform 100, multiple research nodes 200, research user terminals 300, management terminals 400, and an internal business system 500.

[0024] In this embodiment, the regional collaborative management platform 100 can be deployed at a regional medical research collaboration center, a regional medical data platform, a disease-specific alliance research platform, or a platform with unified management capabilities. It is used to manage the access, project application, authorization, federated training, and result output of multiple research nodes 200 within the region. Multiple research nodes 200 are deployed within different medical institutions to complete medical research data acquisition, data processing, training dataset construction, local training, and parameter uploading within their respective institutional environments. The research user terminal 300 is used by researchers to initiate research project applications, view training progress, and obtain results. The management terminal 400 is used by administrators to manage nodes, review projects, and configure policies. The internal business systems 500 include, but are not limited to, electronic medical record systems, laboratory systems, imaging systems, pathology systems, follow-up systems, biobank systems, medical equipment systems, and research database systems.

[0025] The research node 200 acquires and processes the institution's medical research data to form local data resources that can be used by research projects. Under the control of the project application authorization module, the research node 200 constructs a project-level training dataset based on the project authorization strategy and performs local training. Then, the parameter upload, federated aggregation and distribution modules complete the federated aggregation and iterative distribution of model parameters. Finally, the result output module outputs the global model parameters or shared model parameters that meet the preset conditions.

[0026] like Figure 1 As shown, the regional collaborative management platform 100 includes a node access authentication module 110, a project application authorization module 120, a parameter upload, federated aggregation and distribution module 130, and a result output module 140.

[0027] The node access authentication module 110 is used to receive access requests from research nodes 200 and to perform authentication management on research nodes 200. Specifically, the node access authentication module 110 can verify the research node identifier, node certificate, information of the institution to which the node belongs, and communication security status, thereby ensuring that the research node 200 accessing the regional collaborative management platform 100 is a legitimate node.

[0028] The project application authorization module 120 is used to respond to research project applications and generate corresponding project authorization strategies. After a research user initiates a research project application through the research user terminal 300, the project application authorization module 120 can generate a project authorization strategy based on the research objective, participating institutions, data requirements, training objectives, usage period, and output requirements. The project authorization strategy is used to limit the scope of data resources that the corresponding research project can access and the scope of training execution.

[0029] The parameter upload, federated aggregation, and distribution module 130 is used to receive local model update information through the research node 200, perform federated aggregation on the local model update information to generate global model parameters or shared model parameters, and then distribute the global model parameters or shared model parameters to the research node 200 for iterative training. Preferably, the parameter upload, federated aggregation, and distribution module 130 can determine the aggregation method according to the data modality of the participating nodes; when multiple research nodes 200 have the same data modality, unified aggregation can be performed on the corresponding parameters; when multiple research nodes 200 have different data modalities, aggregation can be performed on the shared backbone parameters, while modality-specific branch parameters are retained on the local node or grouped and aggregated among nodes with the same modality.

[0030] In a preferred embodiment, the parameter upload, federated aggregation, and distribution module 130 can also perform weighted aggregation on the local model update information uploaded by multiple research nodes 200. The weights of the weighted aggregation can be determined based on one or more of the following: local sample quantity, data quality score, label completeness, modal completeness, node historical stability, training loss changes, and node contribution score.

[0031] The results output module 140 is used to output the global model parameters or shared model parameters that have reached the preset conditions after iterative training. Preferably, the results output module 140 can also output one or more of the following: model files, evaluation reports, experimental records, or anonymized statistical results, for authorized research users to access or download.

[0032] like Figure 2 , Figure 4 As shown, each research node 200 includes a data acquisition and processing module 210, a dataset construction and local training module 220, and a parameter interaction unit 230.

[0033] The data acquisition and processing module 210 is used to acquire medical research data from the research node 200 and process the research data to form local data resources. Specifically, the data acquisition and processing module 210 can acquire one or more of the following from the internal business system 500: structured medical record data, laboratory test data, medical orders and treatment data, medical imaging data, pathology data, free text medical records, follow-up data, vital sign time series data, and omics data.

[0034] In a preferred embodiment, the data acquisition and processing module 210 processes the research data in at least one of the following ways: desensitization, standardization, cleaning, label verification, quality scoring, and data lineage registration. Through the above processing, the raw medical research data can be transformed into local data resources suitable for use in research tasks.

[0035] In a preferred embodiment, the data acquisition and processing module 210 can also establish a unified patient index for data from different business systems and different data modalities, and complete multimodal sample alignment based on patient identification, visit events, time windows, and sample source relationships. Through the above processing, data from different systems and different modalities can be associated around the same research sample, facilitating the construction of subsequent training datasets.

[0036] The dataset construction and local training module 220 is used to construct a project-level training dataset from local data resources based on the project authorization strategy, and to perform model training in a local controlled environment to generate local model update information.

[0037] In this embodiment, the project-level training dataset may have a corresponding dataset version identifier. Preferably, the dataset version identifier includes at least one or more of the following information: data source range, sample arrangement conditions, time window, label rule version, desensitization rule version, cleaning rule version, construction time, construction node identifier, and project identifier.

[0038] The local controlled environment can be a container environment, a virtual machine environment, or other isolated training environment. The dataset construction and local training module 220 performs training tasks in this environment, and the generated local model update information can be parameter update information, gradient information, or other information that can be federated and aggregated.

[0039] The parameter interaction unit 230 is used to upload local model update information to the regional collaborative management platform 100 through the research node 200, and to receive global model parameters or shared model parameters issued by the regional collaborative management platform 100, so that the dataset construction and local training module 220 can continue to perform iterative training.

[0040] Figure 3 This is a flowchart illustrating a multimodal AI data governance and federated training support method for regional medical research collaboration, according to an exemplary embodiment. Figure 3 As shown, it may include the following steps.

[0041] In step S201, an access request from a research node is received, and the research node is authenticated and managed.

[0042] In the embodiments of this specification, the research node can be a computing node deployed within various medical institutions to carry out local medical research data governance and model training execution. It can be associated with attributes such as medical institution identifier, node network region, security level, available data modality, computing power resources, and supported training framework, and additional configuration parameters, including node certificate status, communication encryption method, access permission scope, types of research projects that can be participated in, node stability level, and data outbound control strategy.

[0043] In one possible implementation, access requests from research nodes can be received from node terminals deployed on the medical institution side. Node terminals can include various regional medical collaboration access devices, such as local research servers within the medical institution, edge computing nodes, dedicated medical research terminals, or regional collaboration access gateways. After receiving the node access request, the regional collaboration management platform can forward the request to the node access authentication module for subsequent verification and management operations.

[0044] In one alternative implementation, the regional collaborative management platform may need to interact with research nodes multiple times to complete identity verification and access authorization. For example, during the node access process, steps such as certificate verification, institutional qualification verification, communication security handshake, and node capability reporting may need to be performed. These steps can be completed through multiple interactions between the regional collaborative management platform and the research nodes.

[0045] In another possible implementation, access requests can be received through real-time secure communication with research nodes. For example, a research node initiates an access request via an encrypted communication protocol, and the regional collaborative management platform establishes a trusted connection with the research node through a secure interface, receives the access information, and performs integrity verification to ensure the authenticity and legitimacy of the node's access request.

[0046] In one optional implementation, the regional collaborative management platform can manage the information of registered and pending research nodes through an authentication management database. This database records the authentication information and configuration parameters of all compliant research nodes. Upon receiving a research node's access request, the regional collaborative management platform can query this database to complete identity verification and permission matching, ensuring that the accessing research node is a legitimate and compliant node and meets the access requirements for regional medical research collaboration.

[0047] In one possible implementation, when managing the authentication of research nodes, priority can be given to verifying the node's identity legitimacy, certificate validity, institutional affiliation authenticity, and communication security status, so as to make subsequent access management and task allocation more efficient and reliable.

[0048] In practical applications, by receiving access requests from research nodes and completing authentication management, a trusted regional collaborative access environment can be built, ensuring that all nodes participating in research collaboration are compliant nodes with matching computing power and data capabilities, thereby improving the security, stability, and overall operational efficiency of regional medical research collaborative training.

[0049] In step S202, medical research data from the research node is acquired, and the research data is processed to form local data resources.

[0050] In the embodiments of this specification, medical research data can be multimodal data generated by the internal business system of a medical institution, containing information on all dimensions of patient diagnosis and treatment. It can be associated with data types such as structured medical records, laboratory examinations, medical orders, medical images, pathology, free text medical records, follow-up, vital sign time series, and omics, and is supplemented with multiple governance parameters, including desensitization rules, standardized mapping relationships, cleaning thresholds, label verification standards, data quality scoring dimensions, and data lineage registration fields.

[0051] In one possible implementation, medical research data can be obtained from the medical institution's internal business systems. These internal business systems may include various medical data generation and storage systems, such as electronic medical record systems, laboratory information systems, medical image archiving and communication systems, pathology information systems, follow-up management systems, and biobank systems. After receiving data pushed by the internal business systems, the research node can transfer the data to the data acquisition and processing module for subsequent processing and handling.

[0052] In one alternative implementation, research nodes may need to interact with internal business systems multiple times to complete multimodal data retrieval and integrity verification. For example, during the data acquisition process, it is necessary to extract structured data, image files, text data, and time-series data in batches and verify data integrity and timeliness. These steps can be achieved through multiple interactions between research nodes and internal business systems.

[0053] In another possible implementation, medical research data can be obtained through real-time data integration with internal business systems. For example, internal business systems can synchronously add new diagnostic and treatment data through standardized data interfaces, while research nodes can establish a stable connection with internal business systems through encrypted interfaces, receive research data, and perform format verification to ensure the accuracy and standardization of medical research data.

[0054] In one alternative implementation, a data governance database within the research node can be used to manage governance rules and processing logs for multimodal medical research data. This database records all data processing rules, transformation logic, and quality results. After acquiring medical research data, the research node can invoke the governance rules in this database to complete data processing, ensuring that the resulting local data resources comply with medical research usage standards.

[0055] In one possible implementation, when processing research data, desensitization, standardization, cleaning, unified patient indexing, and multimodal sample alignment can be prioritized to make subsequent project dataset construction more efficient and compliant.

[0056] In practical applications, by acquiring medical research data and completing localized processing, a unified and standardized research data resource can be formed within the institution, ensuring that the data is available, reliable, and traceable, while meeting the requirements for patient privacy protection and data security, and providing a reliable data foundation for subsequent medical research model training.

[0057] In step S203, in response to the research project application, the corresponding authorization strategy for the project application is generated.

[0058] In the embodiments of this specification, a research project application can be a project request initiated by a research user through a research user terminal, which includes information on all dimensions of research. It can be associated with information such as research purpose, target disease, scope of participating institutions, data modality requirements, sample inclusion and exclusion conditions, training objectives, and usage period, and can be supplemented with multiple approval parameters, including data sensitivity level, permission validity period, result output range, model usage restrictions, ethical review status, project priority, etc.

[0059] In one possible implementation, research project applications can be obtained from research user terminals. These terminals can include various network-connected devices, such as researchers' office computers, mobile research terminals, and regional research collaboration platform clients. After receiving the research project application, the regional collaboration management platform can forward it to the project application authorization module for subsequent review and policy generation.

[0060] In one alternative implementation, the regional collaborative management platform may need to interact with the management terminal multiple times to complete project review and permission confirmation. For example, during the project approval process, steps such as compliance verification, data permission assessment, ethical review, and institutional authorization confirmation need to be performed. These steps can be completed through multiple interactions between the regional collaborative management platform and the management terminal.

[0061] In another possible implementation, research project applications can be obtained through real-time communication with research user terminals. For example, research users fill out and submit project applications through the platform interface, and the regional collaborative management platform receives the application information through a secure interface and performs compliance verification to ensure the integrity and rationality of the research project applications.

[0062] In one optional implementation, research project application information and authorization policy templates can be managed through a project management database within the regional collaborative management platform. This database records all project approval rules, permission configurations, and policy templates. Upon receiving a research project application, the regional collaborative management platform can query this database to generate a matching authorization policy, ensuring that the policy complies with research compliance requirements and data usage standards.

[0063] In one possible implementation, when generating a project authorization policy, priority can be given to limiting the scope of data use, training execution boundaries, and result delivery permissions, so that subsequent dataset construction and model training strictly comply with compliance requirements.

[0064] In practical applications, by responding to research project applications and generating authorization policies, fine-grained access control for research projects can be achieved, ensuring that data use and model training are carried out within the authorized scope, and meeting the requirements of medical research ethics and data security management.

[0065] In step S204, based on the project authorization strategy, a project-level training dataset is constructed from the local data resources, and model training is performed in a local controlled environment to generate local model update information.

[0066] In the embodiments of this specification, the project-level training dataset can be a special scientific research dataset selected and processed from local data resources based on the project authorization strategy. It can be associated with information such as training set, validation set, test set, sample sorting results, and multimodal alignment data, and is supplemented with multiple version parameters, including data source range, time window, label rule version, desensitization rule version, construction time, node identifier, project identifier, etc.

[0067] In one possible implementation, a project-level training dataset can be constructed by extracting data from local data resources based on a project authorization policy. The local data resources can be a well-governed collection of multimodal medical research data, encompassing various data types such as structured, text, image, pathology, and time-series data. After obtaining the authorization policy, the research node can pass the policy to the dataset construction and local training module to carry out dataset construction and training operations.

[0068] In one alternative implementation, research nodes may need to filter and process local data resources multiple times to complete the construction of a project-level training dataset. For example, the dataset construction process requires steps such as sample sorting, multimodal alignment, feature processing, data augmentation, and version tagging, which can be completed multiple times by the data processing components of the research node.

[0069] In another possible implementation, model training and update information can be performed in a local, controlled environment. This local, controlled environment can include secure training scenarios such as container environments, virtual machines, and local isolated computing environments. Research nodes load the project dataset in this controlled environment, perform model training, and output only updated information such as model parameters and gradients, ensuring that the original data is not leaked.

[0070] In one alternative implementation, a dataset management database within the research node can be used to manage version information and build logs of project-level training datasets. This database can record all dataset versions, build rules, and usage records. After building a dataset, the research node can store the version information in the database, ensuring that the dataset is traceable and reproducible.

[0071] In one possible implementation, when performing local model training, an isolated environment can be prioritized, data access permissions can be restricted, and only model update information can be output, so as to complete local training while ensuring data security.

[0072] In practical applications, by constructing project-level datasets and training them locally under controlled conditions, model training can be completed without the original data leaving the domain, balancing research efficiency and data security, and improving the compliance and stability of regional collaborative training.

[0073] In step S205, the local model update information is uploaded through the research node, federated aggregation is performed on the local model update information to generate global model parameters or shared model parameters, and then distributed to the research node for iterative training.

[0074] In the embodiments of this specification, the local model update information can be aggregated information such as model parameters, gradients, and feature summaries generated by local training on research nodes. It can be associated with information such as model structure, training rounds, node identifiers, data modalities, and loss values, and is supplemented with multiple aggregated parameters, including node sample size, data quality score, label completeness, modal completeness, node stability, and contribution score.

[0075] In one possible implementation, local model update information can be uploaded to the regional collaborative management platform via research nodes. These research nodes can be nodes from medical institutions that have completed local training and support communication capabilities such as encrypted transmission, parameter verification, and resume capability. After generating local model update information, the research node can upload it through the federated parameter interaction module to initiate subsequent federated aggregation operations.

[0076] In one alternative implementation, the regional collaborative management platform may need to interact with multiple research nodes multiple times to complete parameter reception and federated aggregation. For example, the aggregation process requires steps such as receiving parameters from multiple nodes, verifying the validity of information, performing aggregation according to modality and weight, and distributing global parameters. These steps can be achieved through multiple interactions between the platform and the research nodes.

[0077] In another possible implementation, differentiated federated aggregation can be performed based on the data modality configuration of the participating nodes. When multiple nodes have the same modality, aggregation with unified model parameters is performed; when the modalities are different, only the shared backbone parameters are aggregated, while modality-specific branch parameters are retained locally or grouped for aggregation, adapting to heterogeneous data scenarios.

[0078] In one alternative implementation, model update information, aggregation processes, and parameter versions can be managed through a federated aggregation database within the regional collaborative management platform. This database records all node upload information, aggregation weights, and global parameter history. When performing aggregation, the platform can call this database to complete weighted calculations and parameter storage, ensuring the traceability of the aggregation process.

[0079] In one possible implementation, when performing federated aggregation, weighted aggregation and modality-adaptive aggregation strategies can be prioritized to improve the accuracy and stability of the global model.

[0080] In practical applications, by uploading model update information and performing federated aggregation and distribution, cross-institutional model iteration can be completed without aggregating the original data, thereby improving the generalization ability and training effect of regional collaborative scientific research models.

[0081] In step S206, the global model parameters or shared model parameters that have reached the preset conditions after iterative training are output.

[0082] In the embodiments of this specification, the preset conditions can be the termination criteria for federated iterative training, which can be associated with information such as training rounds, model performance thresholds, loss function convergence values, and evaluation metric achievement values, and can be supplemented with multiple output parameters, including authorized user scope, delivery form, anonymization requirements, audit records, and usage period.

[0083] In one possible implementation, the regional collaborative management platform can output the final model results to authorized research users. The results may include global model parameters, shared model parameters, model files, evaluation reports, experimental records, and anonymized statistical results. After the regional collaborative management platform determines that the training has met the standards, it can transfer the results to the results output module for compliant delivery.

[0084] In one alternative implementation, the regional collaborative management platform may need to perform multiple verifications and anonymization processes on the output results to achieve compliant delivery. For example, during the output process, steps such as permission verification, result anonymization, content encryption, and audit logging need to be performed. These steps can be completed multiple times by the platform's result delivery component.

[0085] In another possible implementation, results can be delivered to research user terminals via encrypted delivery and authorization verification. Research user terminals can only access results within the authorized scope and cannot access raw data from across institutions, thus ensuring data security and privacy compliance.

[0086] In one alternative implementation, output results, delivery records, and audit logs can be managed through a result management database within the regional collaborative management platform. This database can record all delivered content, receiving users, delivery times, and usage permissions. After outputting results, the platform can store delivery information in the database, achieving full-process auditability and traceability.

[0087] In one possible implementation, when outputting the final result, authorization permissions can be verified first, de-identification processing can be performed, and the export of original data can be prevented in order to ensure that the delivery of the result is compliant and secure.

[0088] In practical applications, by outputting the global model parameters after they meet the standards, the entire process of regional medical research collaborative training can be completed, providing researchers with usable, compliant, and reliable model results to support medical research and clinical translation.

[0089] Please see Figure 5 The embodiments of the present invention provide a multimodal AI data governance and support device for medical research collaboration, including a regional collaborative management platform and multiple research nodes; The regional collaborative management platform is used to communicate and connect with scientific research user terminals, management terminals, and multiple scientific research nodes. Each of the aforementioned research nodes is used to communicate and connect with the internal business systems of this institution; The device includes: The node access authentication module 110 is used to receive access requests from research nodes and to perform authentication management on the research nodes. The data acquisition and processing module 210 is used to acquire medical research data from research nodes, process the research data, and form local data resources. The project application authorization module 120 is used to respond to scientific research project applications and generate corresponding project application authorization strategies; The dataset construction and local training module 220 is used to construct a project-level training dataset from the local data resources based on the project authorization strategy, and to perform model training in a local controlled environment to generate local model update information. The parameter upload, federated aggregation and distribution module 130 is used to upload the local model update information through the research node, perform federated aggregation on the local model update information, generate global model parameters or shared model parameters, and distribute them to the research node for iterative training. The result output module 140 is used to output the global model parameters or shared model parameters that have reached the preset conditions after iterative training.

[0090] Figure 6 is a block diagram of an electronic device for supporting multimodal AI data governance and federated training in regional medical research collaboration, according to an exemplary embodiment. This electronic device can be a regional collaborative management platform server, a medical institution-side research node server, a research user terminal, or a management terminal, and its internal structure is shown in Figure 6. The electronic device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities to support multimodal data governance, federated training scheduling, model aggregation, and access control. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and data such as medical research data governance rules, federated training configurations, node authentication information, and project authorization policies. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the electronic device is used to communicate with external medical institution research nodes, business systems, research user terminals, and management terminals via an encrypted network connection to complete node access authentication, parameter interaction, and federated aggregation. When executed by a processor, this computer program implements a multimodal AI data governance and federated training support method for regional medical research collaboration. The electronic device's display screen can be an LCD screen or an e-ink screen, used to display research project progress, federated training status, data quality scores, and model evaluation results. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the device's casing, or an external keyboard, touchpad, or mouse, used for initiating research project applications, configuring nodes, editing authorization policies, and managing training tasks.

[0091] Those skilled in the art will understand that the structure shown in FIG6 is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0092] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the multimodal AI data governance and federated training support method for regional medical research collaboration in this disclosure embodiment.

[0093] In an exemplary embodiment, a computer-readable storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the multimodal AI data governance and federated training support method for regional medical research collaboration as described in this disclosure. The computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0094] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the multimodal AI data governance and federated training support method for regional medical research collaboration as described in this disclosure.

[0095] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0096] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0097] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal AI data governance and support method for collaborative medical research, characterized in that, Includes the following steps: Receive access requests from research nodes and perform authentication management on the research nodes; Acquire medical research data from research nodes, process the research data to form local data resources, wherein the processing of research data includes: establishing a unified patient index for data from different business systems and different data modalities, and completing multimodal sample alignment based on patient identifiers, medical events, time windows and sample source relationships, and the processing of research data also includes performing data quality scoring. Respond to research project applications and generate corresponding authorization strategies for the applications; Based on the project authorization strategy, a project-level training dataset is constructed from the local data resources, and model training is performed in a local controlled environment to generate local model update information; The local model update information is uploaded through research nodes, and federated aggregation is performed on the local model update information to generate global model parameters or shared model parameters, which are then distributed to research nodes for iterative training. The federated aggregation of local model update information includes weighted aggregation of local model update information uploaded from multiple research nodes, with the weights determined based on the data quality score. When multiple research nodes have different data modalities, a federated training task containing shared backbone parameters and modality-specific branch parameters is distributed. During federated aggregation, only the shared backbone parameters are aggregated; the modality-specific branch parameters are retained on the local nodes or aggregated in groups among research nodes with the same data modality. The global model parameters or shared model parameters that meet preset conditions after iterative training are output.

2. The method according to claim 1, characterized in that, The research data is multimodal medical research data, which includes at least one of the following: structured medical record data, laboratory test data, medical orders and treatment data, medical imaging data, pathology data, free text medical records, follow-up data, vital sign time series data, and omics data.

3. The method according to claim 1, characterized in that, The processing of scientific research data also includes at least one of the following: desensitization, standardization, cleaning, label verification, and data lineage registration.

4. The method according to claim 1, characterized in that, The project-level training dataset has a corresponding dataset version identifier, which includes at least one or more of the following information: data source range, sample arrangement conditions, time window, label rule version, desensitization rule version, cleaning rule version, construction time, construction node identifier, and project identifier.

5. The method according to claim 1, characterized in that, Also includes: Federated training tasks are generated based on the data modality configuration of the research nodes participating in the iterative training. When multiple research nodes have the same data modality, a federated training task with a unified model structure is issued.

6. The method according to claim 1, characterized in that, The weights of the weighted aggregation are also determined based on at least one of the following factors: the number of local samples, label completeness, modality completeness, node historical stability, training loss variation, and node contribution score.

7. A multimodal AI data governance and federated training support system for regional medical research collaboration, characterized in that, This includes a regional collaborative management platform and multiple research nodes; The regional collaborative management platform is used to communicate with research user terminals, management terminals, and multiple research nodes; each research node is used to communicate with the internal business system of the institution. The system includes: The node access authentication module is used to receive access requests from research nodes and to perform authentication management on the research nodes. The data acquisition and processing module is used to acquire medical research data from research nodes, process the research data to form local data resources, wherein the processing of research data includes: establishing a unified patient index for data from different business systems and different data modalities, and completing multimodal sample alignment based on patient identification, medical events, time windows and sample source relationships, and the processing of research data also includes performing quality scoring on the data. The project application authorization module is used to respond to research project applications and generate corresponding project application authorization strategies; The dataset construction and local training module is used to construct a project-level training dataset from the local data resources based on the project authorization strategy, perform model training in a local controlled environment, and generate local model update information. The parameter upload, federated aggregation, and distribution module is used to upload the local model update information through research nodes, perform federated aggregation on the local model update information to generate global model parameters or shared model parameters, and distribute them to research nodes for iterative training. The federated aggregation of the local model update information includes weighted aggregation of local model update information uploaded by multiple research nodes, where the weights are determined based on the data quality score. Furthermore, when multiple research nodes have different data modalities, a federated training task containing shared backbone parameters and modality-specific branch parameters is distributed. During federated aggregation, only the shared backbone parameters are aggregated, while the modality-specific branch parameters are retained on the local node or aggregated in groups among research nodes with the same data modality. The results output module is used to output the global model parameters or shared model parameters that have reached the preset conditions after iterative training.

8. The system according to claim 7, characterized in that, The parameter uploading, federated aggregation, and distribution module is configured to perform federated training based on the data modality configuration of the research nodes participating in iterative training; When multiple research nodes have the same data modality, the corresponding local model update information is uniformly aggregated. The parameter upload, federated aggregation, and distribution module performs weighted aggregation of local model update information uploaded by multiple research nodes. The weight of the weighted aggregation is determined based on at least one of the following factors: number of local samples, label completeness, modality completeness, historical stability of nodes, changes in training loss, and node contribution score. The result output module is used to output one or more of the following: global model parameters, shared model parameters, model files, evaluation reports, experimental records, or desensitized statistical results.

Images