Cross-facility model training method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business system platforms such as medical health and financial science and technology, and discloses a cross-facility model training method, which comprises the following steps: acquiring original data of each facility, storing the original data in a local storage system, and taking the original data as a training sample; performing analysis processing on the parameterized template, and extracting workflow parameters and data features; according to the workflow parameters and / or the data features, determining a training sample distribution strategy of each facility, and determining a use rule of training samples of each facility through the training sample distribution strategy; and training the facilities based on the initial model parameters, the training samples and the use rules to generate a global model. According to the method, data distribution is configured through a parameterized workflow template, each facility independently trains a model based on original data, and a federated average algorithm is utilized to aggregate parameters to generate a global model. And the targets of privacy protection and efficient training are achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a cross-facility model training method, device, equipment and storage medium. Background Art

[0002] Traditional federated learning frameworks (such as FedAvg) were not designed for heterogeneous facilities like supercomputing centers and cloud computing, and lack cross-regional resource scheduling capabilities. Hybrid workflow technologies are primarily used in scientific computing (such as bioinformatics) but, when combined with deep learning, lack adaptability for dynamic model training. Distributed training of large-scale models (such as LLM) relies on data or model parallelism, but communication latency can easily become a performance bottleneck in cross-facility scenarios. A single data center has limited resources, such as the number of compute nodes and GPU memory. Training trillion-parameter models requires complex model parallel partitioning and poses a single point of failure risk. Traditional distributed training (such as data parallelism) requires frequent gradient synchronization, requiring massive cross-facility data transfer (e.g., hundreds of GB per gradient update round), while public network bandwidth is limited (for example, in experiments, transferring models across European supercomputing centers took minutes). Hardware architectures (e.g., NVIDIA A100 vs. AMD EPYC) and software stacks (e.g., PyTorch versions, SLURM scheduling system) vary significantly between facilities, making manual adaptation costly and difficult to reuse. Training large models requires integrating data from multiple institutions (such as multilingual corpora and cross-industry datasets), but centralized training models no longer meet efficiency and privacy requirements. Public supercomputing centers need to allocate resources fairly to prevent a single task from occupying all nodes for a long time. Cross-facility collaboration can improve resource utilization through load balancing.

[0003] Universities and small and medium-sized enterprises face difficulties in obtaining or monopolizing large-scale computing resources (such as supercomputing centers). Meanwhile, training models with hundreds of billions of parameters (such as GPT-4 and LLaMAv2) in industry requires tens of thousands of GPU hours, which is costly and limited by the resource capacity of a single data center. Furthermore, in sensitive fields (such as healthcare and finance), data cannot be stored centrally, making traditional centralized training models difficult to meet privacy protection requirements. While federated learning can address the issue of data immobility, it faces challenges in cross-facility collaboration, such as high communication overhead and difficulty adapting to heterogeneous environments.

[0004] In the financial sector, financial data contains a large amount of sensitive information, such as customer identities, transaction records, and asset status. Traditional centralized training models struggle to meet privacy protection requirements because data cannot be centrally stored. While federated learning can address data immobility, cross-facility collaboration faces challenges such as high communication overhead and difficulty adapting to heterogeneous environments, making it difficult to balance privacy protection and data security. Furthermore, the hardware architectures and software stacks of different financial institutions vary significantly, making manual adaptation costly and difficult to reuse, further complicating data sharing and collaborative training. Financial institutions need to integrate data from multiple institutions (such as multilingual corpora and cross-industry datasets) to improve model generalization and accuracy, but centralized training models no longer meet efficiency and privacy requirements. Cross-facility collaboration can improve resource utilization through load balancing, but traditional federated learning frameworks (such as FedAvg) are not designed for heterogeneous facilities such as supercomputing centers and the cloud, and lack cross-regional resource scheduling capabilities. Traditional distributed training (e.g., data parallelism) requires frequent gradient synchronization, requiring massive cross-facility data transfer (e.g., hundreds of GB per gradient update round). Limited public network bandwidth leads to inefficient cross-facility collaboration.

[0005] In the healthcare sector, medical data contains highly sensitive information such as patient personal information, medical histories, and diagnostic results. Traditional centralized training models also struggle to meet privacy protection requirements because data cannot be centrally stored. While federated learning can address data immobility, it faces challenges such as high communication overhead and difficulty adapting to heterogeneous environments when collaborating across facilities, making it difficult to balance privacy protection and data security. Hardware architectures and software stacks vary significantly across medical institutions, making manual adaptation costly and difficult to reuse, further complicating data sharing and collaborative training. Model training in the healthcare sector (such as medical image analysis and disease prediction models) requires processing large amounts of data and complex models, but universities and small and medium-sized enterprises face difficulties in accessing or monopolizing large-scale computing resources (such as supercomputing centers). Training a model with hundreds of billions of parameters requires tens of thousands of GPU hours, which is costly and limited by the resource capacity of a single data center. A single data center has limited resources such as the number of compute nodes and GPU memory. Training a model with trillions of parameters requires complex parallel model segmentation, which poses a single point of failure and compromises the stability and reliability of healthcare services. Summary of the Invention

[0006] The main purpose of the present invention is to provide a cross-facility model training method, device, equipment and storage medium, aiming to solve the problems of resource limitations, insufficient privacy protection and high communication overhead faced by cross-facility large-scale model training in the existing technology.

[0007] To achieve the above objectives, the present invention provides a cross-facility model training method, comprising:

[0008] Obtaining raw data of each facility, storing the raw data in a local storage system, and using the raw data as training samples;

[0009] Parse the parameterized template to extract workflow parameters and data features;

[0010] Determining a training sample distribution strategy for each facility based on the workflow parameters and / or data characteristics, and determining usage rules for the training samples of each facility based on the training sample distribution strategy;

[0011] Each facility is trained based on the initial model parameters, training samples, and usage rules to generate a global model.

[0012] Furthermore, to achieve the above-mentioned purpose, the present invention provides a cross-facility model training device, comprising:

[0013] A training sample module, configured to obtain raw data of each facility, store the raw data in a local storage system, and use the raw data as training samples;

[0014] Template parsing module, used to parse parameterized templates and extract workflow parameters and data features;

[0015] a distribution strategy module, configured to determine a training sample distribution strategy for each facility based on the workflow parameters and / or data characteristics, and determine usage rules for the training samples of each facility based on the training sample distribution strategy;

[0016] The model training module is used to train each facility based on initial model parameters, training samples and usage rules to generate a global model.

[0017] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a cross-facility model training program stored in the memory and runnable on the processor, and when the cross-facility model training program is executed by the processor, the steps of the cross-facility model training method described above are implemented.

[0018] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a cross-facility model training program is stored, and when the cross-facility model training program is executed by a processor, the steps of the cross-facility model training method as described above are implemented.

[0019] Beneficial effects: The present invention relates to the field of artificial intelligence technology and can be applied to business system platforms such as communications, healthcare, and financial technology. A cross-facility model training method is disclosed, including: obtaining the original data of each facility, storing the original data in a local storage system, and using the original data as training samples; parsing and processing the parameterized template to extract workflow parameters and data features; determining the training sample distribution strategy for each facility based on the workflow parameters and / or data features, and determining the usage rules of the training samples of each facility through the training sample distribution strategy; training each facility based on the initial model parameters, training samples, and usage rules to generate a global model. The present invention configures data distribution through parameterized workflow templates, and each facility independently trains a model based on local data, and uses a federated averaging algorithm to aggregate parameters to generate a global model. It reduces communication overhead, protects data privacy, improves resource utilization, supports distributed training of large language models with hundreds of billions of parameters, and meets privacy compliance requirements in sensitive fields such as healthcare and finance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0021] Figure 1 A schematic diagram of an application environment of a cross-facility model training method according to an embodiment of the present invention;

[0022] Figure 2 This is a flow chart of an embodiment of a cross-facility model training method of the present invention;

[0023] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the cross-facility model training device of the present invention;

[0024] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0025] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] The cross-facility model training method provided by the embodiment of the present invention can be applied in Figure 1In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can obtain the raw data of each facility through the user terminal, store the raw data in a local storage system, and use the raw data as a training sample; parse and process the parameterized template to extract workflow parameters and data features; determine the training sample distribution strategy of each facility according to the workflow parameters and / or data features, and determine the usage rules of the training samples of each facility through the training sample distribution strategy; train each facility based on the initial model parameters, training samples and usage rules to generate a global model. The present invention combines a hybrid workflow management system (WMS) with federated learning to achieve collaborative training across multiple independent computing facilities (such as supercomputing centers, cloud platforms, and edge nodes). The core technologies include cross-facility model parameter aggregation, resource scheduling and communication optimization, and a distributed training architecture for large language models (LLMs) with hundreds of billions of parameters. The method achieves privacy protection, communication overhead optimization, and heterogeneous environment adaptation through parameterized workflow templates, a combination of model parallelism and data parallelism, and containerization technology, significantly improving training efficiency and resource utilization. The user end may be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server end may be implemented as an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below using specific embodiments.

[0028] See also Figure 2 , Figure 2 This is a flowchart of an embodiment of the cross-facility model training method provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here.

[0029] like Figure 2 As shown, the cross-facility model training method proposed in the present invention includes the following steps:

[0030] S100, obtaining raw data of each facility, storing the raw data in a local storage system, and using the raw data as training samples;

[0031] S200, parsing the parameterized template to extract workflow parameters and data features;

[0032] S300, determining a training sample distribution strategy for each facility based on the workflow parameters and / or data features, and determining usage rules for the training samples of each facility based on the training sample distribution strategy;

[0033] S400: Train each facility based on initial model parameters, training samples, and usage rules to generate a global model.

[0034] In this embodiment, each participating facility (e.g., supercomputing center, cloud platform) stores raw data locally and does not transfer it across facilities to meet privacy compliance requirements, such as HIPAA compliance for medical data. Data is preprocessed locally, including operations such as cleaning, tokenization (for text data), and normalization (for numeric data). For example, Italian and Czech language corpora are stored in different facilities, and preprocessing is used to generate samples suitable for model training, such as the 2048-token sequences required by LLaMAv2. The data distribution strategy is configured using parameterized CWL templates, supporting non-independent and identically distributed (Non-IID) scenarios. For example, the data ratio for each facility can be specified through workflow parameters (e.g., facility A is responsible for 60% of the samples, facility B for 40%), or automatically distributed based on data characteristics (e.g., language, modality). The central server distributes initial model parameters (e.g., weights for LLaMAv2 7B) to each facility via a workflow management system (e.g., StreamFlow). Each facility independently trains models based on local data and uses fully sharded data parallelism (FSDP) technology to split model layers within a single facility (such as distributing the Transformer encoder layer to different GPU nodes) to reduce memory usage.

[0035] After each round of training, each facility uploads the model parameters to a central node (such as the Ada cloud server). The central node aggregates and generates a global model using the Federated Averaging Algorithm (FedAvg). Only the model weights (e.g., 13GB) are transmitted, encrypted using the SSH or S3 protocol, and the single transmission time is controlled within hundreds of seconds (an average of 217-242 seconds in the dual-facility experiment). The `cwltool:Loop` extension is used to automate training rounds and set convergence conditions (such as ROUGE score requirements or maximum round limits). The load of each facility (such as GPU utilization) is monitored in real time, and training tasks for certain facilities are automatically skipped or repeated based on the facility load to balance the computing pressure across facilities. For example, if the GPU utilization of a facility is too high, its training task load can be reduced.

[0036] The aggregated global model can be deployed on a single high-performance node (such as a central cloud server) for tasks such as text generation and question answering. At the same time, it supports the direct use of local models for inference in edge devices or distributed facilities to reduce the load on central nodes. For example, medical edge devices can analyze imaging data in real time. Through containerization technology (such as Singularity), the trained model and dependent environment are encapsulated and quickly migrated to different hardware platforms (such as migrating from NVIDIA GPU to AMD GPU). For example, the trained LLaMAv2 model is deployed on the local server of a financial institution for compliant customer consultation dialogue generation to avoid data transmission.

[0037] When new data is added, each facility performs incremental fine-tuning based on the latest global model and only uploads the updated parameter differences (such as LoRA adapter weights) to reduce communication volume. For example, the news domain model can regularly access new corpus and update it through cross-facility federated learning to maintain timeliness. Use StreamFlow's built-in monitoring tools to track cross-facility training indicators (such as time per round, parameter transmission delay, and model convergence curve). If the training speed of a facility lags significantly, automatically adjust its data allocation for the next round. Introduce model compression (such as quantization) to reduce the amount of transmitted data, or switch to an asynchronous aggregation strategy to alleviate synchronization bottlenecks.

[0038] New nodes (such as new edge computing clusters) are quickly configured through CWL templates to expand the scale of the federated learning network. The new facility automatically adapts the training framework (such as PyTorch version consistency check) through the containerized environment to ensure the stability of training across heterogeneous environments. The present invention realizes large-scale distributed training of models across multiple independent computing facilities (such as supercomputing centers, cloud platforms, and edge nodes) through the combination of a hybrid workflow management system and federated learning. This method is suitable for multi-language, multi-modal, or non-independent and identically distributed data scenarios, and can make full use of the computing resources of each facility while meeting the requirements of privacy protection and computing efficiency.

[0039] For example, in the healthcare business, medical data is often scattered across different hospitals, clinics, and research institutions, and involves patient privacy, making it difficult to centrally process. Federated learning allows for model training to be conducted using data from multiple medical institutions without sharing the original data. Each medical institution first pre-processes local medical imaging (such as X-rays, CT scans) and electronic medical records (EMRs), and then uses a local model for training. Through the federated learning framework, the model parameters of each institution are aggregated to a central node to generate a global model. Ultimately, this model can be used for early prediction and diagnosis of diseases, such as cancer screening and cardiovascular disease risk assessment.

[0040] In the fintech sector, financial institutions need to assess their customers' credit risk. However, customer data is private and dispersed across various banks, financial institutions, and third-party data providers. Federated learning technology enables distributed training of customer transaction records, credit histories, and financial data across various financial institutions. Through a hybrid workflow management system, tasks such as data preprocessing, feature extraction, and model training are broken down into multiple subtasks and distributed across different computing facilities (such as cloud platforms and edge nodes). The resulting global model can be used for accurate risk assessment and credit scoring, helping financial institutions better manage credit risk.

[0041] In one embodiment, the step S100 includes:

[0042] S101, obtaining original data of each facility and encrypting the original data;

[0043] S102, storing the encrypted original data in a local storage system;

[0044] S103: Preprocess the original data stored in the local storage system to generate training samples.

[0045] In this embodiment, each participating facility (e.g., supercomputing center, cloud platform, edge node, etc.) possesses its own raw data, which may come from diverse fields, such as medical data, financial transaction records, and language corpora. For example, Italian and Czech language corpora are stored in different supercomputing centers (Leonardo@CINECA and Karolina@IT4I), respectively. Because this data may involve sensitive information (e.g., medical data must comply with HIPAA), it must be stored locally and cannot be transferred across facilities to meet privacy protection requirements. Data acquisition can be pre-stored in the facility's local storage system or imported securely from external data sources. For example, patient data at a medical institution may be stored on a local hospital server, while financial data may come from a financial institution's internal database. Raw data may exist in a variety of formats, such as text files, database records, image files, and video files. The data format and structure must be determined based on the specific application scenario.

[0046] To further enhance data security and prevent unauthorized access during local storage or processing, raw data must be encrypted. Symmetric encryption (such as AES) or asymmetric encryption (such as RSA) can be used. Symmetric encryption is fast and suitable for processing large amounts of data, but requires secure key management. Asymmetric encryption offers high security but high computational overhead, making it suitable for key exchange or encrypting small amounts of data. Furthermore, homomorphic encryption allows certain types of computation to be performed on encrypted data without first decrypting it, offering privacy advantages. However, the technical maturity and performance still require improvement. Encryption keys must be properly managed to ensure access only by authorized users and systems. A key management system (K)MS can be used to store and manage keys, ensuring that key distribution and usage comply with security standards. During the encryption process, block encryption can be used to split large files into multiple smaller blocks for separate encryption to improve efficiency. Furthermore, metadata about the encrypted data, such as the encryption algorithm, key identifier, and block size, must be recorded for subsequent decryption and processing. While encrypting data, a summary of the data (such as a SHA-256 hash value) can also be generated to verify the integrity of the data during storage and transmission to prevent data tampering.

[0047] The encrypted raw data needs to be stored in a local storage system. Each facility can choose an appropriate local storage system based on its needs, such as a distributed file system (such as HDFS), an object storage system (such as S3), or a local disk array. Distributed file systems (such as Hadoop Distributed File System HDFS and Ceph) are suitable for large-scale data storage and offer high availability and fault tolerance. Object storage systems (such as Amazon S3 and Alibaba Cloud OSS) support high concurrent access and are suitable for storing unstructured data. For facilities with smaller data volumes, local disk storage can be used directly, but data backup and disaster recovery measures must be considered. Encrypted data can be stored as files or as records in a database. Data metadata (such as data source, encryption algorithm, and key identifier) ​​must be recorded during storage to facilitate subsequent decryption and processing. To prevent data loss, it is recommended to back up encrypted data and implement redundant storage in the local storage system, such as through RAID technology or distributed storage replication. Furthermore, strict access permissions must be set to ensure that only authorized users and systems can access encrypted data. Role-based access control (RBAC) or attribute-based access control (ABAC) policies can be used. Additionally, regularly back up encrypted data to prevent data loss. Backup data also needs to be encrypted to ensure data security. For the storage system itself, you can enable storage encryption features, such as HDFS's transparent encryption or S3's server-side encryption, to provide additional data security.

[0048] Before preprocessing the raw data stored in the local storage system to generate training samples, the encrypted data must be decrypted using the corresponding key to restore the original data. The preprocessing process includes data cleansing, data standardization, and data labeling. Data cleansing removes noise, duplicate records, and missing values ​​to ensure data quality. Data standardization converts data to a uniform format and range, such as tokenizing text data, removing stop words, and normalizing numerical data. Data labeling labels data for tasks requiring supervised learning, such as classifying and labeling images in image recognition. Based on model training requirements, the preprocessed data is divided into training samples. For example, in language model training, text data can be divided into fixed-length sequences (such as 2048-token sequences) to generate corresponding input and label pairs. Finally, the generated training samples are stored in the local storage system for subsequent model training. Training samples can be stored in files in specific formats (such as TFRecord and Parquet) for efficient reading and loading.

[0049] For example, in the healthcare sector, raw patient data, such as medical records, test results, and genetic data, is often dispersed across different medical institutions and involves highly sensitive personal information. First, each medical institution obtains the raw data locally and encrypts it using advanced encryption technologies (such as homomorphic encryption or zero-knowledge proofs). This ensures that even if the data is intercepted during transmission or storage, it cannot be deciphered. The encrypted data is stored in a local storage system, ensuring that it remains within the medical institution's control and complies with privacy regulations such as HIPAA. The medical institution then preprocesses the stored encrypted data to generate training samples. This preprocessing may include data cleansing (removing erroneous or duplicate records), standardization (unifying the data format), and feature extraction (e.g., extracting key symptoms and diagnostic results from medical records). These processes are performed locally, ensuring data privacy. The resulting training samples can be used to develop medical diagnostic models, disease prediction models, or personalized treatment plans, thereby improving the quality and efficiency of healthcare services while protecting patient privacy.

[0050] In the field of FinTech, raw data such as customer identity information, transaction records, and credit scores also need to be strictly protected. Financial institutions first obtain local raw data and then process it using encryption technology. The encrypted data is stored on local servers or secure cloud storage systems to ensure data security and compliance, in line with the requirements of privacy regulations such as GDPR. Next, financial institutions preprocess the encrypted data to generate samples suitable for machine learning model training. Preprocessing may include data denoising (removing outliers), data normalization (adjusting values ​​to a uniform range), and feature engineering (such as extracting consumption patterns and risk indicators from transaction records). These processed training samples can be used to develop credit scoring models, risk assessment models, or fraud detection systems to help financial institutions more accurately assess customer credit, manage financial risks, and improve operational efficiency, while ensuring the security and privacy of customer data.

[0051] In one embodiment, step S200 includes:

[0052] S201: Acquire a parameterized template, analyze the parameterized template, and obtain a template path;

[0053] S202: The parsing engine verifies whether the parameterized template is legal based on the template path;

[0054] S203: If the parameterized template is legal, the parsing engine parses the parameterized template to obtain workflow parameters and data features.

[0055] In this embodiment, parameterized templates (such as CWL templates) are pre-defined according to specific training task requirements and are usually created by system administrators or data scientists. These templates can be stored in a template library on a central server or obtained from an external file system, such as local storage, a distributed file system, or cloud storage. The format of the template is usually YAML or JSON, which are not only easy to read and write, but also support complex data structures. Users can manually select and load template files through a graphical interface or command line tool, and the system automatically loads the corresponding template based on the preset configuration file or task requirements. In addition, the system supports version management of templates to ensure that the template loaded each time is the latest version, thereby avoiding problems caused by inconsistent template versions.

[0056] The template path refers to the specific location of the template file in the storage system. It can be a local file path, a network path (such as an HTTP / HTTPS URL), a distributed file system path (such as an HDFS path), or a cloud storage path (such as an S3 path). The template path is parsed through the file system interface or network protocol, and the correct format of the path is verified, such as whether it conforms to the URI specification. The standard library provided by the programming language (such as Python's os.path or urllib.parse) can be used to parse the path and check whether the path exists and whether the file is accessible. For example, for a local path, you can use os.path.exists to check whether the file exists; for a network path, you can verify the validity of the path through an HTTP request.

[0057] Before loading a template file, you need to verify the integrity and format of the template file. You can verify the integrity of the file by checking whether the file size is as expected or by checking whether the file content has been tampered with through a hash value. At the same time, you need to verify whether the format of the template file complies with the YAML or JSON specification. You can use a parsing library (such as Python's yaml.safe_load() or json.loads()) to parse the file content and capture format errors. In addition, you also need to check whether the parameters and structures in the template file conform to the predefined pattern, such as verifying whether the necessary fields (such as model type, facility list, data distribution strategy, etc.) are included, and whether the field values ​​are within the legal range. You can use pattern matching tools (such as JSONSchema or YAML Schema) to define the legal structure of the template and use these tools to verify whether the template conforms to the pattern.

[0058] After verification, use a parsing library (such as yaml.safe_load() or json.loads()) to load the template file's contents. Extract workflow-related parameters from the template, such as the facility list, model type, training rounds, and data distribution strategy. These parameters will be used for subsequent workflow orchestration and task scheduling. Also, extract data-related features from the template, such as the data distribution strategy (such as the Dirichlet distribution), data ratio, and data preprocessing steps. These features will be used for data distribution and preprocessing. Encapsulate the extracted parameters into workflow parameter objects, and encapsulate the extracted feature data into data object features.

[0059] Finally, the generated workflow parameters and data features need to be verified. Verify that the workflow parameters are complete and logical, for example, by checking whether the facility list is empty and whether the number of training rounds is a positive integer. Also, verify that the data features are reasonable, for example, by checking whether the data distribution ratio is within the range [0, 1] and whether the data preprocessing steps are supported. These verification steps ensure the correctness and usability of the template file, providing a reliable foundation for subsequent training tasks.

[0060] For example, in the healthcare sector, parameterized templates can be used to define and manage complex medical data processing and model training processes. First, a medical institution obtains a predefined parameterized template, which may include parameters such as data distribution strategy, model type, training rounds, and convergence criteria. For example, a template might specify the use of specific medical imaging data (such as CT scans) for training a disease diagnosis model, and set the convergence criteria to 90% model accuracy. A parsing engine analyzes the parameterized template to determine the template path—the specific path to each task and parameter defined in the template. The parsing engine verifies the validity of the template path, ensuring that the parameters and task configurations in the template meet expectations. For example, it checks whether the data path is correct and whether the model type is supported. If the template is valid, the parsing engine parses the parameters and data features in the template to generate specific workflow parameters and data features. These parameters and features are used to guide subsequent steps such as data preprocessing, model training, and parameter aggregation. For example, the training process of a medical imaging diagnostic model may involve multiple facilities, each responsible for processing different types of imaging data. Parameterized templates allow for flexible configuration of each facility's tasks, such as data preprocessing methods and model initialization parameters, enabling efficient distributed training. Parameterized templates can also define how data features are extracted, such as extracting key features from imaging data for model training, ensuring the model can accurately identify disease characteristics.

[0061] Parameterized templates also play a vital role in the FinTech sector. Financial institutions can use them to define and manage financial data processing and model training processes. For example, a template might specify the use of customer transaction records and credit score data to train a credit risk assessment model, with the convergence condition set at 95% prediction accuracy. The parsing engine analyzes the parameterized template, determines the template path, and verifies its validity. This includes checking whether the data path is correct, the model type is supported, and whether the number of training rounds is reasonable. If the template is valid, the parsing engine parses the parameters and data features in the template to generate workflow parameters and data features. These parameters and features are used to guide subsequent steps such as data preprocessing, model training, and parameter aggregation. For example, in training a credit risk assessment model, a parameterized template can define how to extract key features from transaction records, such as transaction frequency and transaction amount, and use these features in model training. The template can also define model initialization parameters and training rounds to ensure efficient and accurate model training. Through parameterized templates, financial institutions can flexibly configure the tasks of different facilities, such as data preprocessing methods and model initialization parameters, thereby achieving efficient distributed training.

[0062] In one embodiment, step S300 includes:

[0063] S3011. Predefine workflow parameters and store the workflow parameters in a configuration file;

[0064] S3012. When the workflow is running, the workflow parameters of the configuration file are read by the workflow runner;

[0065] S3013, parsing the workflow parameters to obtain usage rules;

[0066] S3014, dividing the training sample into multiple data sample subsets according to the usage rule;

[0067] S3015. Distribute the data sample subsets to each facility according to the data ratio of the usage rules, and store each subset in the local storage system of each facility.

[0068] In this embodiment, workflow parameters are defined by parameterizing the Common Workflow Language (CWL) template. These parameters include a facility list, i.e., the identification information of each computing facility involved in the training (such as a supercomputing center, a cloud platform, etc.); a data distribution strategy, such as the data distribution ratio (e.g., facility A is responsible for 60% of the samples, facility B is responsible for 40%) or a distribution rule based on data features (such as language, modality); a model type, which specifies the type of large language model (LLM) to be trained, such as LLaMAv2 7B; training rounds, i.e., the predefined number of training iterations or convergence conditions; and a communication protocol, which is a protocol for transmitting model parameters across facilities, such as SSH or S3. These parameters are stored in a configuration file, usually a file in YAML or JSON format.

[0069] When the workflow runs, the workflow runner (such as StreamFlow or cwltool) loads the above configuration file. The workflow runner is a tool used to parse and execute the tasks defined in the workflow template. It parses the parameters in the configuration file and converts them into specific usage rules. For example, the data_ratio parameter determines the data ratio to be allocated to each facility; the rounds and convergence_metric parameters determine the number of training iterations and convergence conditions; and the protocol and encryption parameters determine the method and security of cross-facility communication.

[0070] Based on the parsed usage rules, the workflow runner will split the training samples into multiple data sample subsets. It reads the preprocessed training samples (such as cleaned and segmented data) from the local storage system and splits the training samples into multiple subsets according to the data distribution ratio defined in the configuration file (such as 60% and 40%). For example, if there are 1,000 samples in total, facility A will get 600 samples and facility B will get 400 samples. If the configuration file defines allocation rules based on data features (such as language, modality), the workflow runner will classify the data according to these features and distribute it to each facility according to the rules.

[0071] After the allocation is completed, the workflow runner will transfer each data sample subset to the corresponding facility and store it in its local storage system. It uses the communication protocol specified in the configuration file (such as SSH or S3) to transfer the data sample subset to each facility, and encryption will be performed during the transmission process to ensure the security of the data. After each facility receives the data sample subset, it stores it in the local storage system, and this data will be used for subsequent local model training. Since the data is always stored locally, the original data will not be transferred across facilities, thus meeting privacy compliance requirements (such as GDPR, HIPAA, etc.).

[0072] Through the above steps, efficient segmentation and distribution of training samples across facilities are achieved while ensuring privacy protection and communication efficiency.

[0073] For example, in the healthcare sector, data privacy and security are crucial, while efficient utilization of data resources distributed across different medical institutions is crucial. By predefining workflow parameters and storing them in a configuration file, key parameters such as data allocation strategies, model types, and training rounds can be flexibly defined. For example, data allocation ratios can be specified for different medical institutions (such as hospitals and clinics), or non-independent and identically distributed (Non-IID) data allocation can be performed based on data characteristics (such as disease type and patient age). During workflow execution, the workflow runner reads the workflow parameters in the configuration file and parses them to generate specific usage rules. These rules guide how to split training samples into multiple data sample subsets and distribute them to each facility based on data ratios. For example, a large hospital may be responsible for processing 60% of the samples, while smaller clinics may handle the remaining 40%. These data sample subsets are stored in each facility's local storage system, ensuring that data does not leave the control of the medical institution and complying with privacy regulations such as HIPAA.

[0074] In the fintech sector, customer data also raises privacy and security concerns, and financial institutions need to efficiently utilize data resources dispersed across various departments or branches. Pre-defining workflow parameters and storing them in configuration files allows for flexible definition of parameters such as data allocation strategies, model types, and training rounds. For example, data allocation ratios can be specified for different financial institutions (e.g., banks, fintech companies), or non-independent and identically distributed data can be allocated based on data characteristics (e.g., customer type, transaction amount).

[0075] The workflow runner reads the workflow parameters in the configuration file and parses them to derive specific usage rules. These rules guide how to split the training sample into multiple data sample subsets and distribute them to each facility based on the data ratio. For example, a large bank might handle the majority of high-value customer transaction data, while smaller financial institutions handle the remaining data. These data sample subsets are stored in the local storage system of each facility, ensuring that the data does not leave the financial institution's control and comply with privacy regulations such as GDPR.

[0076] In one embodiment, step S300 includes:

[0077] S3021. Analyze the data features to obtain key features and feature mapping relationships;

[0078] S3022. Extracting a feature vector of global data based on the key features and feature mapping relationship;

[0079] S3023. Determine the data type of the training sample using the feature vector, and determine a usage rule based on the data type;

[0080] S3024. Distribute the training samples to each facility according to the usage rules.

[0081] In this embodiment, the parsing and processing of data features is a key step in achieving efficient data distribution and privacy protection in cross-facility large-scale model training. The raw data is preprocessed locally in each facility, including data cleaning, word segmentation (for text data), normalization (for numerical data) and other operations. For example, in language model training, text data needs to be segmented into words or subword units (such as BPE encoding). By analyzing the preprocessed data, key features are extracted. These features may include: language type (for multilingual corpora, language is one of the key features, for example, Italian and Czech data need to be processed separately); data modality (data may be different modalities such as text, image or audio. For multimodal learning, modality type is an important feature); data label (for supervised learning tasks, data label is a key feature, such as disease classification label of medical imaging data); and data source (data may come from different data sets or institutions, and source information is also one of the features).

[0082] Next, a feature mapping relationship is established to map the extracted key features to the feature vectors of the global data. For example, for a multilingual corpus, each language can be mapped to a feature vector. The key features of each data sample are converted into a feature vector. For example, the feature vector of an Italian text sample can be represented as [1,0,0]. In cross-facility training, the feature vectors of the global data are used to guide data allocation and model training. The key features extracted from each facility are aggregated to the central node to form a feature set of the global data. Based on the feature mapping relationship, each sample of the global data is converted into a feature vector. For example, if the global data contains Italian and Czech samples, the feature vector of each sample will be encoded according to its language type.

[0083] The feature vectors of global data are stored in a central node or distributed storage system for subsequent training sample allocation and model training. The feature vectors are used to determine the data type of the training samples, and usage rules are established based on the data type. For example, the feature vector [1,0,0] represents an Italian text sample, and [0,1,0] represents a Czech text sample. Training samples are classified according to data type, for example, all Italian samples are grouped into one category and Czech samples into another. Usage rules are established based on data type. For example, if a facility excels at processing Italian data, most Italian samples can be allocated to that facility; if a facility's hardware is more suitable for processing image data, image samples can be allocated to that facility. For data that is not independent and identically distributed, more complex allocation rules can be established based on data characteristics (such as language mode and state). For example, data in different languages ​​can be allocated to different facilities in proportion to simulate real-world data distribution.

[0084] Based on the established usage rules, training samples are allocated to each facility and stored in its local storage system. For example, based on language type, Italian samples are allocated to the Leonardo@CINECA facility, and Czech samples are allocated to the Karolina@IT4I facility. If the load on a facility is too high or too low, the allocation rules can be dynamically adjusted based on real-time monitoring data to balance the computing pressure across facilities. The training samples allocated to each facility are stored in its local storage system, and this data will be used for subsequent local model training. Since the data is always stored locally, the original data will not be transferred across facilities, thus meeting privacy compliance requirements (such as GDPR, HIPAA, etc.).

[0085] Through the above steps, this patent achieves efficient training sample allocation based on data features. By analyzing data features, extracting feature vectors, determining data types, and establishing usage rules, training samples can be flexibly allocated to various facilities while meeting the requirements of privacy protection and computational efficiency. This method is particularly suitable for multilingual, multimodal, or non-independent and identically distributed data scenarios, fully leveraging the strengths of each facility to improve the overall performance of cross-facility training.

[0086] For example, in the healthcare sector, parsing and processing data features is crucial for improving model accuracy and generalization. First, medical institutions obtain raw patient data, such as electronic medical records (EMRs), medical images, and gene sequencing data. By parsing this data, key features are extracted. These key features are converted into feature vectors using feature mapping relationships. For example, disease types can be mapped to [1, 0, 0] (cancer) and [0, 1, 0] (cardiovascular disease). The feature vectors can be used to determine the data type of training samples. For example, all cancer-related samples can be grouped into one category, and cardiovascular disease-related samples into another. Based on the data type, medical institutions can establish usage rules to allocate training samples to different facilities. For example, large hospitals may be responsible for processing complex cancer case data because they have more advanced imaging equipment and specialized oncologists. Community clinics may be responsible for processing samples of common diseases (such as cardiovascular disease) because the data volume is large and relatively simple.

[0087] In the field of FinTech, the analysis and processing of data features is equally important. Financial institutions obtain raw customer data, such as transaction records, credit scores, and income levels. By analyzing and processing this data, key features are extracted. Key features are converted into feature vectors through feature mapping relationships. For example, customer type can be mapped to [1,0] (individual customer) and [0,1] (corporate customer). The feature vector can be used to determine the data type of the training sample. For example, all transaction data of individual customers can be classified into one category, and transaction data of corporate customers into another category. Based on the data type, financial institutions can establish usage rules and allocate training samples to different facilities. For example, large banks may be responsible for processing the transaction data of high-value corporate customers because they have more robust risk assessment systems. Small financial institutions may be responsible for processing the transaction data of individual customers because the data volume is large and relatively simple.

[0088] In one embodiment, step S400 includes:

[0089] S4011, the central server transmits the initial model parameters to each facility through the workflow management system;

[0090] S4012. Distribute the model to each node in each facility using fully sharded data parallel technology.

[0091] S4013. Each facility performs model training based on the training samples, initial model parameters, usage rules, and the nodes;

[0092] S4014. Obtain model parameters of each facility after each round of training, and transmit the model parameters to the central server;

[0093] S4015. Aggregate the model parameters of each facility by using a federated averaging algorithm to generate a global model.

[0094] In this embodiment, the first step in cross-facility large-scale model training is the transmission of initial model parameters, ensuring that each facility begins training from the same starting point. A central server, acting as the global coordination node, is responsible for storing and distributing the initial model parameters and communicating with each facility through a workflow management system (WMS). The WMS orchestrates tasks, breaking down training tasks into multiple subtasks and assigning them to each facility. It also monitors the training progress of each facility and dynamically adjusts task allocation.

[0095] The central server first loads the pre-trained initial model parameters, such as the weights for LLaMAv2 7B. The model parameters are then securely transmitted to each facility using an encrypted communication protocol (such as SSH or S3) to ensure data security. After receiving the initial model parameters, each facility sends a confirmation message to the central server, indicating that the parameters have been successfully received and is ready to begin local training.

[0096] After receiving the initial model parameters, each facility needs to perform model training locally. In order to efficiently utilize computing resources, fully sharded data parallelism (FSDP) technology is used to assign various layers of the model (such as the Transformer encoder layer) to different computing nodes (such as GPUs). For example, a facility may have multiple GPU nodes, each of which is responsible for a part of the model. Each node processes the subset of data assigned to it locally, accelerating the training process through parallel computing. Through sharding technology, each node only stores a part of the model's weights, thereby reducing the memory requirements of a single node. Based on the hardware configuration of the facility (such as the number and type of GPUs), the various parts of the model are assigned to different nodes, and a scheduling system (such as SLURM) is used to manage node resources to ensure that each node runs efficiently.

[0097] Within the local facility, model training is performed based on the assigned data samples and initial model parameters. The training samples assigned to the facility are read from the local storage system, and the data is preprocessed as necessary, such as normalization and word segmentation. Each facility uses the data samples and initial model parameters assigned to it for local training. During the training process, the weights of the model are updated according to the training samples. According to the usage rules (such as data type, training rounds, etc.), adjust the training process, such as selecting different optimizers or learning rates based on the data type. At the same time, use monitoring tools (such as StreamFlow's built-in monitoring tool) to track the training progress, including the time taken for each round, model convergence, etc., and dynamically adjust the training tasks based on the monitoring results, such as adjusting the data allocation ratio or optimization strategy.

[0098] After each round of training, each facility needs to transmit the updated model parameters back to the central node. The updated model parameters are collected from each node and aggregated into the global model parameters for that facility. The model parameters are then securely transmitted to the central node using an encrypted protocol (such as SSH or S3). Optimized transmission protocols and data compression techniques are used to reduce transmission time and bandwidth consumption. After receiving the model parameters from each facility, the central node sends a confirmation message to each facility to ensure the complete data transmission.

[0099] At the central node, the Federated Averaging Algorithm (FedAvg) is used to aggregate the model parameters of each facility to generate a global model. A weighted average is calculated based on the model parameters of each facility to generate the global model parameters. The weights can be adjusted based on the amount of data or training results of each facility. The aggregated global model parameters are then updated in the central server's model and broadcast to each facility, serving as the initial model parameters for the next round of training.

[0100] Through the above steps, efficient cross-facility collaboration for large-scale model training is achieved. A central server transmits initial model parameters to each facility through a workflow management system. Each facility then performs local model training and aggregates these parameters to create a global model using a federated averaging algorithm. This approach not only fully utilizes the computing resources of each facility but also protects data privacy through federated learning. Furthermore, dynamic adjustment and optimization strategies ensure efficient and convergent training.

[0101] For example, in the healthcare sector, data privacy and security are crucial. To train efficient medical diagnostic models, a central server transmits initial model parameters to various medical institutions (such as hospitals and clinics) through a workflow management system. These institutions, acting as independent facilities, receive the initial model parameters and distribute the model to computing nodes (such as GPU clusters) using fully sharded data parallelism. Each facility trains the model based on local training samples (such as medical images and electronic medical records) and predefined usage rules (such as data allocation ratios and model training strategies). During training, each node processes only its assigned subset of data, accelerating the training process through parallel computing. After each round of training, each facility transmits the updated model parameters back to the central node. The central server aggregates the model parameters across facilities using a federated averaging algorithm to generate a global model. This approach ensures efficient model training while avoiding cross-institutional transmission of raw data, complying with privacy regulations such as HIPAA. For example, this approach can be used to train models for early cancer screening, disease diagnosis, or personalized treatment recommendation, improving the quality and efficiency of healthcare services.

[0102] In the FinTech sector, data privacy is equally important, and financial institutions need to efficiently utilize decentralized data resources. A central server transmits initial model parameters to various financial institutions (such as banks and FinTech companies) through a workflow management system. These institutions, acting as independent facilities, distribute the model to their computing nodes using fully sharded data parallelism. Each facility trains the model based on local training samples (such as customer transaction records and credit scores) and predefined usage rules (such as data allocation ratios and model training strategies). During training, each node processes only its assigned subset of data, accelerating the training process through parallel computing. After each round of training, each facility transmits the updated model parameters back to the central node. The central server aggregates the model parameters across facilities using a federated averaging algorithm to generate a global model. This approach ensures efficient model training while avoiding the cross-institutional transmission of raw data, complying with privacy regulations such as GDPR. For example, this approach can be used to train models for credit risk assessment, fraud detection, or personalized financial service recommendations, improving the quality and efficiency of financial services.

[0103] In one embodiment, the step S400 further includes:

[0104] S4021. Setting a convergence condition through the parameterized template;

[0105] S4022. Training each facility based on initial model parameters and training samples;

[0106] S4023. Monitor the load of each facility during the model training process in real time, and dynamically adjust the training tasks according to the load until the convergence condition is met to generate a global model.

[0107] In this embodiment, the convergence conditions are set through parameterized workflow templates (such as CWL templates). These templates are written in YAML or JSON format, allowing users to flexibly configure various parameters, including convergence conditions. Users can select appropriate convergence indicators according to task requirements, such as ROUGE score (for text generation tasks) or accuracy (for classification tasks), and set a convergence threshold. When the model performance reaches or exceeds the threshold, the model is considered to have converged. For example, a ROUGE score of 0.9 indicates that the model performance meets the requirements.

[0108] During the cross-facility large model training process, the central server transmits the initial model parameters to each facility through a workflow management system (such as StreamFlow). After receiving the initial model parameters, each facility loads them into the local training environment and loads the training samples assigned to it from the local storage system. After initializing the local model using the received initial model parameters, each facility performs local model training based on the assigned training samples. During the training process, the model weights are updated based on the training samples. In order to efficiently utilize computing resources, the training task is decomposed into multiple subtasks and assigned to each computing node (such as GPU). Through fully sharded data parallelism (FSDP) technology, various parts of the model are assigned to different nodes, thereby accelerating the training process through parallel computing.

[0109] During the training process, the load of each facility is monitored in real time, and training tasks are dynamically adjusted based on the load. Use the built-in monitoring tools of the workflow management system (such as StreamFlow) to track the training progress and load of each facility in real time. Based on the monitoring results, the training tasks of each facility are dynamically adjusted to balance the load. For example, if the GPU utilization of a facility is too high, its training task volume can be appropriately reduced; if the GPU utilization of a facility is low, its training task volume can be increased. In addition, the distribution of training tasks can be dynamically adjusted based on real-time monitoring data, such as skipping training tasks for certain facilities and assigning them to other facilities with lower loads, or repeating training tasks for certain facilities to ensure uniformity of model training. If the model converges slowly at a facility, its learning rate or optimizer can be adjusted; if the resources of a facility are tight, its data allocation amount or model segmentation method can be adjusted.

[0110] After each round of training, the model performance is continuously monitored until the convergence conditions are met. The performance of the global model is evaluated, for example, by calculating the ROUGE score of the global model. If the model performance reaches or exceeds the preset convergence threshold (such as a ROUGE score of 0.9), the model is considered to have converged. After the convergence conditions are met, the model parameters of each facility are aggregated using the Federated Averaging Algorithm (FedAvg) to generate a global model. The aggregated global model parameters are updated to the model on the central server, and the final global model parameters are broadcast to each facility as a reference for subsequent tasks.

[0111] Through the above steps, efficient cross-facility collaboration and dynamic adjustment of large-scale model training are achieved. Convergence conditions are set through parameterized templates to ensure clear training objectives. Real-time monitoring and dynamic adjustment optimize the distribution of training tasks and fully utilize the computing resources of each facility. Finally, a federated averaging algorithm is used to generate a global model that meets convergence conditions and ensures model performance. This approach not only improves training efficiency but also enhances the robustness and adaptability of the system through dynamic adjustment mechanisms.

[0112] For example, in the healthcare sector, data privacy and model accuracy are crucial. By setting convergence criteria (such as achieving 90% model accuracy or 100 training rounds) through parameterized templates, medical institutions can flexibly define model training goals based on their needs. Each facility (such as a different hospital or research institution) trains the model based on the initial model parameters and local training samples (such as medical images and electronic medical records). Due to the complexity and diversity of medical data, computing resources and data volumes may vary across facilities. By monitoring the load of each facility in real time (such as GPU utilization and memory usage), the system can dynamically adjust the allocation of training tasks. For example, if a hospital has limited GPU resources, the system can allocate some of its training tasks to a facility with more abundant resources. This dynamic adjustment mechanism not only improves resource utilization efficiency but also ensures fairness and efficiency in the training process. When model performance reaches the preset convergence criteria, the system generates a global model, which can be used for tasks such as disease diagnosis, treatment recommendation, and patient health management, thereby improving the quality and efficiency of medical services.

[0113] In the FinTech sector, data privacy and model real-time performance are equally important. By setting convergence criteria (such as achieving 95% model prediction accuracy or 200 training epochs) through parameterized templates, financial institutions can flexibly define model training objectives based on business needs. Each facility (e.g., a different bank or FinTech company) trains the model based on initial model parameters and local training samples (such as customer transaction records and credit scores). Due to the sensitivity and real-time requirements of financial data, computing resources and data volumes may vary across facilities. By monitoring the load of each facility in real time (such as CPU utilization and data processing speed), the system can dynamically adjust the allocation of training tasks. For example, if a bank's transaction data volume suddenly increases, the system can allocate some of its training tasks to a facility with more abundant resources. This dynamic adjustment mechanism not only improves resource utilization efficiency but also ensures fairness and efficiency in the training process. When model performance reaches the preset convergence criteria, the system generates a global model, which can be used for tasks such as credit risk assessment, fraud detection, or personalized financial service recommendations, thereby improving the quality and efficiency of financial services.

[0114] In one embodiment, a cross-facility model training device is provided, which corresponds one-to-one to the cross-facility model training method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the cross-facility model training device of the present invention. It includes a training sample module 10, a template parsing module 20, a distribution strategy module 30, and a model training module 40. The functional modules are described in detail as follows:

[0115] A training sample module 10 is configured to obtain raw data of each facility, store the raw data in a local storage system, and use the raw data as training samples;

[0116] The template parsing module 20 is used to parse the parameterized template and extract workflow parameters and data features;

[0117] A distribution strategy module 30 is configured to determine a training sample distribution strategy for each facility based on the workflow parameters and / or data characteristics, and determine usage rules for the training samples of each facility based on the training sample distribution strategy;

[0118] The model training module 40 is used to train each facility based on initial model parameters, training samples and usage rules to generate a global model.

[0119] In one embodiment, the training sample module 10 includes:

[0120] A data acquisition and encryption unit, used to acquire the original data of each facility and encrypt the original data;

[0121] A data storage unit, used to store the encrypted original data in a local storage system;

[0122] The data processing unit is used to preprocess the original data stored in the local storage system to generate training samples.

[0123] In one embodiment, the template parsing module 20 includes:

[0124] A template analysis unit, configured to obtain a parameterized template, analyze the parameterized template, and obtain a template path;

[0125] A legal judgment unit is used for the parsing engine to verify whether the parameterized template is legal according to the template path;

[0126] The template parsing unit is used to parse the parameterized template by a parsing engine if the parameterized template is legal, so as to obtain workflow parameters and data features.

[0127] In one embodiment, the distribution strategy module 30 includes:

[0128] A configuration file unit, used to predefine workflow parameters and store the workflow parameters in a configuration file;

[0129] The configuration reading unit is used to read the workflow parameters of the configuration file through the workflow runner when the workflow is running;

[0130] A usage rule unit is used to parse the workflow parameters to obtain usage rules;

[0131] a sample segmentation unit, configured to segment the training sample into a plurality of data sample subsets according to the usage rule;

[0132] The data distribution unit is used to distribute the data sample subsets to each facility according to the data ratio of the usage rule, and store each subset in the local storage system of each facility.

[0133] In one embodiment, the distribution strategy module 30 includes:

[0134] A feature parsing unit, configured to parse the data features to obtain key features and feature mapping relationships;

[0135] A feature vector unit, configured to extract a feature vector of global data based on the key features and the feature mapping relationship;

[0136] A data type unit, configured to determine the data type of the training sample through the feature vector and determine a usage rule according to the data type;

[0137] A data distribution unit is configured to distribute the training samples to each facility according to the usage rule.

[0138] In one embodiment, the model training module 40 includes:

[0139] an initialization unit, configured to transmit the initial model parameters to each facility via a workflow management system from a central server;

[0140] Model parallelism unit, used to distribute the model to each node in each facility using fully sharded data parallel technology;

[0141] A model training unit, configured for each facility to perform model training based on training samples, initial model parameters, usage rules, and the nodes;

[0142] A parameter uploading unit, used to obtain the model parameters of each facility after each round of training and transmit the model parameters to the central server;

[0143] The aggregation logic unit is used to aggregate the model parameters of each facility by using a federated averaging algorithm to generate a global model.

[0144] In one embodiment, the model training module 40 further includes:

[0145] A convergence condition unit, configured to set a convergence condition using the parameterized template;

[0146] A model training unit, used to train each facility based on initial model parameters and training samples;

[0147] The task adjustment unit is used to monitor the load of each facility in real time during the model training process, and dynamically adjust the training tasks according to the load until the convergence condition is met to generate a global model.

[0148] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a cross-facility model training method.

[0149] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a cross-facility model training method.

[0150] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0151] Obtaining raw data of each facility, storing the raw data in a local storage system, and using the raw data as training samples;

[0152] Parse the parameterized template to extract workflow parameters and data features;

[0153] Determining a training sample distribution strategy for each facility based on the workflow parameters and / or data characteristics, and determining usage rules for the training samples of each facility based on the training sample distribution strategy;

[0154] Each facility is trained based on the initial model parameters, training samples, and usage rules to generate a global model.

[0155] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0156] Obtaining raw data of each facility, storing the raw data in a local storage system, and using the raw data as training samples;

[0157] Parse the parameterized template to extract workflow parameters and data features;

[0158] Determining a training sample distribution strategy for each facility based on the workflow parameters and / or data characteristics, and determining usage rules for the training samples of each facility based on the training sample distribution strategy;

[0159] Each facility is trained based on the initial model parameters, training samples, and usage rules to generate a global model.

[0160] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0161] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0162] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0163] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A cross-facility model training method, characterized in that: The following steps are involved: Obtaining raw data of each facility, storing the raw data in a local storage system, and using the raw data as training samples; Parse the parameterized template to extract workflow parameters and data features; Determining a training sample distribution strategy for each facility based on the workflow parameters and / or data characteristics, and determining usage rules for the training samples of each facility based on the training sample distribution strategy; Each facility is trained based on the initial model parameters, training samples, and usage rules to generate a global model.

2. The cross-facility model training method according to claim 1, characterized in that: The obtaining of raw data of each facility, storing the raw data in a local storage system, and using the raw data as training samples includes: Obtaining original data of each facility and encrypting the original data; Storing the encrypted original data in the local storage system; Preprocess the raw data stored in the local storage system to generate training samples.

3. The cross-facility model training method according to claim 1, characterized in that: The process of parsing the parameterized template and extracting workflow parameters and data features includes: Obtaining a parameterized template, analyzing the parameterized template, and obtaining a template path; The parsing engine verifies whether the parameterized template is legal based on the template path; If the parameterized template is legal, the parsing engine will parse the parameterized template to obtain workflow parameters and data features.

4. The cross-facility model training method according to claim 1, characterized in that: Determining a training sample distribution strategy for each facility based on the workflow parameters, and defining usage rules for training samples for each facility through the training sample distribution strategy, includes: Predefine workflow parameters and store the workflow parameters in a configuration file; When the workflow is running, the workflow parameters of the configuration file are read by the workflow runner; Parsing the workflow parameters to obtain usage rules; dividing the training sample into a plurality of data sample subsets according to the usage rule; The data sample subsets are distributed to each facility according to the data ratio using the rules, and each subset is stored in the local storage system of each facility.

5. The cross-facility model training method according to claim 1, wherein: Determining a training sample distribution strategy for each facility based on the data characteristics, and defining usage rules for the training samples of each facility through the training sample distribution strategy, includes: Analyzing the data features to obtain key features and feature mapping relationships; Extracting feature vectors of global data based on the key features and feature mapping relationships; Determine the data type of the training sample through the feature vector, and determine the usage rules according to the data type; The training samples are distributed to the facilities according to the usage rules.

6. The cross-facility model training method according to claim 1, characterized in that: The training of each facility based on the initial model parameters, training samples and usage rules to generate a global model includes: The central server transmits the initial model parameters to each facility through a workflow management system; Within each facility, the model is distributed to each node using fully sharded data parallelism. Each facility performs model training based on training samples, initial model parameters, usage rules, and the nodes; Obtaining model parameters of each facility after each round of training, and transmitting the model parameters to a central server; The global model is generated by aggregating the model parameters of each facility using a federated averaging algorithm.

7. The cross-facility model training method according to claim 1, wherein: The training of each facility based on the initial model parameters, training samples and usage rules to generate a global model also includes: Setting convergence conditions through the parameterized template; Train each facility based on the initial model parameters and training samples; The load of each facility during the model training process is monitored in real time, and the training tasks are dynamically adjusted according to the load conditions until the convergence conditions are met to generate a global model.

8. A cross-facility model training device, characterized in that: The cross-facility model training device includes: A training sample module, configured to obtain raw data of each facility, store the raw data in a local storage system, and use the raw data as training samples; Template parsing module, used to parse parameterized templates and extract workflow parameters and data features; a distribution strategy module, configured to determine a training sample distribution strategy for each facility based on the workflow parameters and / or data characteristics, and determine usage rules for the training samples of each facility based on the training sample distribution strategy; The model training module is used to train each facility based on initial model parameters, training samples and usage rules to generate a global model.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a cross-facility model training program stored in the memory and executable on the processor. When the cross-facility model training program is executed by the processor, the steps of the cross-facility model training method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a cross-facility model training program, which, when executed by the processor, implements the steps of the cross-facility model training method according to any one of claims 1 to 7.

Images