Medical large model migration training method for multi-matrix aggregation multi-dimensional features

Abstract

The invention belongs to the technical field of artificial intelligence, and more specifically relates to a medical large model migration training method based on multi-matrix aggregation multi-dimensional features. The method comprises the following steps: collecting professional medical data, and constructing a professional medical data set; federal migration training is performed on the pre-trained large model, and specific professional medical knowledge is injected into the pre-trained large model, specifically, for the pre-trained Baichuan large model, a low-rank matrix is initialized, and the low-rank matrix and the weight are quantized; the quantized low-rank matrix is adapted to a linear layer and an attention layer of a Baichuan large model; extracting three different dimensions of knowledge including original knowledge, low-rank knowledge and global complex knowledge from the professional medical data set through multi-round efficient migration training; and fusing the extracted knowledge of three different dimensions and carrying out weight aggregation. The problems that an existing training method needs a large amount of high-quality data and computing resources and is poor in performance in specific medical scenes with stricter requirements are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and more specifically, relates to a medical large model migration training method that aggregates multi-matrix and multi-dimensional features. Background Art

[0002] In recent years, large-scale pretrained models (LPMs) have achieved breakthroughs in fields such as natural language processing (NLP) and computer vision, and medical artificial intelligence (AI) applications have also benefited from this. However, medical data is highly specialized, and directly using general-purpose large models often fails to meet the needs of medical scenarios. Therefore, fine-tuning existing large models to build specialized medical large models has become a hot topic of research.

[0003] Chinese patent document CN118098562A discloses a medical large language model training method, including: constructing a pre-training data set and a fine-tuning instruction set, the pre-training data set including general text data and medical text data, and the fine-tuning instruction set including a single-round instruction set and a multi-round instruction set; using the general text data and the medical text data to continue pre-training the general large language model to obtain a medical large language model; fine-tuning the medical large language model according to the fine-tuning instruction set, and using the fine-tuned medical large language model as the target medical large language model to be used.

[0004] Currently, transfer training of large medical models primarily relies on full-parameter fine-tuning, efficient parameter fine-tuning, and retrieval-enhanced generation. Full-parameter fine-tuning directly adjusts the parameters of the entire model based on existing general-purpose large models, such as GPT, LLaMA, and BERT, to adapt to medical scenarios. However, this approach requires a large amount of high-quality medical data and has significant computational overhead, making it unsuitable for environments with limited computing resources. Efficient parameter fine-tuning methods, such as LoRA, Adapter, and Prefix Tuning, are widely used to reduce the amount of parameter updates during model fine-tuning to meet the need for lower computational and storage overhead. These methods alleviate the computational resource issues associated with full-parameter fine-tuning to a certain extent, but they still face significant challenges in the training of large medical models. For example, the PEFT method, which is applicable to general fields, may struggle to capture key professional medical knowledge when medical data is scarce, resulting in limited model performance.

[0005] In addition, existing efficient parameter fine-tuning methods may cause catastrophic forgetting after long-term training, making it difficult for the model to maintain its original general knowledge capabilities. RAG combines external medical knowledge bases to enable large models to dynamically retrieve relevant content when generating medical text, alleviating the problems of outdated knowledge and large data requirements within the model. However, its main pain point is its reliance on high-quality, structured medical knowledge bases, while existing medical data is often scattered and lacks standardization, resulting in limited retrieval results. In addition, the retrieval mechanism adapted to large models requires additional computing resources, especially on edge servers or low-computing power devices, which makes it difficult to deploy efficiently.

[0006] Therefore, although the existing transfer training technology has improved the adaptability and efficiency of large medical models to a certain extent, the following key problems still exist: the demand for high-quality medical data is large and the acquisition cost is high; the demand for computing resources is high and the fine-tuning cost is expensive; the knowledge transfer efficiency of the model is low and the adaptability is limited; the lack of knowledge update mechanism may lead to hallucination problems, etc. In response to these problems, there is an urgent need for an efficient transfer training method to extract rich parameter knowledge from scarce professional medical data to reduce the dependence on high-quality data and computing resources, while improving the adaptability and usability of the model in medical scenarios. The technical problem to be solved by the present invention is how to efficiently transfer and fine-tune a well-performing pre-trained large model in a scenario where the acquisition of professional medical data is limited, so that it can meet the needs of high-quality and high-efficiency application in specific downstream medical fields, thereby obtaining a large professional medical model in a subdivided field. Summary of the Invention

[0007] The present invention aims to overcome at least one of the defects of the above-mentioned prior art and provide a medical large model migration training method with multi-matrix aggregation and multi-dimensional features, so as to solve the problem that the existing training methods require a large amount of high-quality data and computing resources and perform poorly in specific medical scenarios with more stringent requirements, so as to be applied to the training of professional medical large models.

[0008] The detailed technical solutions of the present invention are as follows: A multi-matrix aggregation and multi-dimensional feature large medical model migration training method, the method comprising: S1. Collect professional medical data and build professional medical datasets; S2. Federated transfer training pre-trains large models and injects specific professional medical knowledge into them, including: S21. For the pre-trained Baichuan large model, initialize the low-rank matrix and quantize the low-rank matrix and weights; S22. Fix most of the parameters of the Baichuan model, fine-tune only a few parameters, and adapt the quantized low-rank matrix to the linear layer and attention layer of the Baichuan model; S23, extracting knowledge of three different dimensions: original knowledge, low-rank knowledge, and global complex knowledge from professional medical datasets through multiple rounds of efficient transfer training; S24. Fuse the extracted knowledge of three different dimensions and perform weight aggregation.

[0009] Preferably, according to the present invention, the specific steps of step S1 are as follows: S11. Set downstream hospitals, clinics, medical institutions, and medical schools as edge nodes in the federation scenario. With the support of the federation privacy protection algorithm, professional medical data is collected and aggregated from multiple nodes. The collected professional medical data includes: (1) Hospital and clinic data; (2) Experimental analysis data and diagnosis and treatment data from specialized medical institutions and medical schools; (3) Research results in specific fields in authoritative medical databases; S12, professional medical data cleaning and data enhancement; Data cleaning involves: first, using text processing tools to perform preliminary cleaning on the raw data, filtering out low-quality data due to OCR errors or confusing formats; then, using string matching or similarity calculation algorithms to automatically remove duplicate data to ensure the uniqueness and diversity of the dataset; to protect patient privacy, sensitive information is desensitized using regular expressions or NLP technology, and different expressions in the data are standardized using medical knowledge graphs or terminology comparison tables, unifying medical terminology, date formats, and unit conversions, thereby converting multi-source data into a unified format to facilitate subsequent data fusion and model processing; Data augmentation involves leveraging the data preprocessing modules within the deep learning framework to build an automated data augmentation pipeline for online dynamic augmentation. This includes replacing synonyms with medical terms, randomly inserting or deleting non-critical information words to create subtle perturbations without changing semantics, and generating extended descriptions based on pre-trained language models that are semantically similar to the original text. S13. Construct professional medical datasets: perform manual labeling and quality control, invite professional physicians to review the data, proofread and confirm key information to ensure the accuracy and professionalism of the data; divide the organized professional medical data into training, validation, and test sets, and convert the data into the input format required by the model and store it in a unified format for subsequent calls and training; finally, construct a professional medical dataset for a specific department according to the task requirements.

[0010] Preferably, according to the present invention, the specific steps of S21 are as follows: For the pre-trained large model Baichuan, its initial weight is expressed as ; The first is initialization, Create five low-rank matrices M, N, X, Y, Z and introduce two hyperparameters r1 and r2, where r1 represents the output dimension of matrices M, X, and Z, and r2 represents the output dimension of N and the input dimension of X; Then, the matrices M, Y, and X are Gaussian initialized, and the matrices N and Z are zero-initialized. By scaling the low-rank matrix to a smaller matrix, more knowledge information can be obtained while only increasing the limited training parameters. This ensures the high efficiency of training and greatly improves the model's ability to handle specific downstream tasks. Further considering the limited computing resources, in order to minimize the resource requirements of local training while ensuring accuracy, the weights of the pre-trained large model Baichuan are Perform INT4 formal quantization and BF16 formal quantization on the five low-rank matrices. The formula is as follows: (1) (2) In formulas (1) and (2), x represents the form of the instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing. Indicates the quantization process, p1 and p2 are quantization constants, and k-bit represents the specific form of quantization, including FP32, INT4, and NF4; represents the initial weights using NF4 quantization, Represents a low-rank matrix using BF16 quantization. After the final quantization is completed, the digits are unified and expressed as , , .

[0011] According to a preferred embodiment of the present invention, in step S22, adapting the quantized low-rank matrix to the linear layer and attention layer of the Baichuan large model refers to: First, search for all linear layers that meet the insertion criteria in the Baichuan model's feedforward network layer (FFN). Then, insert the five initialized low-rank matrices into the q, k, and v matrices of the Baichuan model's attention layer, and freeze the initial weights of the q, k, and v matrices. At the same time, insert the five low-rank matrices into all linear layers that meet the insertion criteria in the Baichuan model's feedforward network layer (FFN). Similarly, insert the five initialized low-rank matrices into the newly generated head layer of the Baichuan model to extract information that is more relevant to downstream medical tasks. The attention layer formula is as follows: (3) In formula (3), Q represents the query vector of the input data, K represents the key-value vector, V represents the value vector, and T represents the transposition operation of the K vector. It is the dimension of the key-value vector to prevent the dot product value from being too large, which may lead to the gradient disappearance problem. The softmax function converts it into a probability score, and finally obtains the attention score of V to Q. ; The formula of the feedforward network layer FFN is as follows, which represents the result after the input x is transformed by the feedforward network layer FFN: (4) In formula (4), and represents the weights and biases of the first layer, and Represents the weights and biases of the second layer.

[0012] Preferably, according to the present invention, the specific steps of S23 are as follows: Next, fine-tuning is performed to extract multi-level knowledge. Based on the idea and training method of federated learning, the professional medical dataset and the quantized model weights and low-rank matrices are distributed to multiple servers for multiple rounds of efficient fine-tuning training. Each server node performs multiple rounds of training based on the local professional medical dataset, and sends the parameter matrix updated in each training to the central node. The central node updates the global weights based on the federated aggregation algorithm and sends them to the server nodes for training again. Through this step, three different dimensions of knowledge are extracted from the professional medical dataset: original knowledge, low-rank knowledge, and global complex knowledge.

[0013] Preferably, according to the present invention, the three different dimensions of knowledge, namely, original knowledge, low-rank knowledge, and global complex knowledge, are extracted from professional medical data sets as follows: The process of extracting the original knowledge of the model is as follows: x (5) In formula (5), represents the extracted original knowledge, x represents the vector form of the instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing, 、 、 They are the increased dimension matrix, the matrix added by the optimization process and the reduced dimension matrix, which are further expressed as low-rank matrices , , , where r, d are the dimensions of the initial weight matrix of the Baichuan large model; The low-rank knowledge is extracted based on the original LoRA fine-tuning extraction, and the formula is as follows: (6) In formula (6), represents the extracted low-rank knowledge, x represents the form of instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing, and They are the dimensionality-raising and lowering matrices used for fine-tuning, which can be expressed as low-rank matrices , ; Extracting global complex knowledge is done by introducing a scaling operator to obtain global information content and further extracting it. The formula is as follows: x (7) In formula (7), represents the extracted global complex knowledge, x represents the form of the instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing, which can be represented as a low-rank matrix , , 、 Represents the scaling matrix used to extract complex deep knowledge, which can be expressed as a low-rank matrix , , where r1 is a hyperparameter that adjusts the dimension of the low-rank matrix.

[0014] Preferably, in step S24, the extracted knowledge of three different dimensions is integrated and weighted, as follows: After each server has carried out several iterations of efficient fine-tuning training on the Baichuan model, a global model convergence is performed to fuse the three different dimensions of knowledge extracted, namely initial knowledge, low-rank knowledge, and global complex knowledge, to obtain , and further integrate the results of the calculation and pre-trained large model weights The sum is the final output, which is finally expressed as the weight of the pre-trained large model Based on this, we add the fusion calculation result Z(1+Y+XN)M to obtain the professional medical model Baichuan-pro, whose weight is , the formula is as follows: + + (8) (9) It is worth noting that the weights and data involved in the entire efficient fine-tuning training process on the server side are all quantized in BF16 format to ensure the accuracy of the multi-level knowledge extracted during the fine-tuning process. The formula is as follows: (10) In formula (10), represents the input after BF16 quantization, and Represents the initial weight after using BF16 quantization form, , , , , They represent the low-rank matrix after BF16 quantization, and finally the quantized output uplink weight is obtained. .

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention adopts efficient migration and low-cost fine-tuning technology, and only updates the newly added low-rank matrix, thereby significantly reducing computing resources and storage overhead, and is suitable for resource-constrained medical institutions.

[0016] (2) The present invention utilizes a federated learning framework, where each medical institution can independently train models locally and only share model updates. This not only protects patient privacy but also enables cross-institutional knowledge sharing and collaborative modeling, promoting efficient cooperation among hospitals, research institutions, and medical service providers.

[0017] (3) The present invention extracts multiple matrices and aggregates knowledge of different dimensions through an efficient transfer training process. The model can quickly adapt to the ever-changing medical knowledge and clinical needs, thereby playing a greater role in actual diagnosis, treatment recommendations and health management. At the same time, it supports the research and development of telemedicine, intelligent consultation and clinical decision support systems, and comprehensively improves the efficiency and intelligence level of the medical system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of the efficient transfer training optimization method for the large medical model described in the present invention.

[0019] Figure 2 This is the overall framework diagram of the efficient transfer training optimization method for the large medical model described in the present invention.

[0020] Figure 3 This is a comparison of the F1 accuracy of the method of the present invention and other methods in question answering ability. DETAILED DESCRIPTION

[0021] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0023] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0024] In the absence of conflict, the embodiments of the present disclosure and the features thereof may be combined with each other.

[0025] Example 1 Ginseng Figure 1 and Figure 2 This embodiment provides a multi-matrix aggregation and multi-dimensional feature large medical model migration training method, the method comprising: Step S1: Collect professional medical data and build a professional medical data set This step further includes the following steps: Step S11: Acquire professional medical data through multiple channels; Data used in general medicine is often easy to collect and is widely distributed across medical forums and websites, published medical papers and monographs, and public datasets commonly used in theoretical research. However, when focusing on specific departments, such as cardiology, oncology, and endocrinology, medical data involves a large amount of patient privacy and sensitive data, which is subject to strict laws and regulations. Data sharing between different medical institutions presents significant obstacles. Furthermore, data from various hospitals and medical institutions is often distributed across different systems, with varying data formats, annotation standards, and storage methods. This makes it difficult to directly integrate data into a high-quality, unified dataset, making it impossible to collect large amounts for model training. Therefore, the idea of federated learning is being considered as a solution. Federated learning technology allows each medical institution to conduct specialized data training locally and only share model updates. This protects patient privacy while leveraging the data strengths of different departments to better build large professional medical models.

[0026] Specifically, to enable the large medical model to focus on more detailed medical fields, such as cardiovascular and tumor diseases, the federated aggregation concept is combined with secure data sharing technologies such as secure multi-party computing (SMPC) to ensure that data meets privacy protection and compliance requirements during collection and transmission. This embodiment sets downstream hospitals, clinics, and medical institutions as edge nodes in the federated scenario. With the support of the federated privacy protection algorithm, professional data is collected and aggregated from multiple nodes. The edge nodes and data sources include the following: (1) Hospital and clinic data: anonymized electronic medical records, clinical diagnosis cases and surgical records, images, pathology results, etc.

[0027] (2) Specialized medical institutions and medical schools: experimental analysis and diagnosis and treatment data of cardiovascular research centers and tumor treatment centers.

[0028] (3) Authoritative medical database: obtain research results in specific fields, such as tumor gene data, endocrine pathology analysis, and field research papers.

[0029] Step S12: Medical professional data cleaning and data enhancement; Data cleaning is crucial in the integration of medical professional data. First, text processing tools are used to perform preliminary cleaning of the raw data, filtering out low-quality data due to OCR errors or confusing formats. String matching or similarity calculation algorithms, such as Jaccard similarity and Cosine similarity, are then used to automatically remove duplicate data, ensuring the uniqueness and diversity of the dataset. To protect patient privacy, sensitive information such as name, ID number, and contact information is desensitized using regular expressions or NLP techniques. Medical knowledge graphs or terminology comparison tables are used to standardize different expressions in the data, unifying medical terminology, date formats, and unit conversions. This converts multi-source data into a unified format, facilitating subsequent data fusion and model processing.

[0030] In terms of data augmentation, we leverage data preprocessing modules in deep learning frameworks, such as Transforms in TensorFlow or PyTorch, to build an automated data augmentation pipeline for online dynamic augmentation. This includes using medical lexicons for synonym replacement, such as replacing "coronary heart disease" with "coronary artery disease," randomly inserting or deleting non-critical information words to create subtle perturbations without changing semantics, and generating extended descriptions based on pretrained language models that are semantically similar to the original text. These measures significantly improve the richness and coverage of the dataset, providing higher-quality input for subsequent model fine-tuning and ultimately enhancing the accuracy and robustness of medical models in practical applications.

[0031] S13, build professional medical data sets; Manual annotation and quality control are performed, and professional physicians are invited to review the data. Key information, such as disease names, diagnostic indicators, and treatment plans, is proofread and confirmed to ensure data accuracy and professionalism. The compiled professional medical data is divided into training, validation, and test sets. The data is converted into the input formats required by the model, such as question-answer pairs, paragraph summaries, and sequence annotation formats, and stored in a unified format, such as JSON, CSV, and TFRecord, to facilitate subsequent calls and training. Finally, specialized medical datasets for specific departments are constructed according to task requirements, such as diagnosis prediction and text generation.

[0032] Step S2: Federated transfer training pre-trains the large model and injects specific professional medical knowledge into it The purpose of this step is to quickly inject professional medical knowledge into the pre-trained large model, so that it has the ability to be applied in specific medical fields. Therefore, this embodiment selects the Baichuan series pre-trained model Baichuan, which has performed well in multiple tasks, as the base model, and uses the instruction data in the professional medical data set constructed in step S1 to efficiently fine-tune it. Since the efficient parameter fine-tuning methods currently used in most scenarios cannot meet the high-precision requirements in medical scenarios, such as LoRA low-rank fine-tuning, which has the inherent defect of too few training parameters leading to loss of accuracy, it is impossible to fully extract the internal knowledge of a small number of professional data sets. In order to further focus on professional fields with higher specific application value, it is obviously impossible to meet the standards for LoRA fine-tuning based solely on the professional data set collected in step S1, which is far smaller than the general data set. Therefore, the parameter matrix in the training process is increased to fully extract and aggregate the deeper multi-dimensional information, so as to ensure the ability to handle specific medical tasks.

[0033] This embodiment uses federated learning combined with quantization technology to perform efficient migration and fine-tuning training on the Baichuan large model on multiple Nvidia A100 servers. The specific steps are as follows.

[0034] This step further includes the following steps: Low-rank matrix initialization and weight quantization: For the pre-trained large model Baichuan, its initial weight is expressed as , the training update weights for efficient transfer fine-tuning are expressed as The first step is initialization. Create five low-rank matrices M, N, X, Y, Z, and introduce two hyperparameters r1 and r2, where r1 represents the output dimensions of matrices M, X, and Z, and r2 represents the output of N and the input dimension of X; then perform Gaussian initialization on matrices M, Y, and X, and zero matrix initialization on matrices N and Z; compared to the original low-rank decomposition of LoRA, by scaling, the low-rank matrix for extracting complex deep features is limited to a smaller matrix, which can obtain more knowledge information under the condition of only increasing limited training parameters, which ensures the high efficiency of training and greatly improves the model's ability to handle specific downstream tasks; further considering the situation of limited computing resources, in order to minimize the resource requirements of local training while ensuring accuracy, the weights of the pre-trained large model Baichuan are Perform INT4 formal quantization and BF16 formal quantization on the five initialized low-rank matrices. The formula is as follows: (1) (2) In formulas (1) and (2), x represents the form of the instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing. The dataset constructed in step S1 is a collection of instruction data, specifically in the form of medical knowledge question and answer pairs; Indicates the quantization process, p1 and p2 are quantization constants, and k-bit represents the specific form of quantization, including FP32, INT4, and NF4. represents the initial weights using NF4 quantization, Represents a low-rank matrix using BF16 quantization. After the final quantization is completed, the digits are unified and expressed as , , .

[0035] Finally, most of the parameters of the Baichuan model are fixed, and only a few parameters are fine-tuned to adapt the quantized low-rank matrices to the linear and attention layers of this model. First, all linear layers in the Baichuan model that meet the insertion conditions are searched; then, the five initialized low-rank matrices are inserted into the q, k, and v matrices of the Attention layer of the Baichuan model, and the initial weights of the q, k, and v matrices are frozen. At the same time, the five low-rank matrices are inserted into all linear layers in the feedforward network layer FFN of the Baichuan model that meet the insertion conditions; the five initialized low-rank matrices are also inserted into the newly generated head layer of the Baichuan model to extract information that is more similar to downstream medical tasks.

[0036] The Attention layer formula is as follows: (3) In formula (3), Q represents the query vector of the input data, K represents the key-value vector, V represents the value vector, and T represents the transposition operation of the K vector. is the dimension of the key-value vector, which is used to prevent the dot product value from being too large, leading to the gradient vanishing problem. The softmax function converts it into a probability score, and finally obtains the attention score of V on Q, Attention(Q,K,V); The formula of the feedforward network layer FFN is as follows, which represents the result after the input x is transformed by the feedforward network layer FFN: (4) In formula (4), and represents the weights and biases of the first layer, and Represents the weights and biases of the second layer.

[0037] Fine-tune to extract multi-level knowledge and perform weight aggregation: Next, fine-tuning is performed to extract multi-level knowledge. Based on the idea and training method of federated learning, the professional medical dataset, quantized model weights, and low-rank matrices are distributed to multiple Nvidia A100 servers for multiple rounds of training updates. Each Nvidia A100 server node performs multiple rounds of training based on the local professional medical dataset, and sends the parameter matrix updated in each training to the central node. The central node updates the global weights based on the federated aggregation algorithm and sends them to the server nodes for training again. Through this step, three different dimensions of knowledge are extracted from the professional medical dataset: original knowledge, low-rank knowledge, and global complex knowledge. The extracted knowledge parameters are shown in the following formula: The process of extracting the original knowledge of the model is as follows: x (5) In formula (5), represents the extracted original knowledge, x represents the vector form of the instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing, 、 、 They are the increased dimension matrix, the matrix added by the optimization process and the reduced dimension matrix, which are further expressed as low-rank matrices , , , where r and d are the dimensions of the initial weight matrix of the Baichuan large model; The extraction of low-rank knowledge is based on the fine-tuning of the original LoRA method. The formula is as follows: (6) In formula (6), represents the extracted low-rank knowledge, x represents the form of instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing, and They are the dimensionality-raising and lowering matrices used for fine-tuning, which can be expressed as low-rank matrices , , r, d are the dimensions of the initial weight matrix of the Baichuan large model; Extracting global complex knowledge is done by introducing a scaling operator to obtain global information content and further extracting it. The formula is as follows: x (7) In formula (7), represents the extracted global complex knowledge, x represents the form of instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing, 、 Represents a common dimension-raising and lowering matrix, which can be expressed as a low-rank matrix , , 、 Represents the scaling matrix used to extract complex deep knowledge, which can be expressed as a low-rank matrix , , where r and d are the dimensions of the initial weight matrix of the Baichuan large model, and r1 is a hyperparameter that adjusts the dimension of the low-rank matrix.

[0038] After each server has carried out several iterations of efficient fine-tuning training on the Baichuan model, a global model convergence is performed to fuse the three different dimensions of knowledge extracted, namely initial knowledge, low-rank knowledge, and global complex knowledge, to obtain , and further integrate the results of the calculation and pre-trained large model weights The sum is the final output, which is finally expressed as the weight of the pre-trained large model Based on this, we add the fusion calculation result Z(1+Y+XN)M to obtain the professional medical model Baichuan-pro, whose weight is , the formula is as follows: + + (8) (9) It is worth noting that the weights and data involved in the entire efficient fine-tuning training process on the server side are all quantized in BF16 format to ensure the accuracy of the multi-level knowledge extracted during the fine-tuning process. The formula is as follows: (10) In formula (10), represents the input after BF16 quantization, and Represents the initial weight after using BF16 quantization form, , , , , They represent the low-rank matrix after BF16 quantization, and finally the quantized output uplink weight is obtained. .

[0039] Evaluation of professional medical large models based on test sets: After the transfer training and fine-tuning of fine-tuned datasets for different professional fields are completed, multiple specialized medical large models can be obtained, such as a cardiovascular diagnosis assistant large model, an oncology specialist large model, and a urology medical question-and-answer large model. Furthermore, based on specific hardware resource capabilities, pre-trained models 7B and 13B with different parameter values can be used to migrate specialized large models with different capabilities. Furthermore, based on the test set of the specialized medical datasets produced in step S1, the model's capabilities can be evaluated in various aspects: (1) Medical Q&A and diagnostic support: Evaluate the accuracy and relevance of the model’s answers to medical questions, and test its effectiveness in terms of diagnostic recommendations and treatment plans.

[0040] (2) Text generation and summarization: This includes tasks such as medical record generation and medical text summarization. It measures the fluency, professionalism, and matching degree of the generated content with the reference text, such as using indicators such as BLEU and ROUGE.

[0041] (3) Information extraction and entity recognition: This involves identifying and extracting medical entities, such as disease names, symptoms, and drugs, and examining the model's ability to extract key information from text.

[0042] In addition, medical dialogue systems, multi-round communication, and clinical decision support can also be evaluated to ensure the reliability, security, and compliance of the model in practical applications.

[0043] This invention has broad applicability and can be used to build customized, specialized medical models in various medical fields, including cardiovascular, oncology, neurology, respiratory medicine, gastroenterology, and endocrinology. It can also provide solutions for specific application scenarios, such as imaging diagnosis, medical record analysis, clinical decision support, and personalized treatment recommendations. Its core advantage lies in the use of efficient migration and low-cost fine-tuning techniques, which only updates newly added low-rank matrices, significantly reducing computing resources and storage overhead, making it suitable for resource-constrained medical institutions. Furthermore, utilizing a federated learning framework, each medical institution can independently train the model locally and only share model updates, protecting patient privacy while enabling cross-institutional knowledge sharing and collaborative modeling, promoting efficient collaboration between hospitals, research institutions, and healthcare providers. Through this efficient migration and training process, the model can quickly adapt to evolving medical knowledge and clinical needs, thereby playing a greater role in actual diagnosis, treatment recommendations, and health management. It also supports the development of telemedicine, intelligent consultation, and clinical decision support systems, comprehensively improving the efficiency and intelligence of the medical system. In summary, this invention not only solves the current problems of data quality, computing resources and privacy protection in large-scale model fine-tuning, but also provides an efficient and low-cost technical solution for intelligent diagnosis and treatment, precision medicine and cross-institutional collaborative research in the medical field. It has extremely broad application prospects and commercial promotion value in the future.

[0044] The beneficial effects of the present invention were tested in a number of tasks: 1. In the question answering ability test, the QA task was evaluated on the SQuADv1.1 dataset. The proposed method was tested with LoRA, AdaLoRA and different ranks. The rank r was set to {1, 2, 4, 8}, the batch size was set to 16, and AdamW was used as the optimizer. The experimental results are shown in the figure. Figure 3 As shown, Figure 3 The horizontal axis represents the rank of the low-rank matrix used, and the vertical axis represents the F1 precision score, which is an indicator used to measure the accuracy of the model. Obviously, the method proposed in this invention is better than the best baseline AdaLoRA in terms of F1 score. 2. In the medical question-and-answer test with high privacy requirements, two groups of experiments were conducted using Chatdoctor data based on the federated learning framework: one group used the LoRA fine-tuning method as a benchmark, and the other group applied the improved solution of the present invention to compare the performance of the client model and the global model. In the experiment, GPT-4 was used to generate a reference answer for each query, and the query was prompted to score different model combinations on a 10-point scale based on the reference answer, and the reasons for the score were given. In order to ensure the fairness and accuracy of the evaluation, this embodiment used ChatGPT-4 to conduct two rounds of evaluation on the order of the model assistant's answers, took the average value, and used multiple weighted averages as the final result. The results show that the medical local model that was migrated and fine-tuned using the method of the present invention has obvious advantages in generating medical questions and answers.

[0045] 3. To comprehensively evaluate the processing performance of the proposed method, experiments compared the proposed method with full parameter fine-tuning combined with quantization in terms of video memory usage, training time, and throughput. The results are shown in Table 1. Clearly, the proposed local fine-tuning method significantly reduces video memory usage by approximately 10 times compared to full parameter fine-tuning, demonstrating the ability of the proposed algorithm to locally fine-tune models at the exascale scale.

[0046] Table 1 Comparison of the comprehensive performance of the efficient migration method of the present invention and the full fine-tuning method in various aspects

[0047] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A multi-matrix aggregation and multi-dimensional feature medical large model migration training method, characterized by: The method comprises: S1. Collect professional medical data and build professional medical datasets; S2. Federated transfer training pre-trains large models and injects specific professional medical knowledge into them, including: S21. For the pre-trained Baichuan large model, initialize the low-rank matrix and quantize the low-rank matrix and weights; S22. Adapt the quantized low-rank matrix to the linear layer and attention layer of the Baichuan large model; S23, extracting knowledge of three different dimensions: original knowledge, low-rank knowledge, and global complex knowledge from professional medical datasets through multiple rounds of efficient transfer training; S24. Fuse the extracted knowledge of three different dimensions and perform weight aggregation.

2. The method for transferring and training a large medical model with multi-matrix aggregation and multi-dimensional features according to claim 1 is characterized in that: The specific steps of S1 are as follows: S11. Set downstream hospitals, clinics, medical institutions, and medical schools as edge nodes in the federation scenario. With the support of the federation privacy protection algorithm, professional medical data is collected and aggregated from multiple nodes. The collected professional medical data includes: hospital and clinic data; Experimental analysis data and diagnosis and treatment data from specialized medical institutions and medical schools; Research results in specific fields in authoritative medical databases; S12, professional medical data cleaning and data enhancement; Data cleaning involves: first, using text processing tools to perform preliminary cleaning on the raw data, filtering out low-quality data due to OCR errors or confusing formats; then, using string matching or similarity calculation algorithms to automatically remove duplicate data to ensure the uniqueness and diversity of the dataset; to protect patient privacy, sensitive information is desensitized using regular expressions or NLP technology, and different expressions in the data are standardized using medical knowledge graphs or terminology comparison tables, unifying medical terminology, date formats, and unit conversions, thereby converting multi-source data into a unified format to facilitate subsequent data fusion and model processing; Data augmentation involves leveraging the data preprocessing modules within the deep learning framework to build an automated data augmentation pipeline for online dynamic augmentation. This includes replacing synonyms with medical terms, randomly inserting or deleting non-critical information words to create subtle perturbations without changing semantics, and generating extended descriptions based on pre-trained language models that are semantically similar to the original text. S13. Construct professional medical datasets: perform manual labeling and quality control, invite professional physicians to review the data, proofread and confirm key information to ensure the accuracy and professionalism of the data; divide the organized professional medical data into training, validation, and test sets, and convert the data into the input format required by the model and store it in a unified format for subsequent calls and training; finally, construct a professional medical dataset for a specific department according to the task requirements.

3. The method for transferring and training a large medical model with multi-matrix aggregation and multi-dimensional features according to claim 1 is characterized in that: The specific steps of S21 are as follows: For the pre-trained large model Baichuan, its initial weight is expressed as ; The first is initialization, Create five low-rank matrices M, N, X, Y, Z and introduce two hyperparameters r1 and r2, where r1 represents the output dimension of matrices M, X, and Z, and r2 represents the output dimension of N and the input dimension of X; Then, perform Gaussian initialization on matrices M, Y, and X, and zero matrix initialization on matrices N and Z; restrict the low-rank matrix to a smaller matrix by scaling, and adjust the weights of the pre-trained large model Baichuan. Perform INT4 formal quantization and BF16 formal quantization on the five low-rank matrices. The formula is as follows: （1） （2） In formula (1) and (2), x represents the form of the instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing. Indicates the quantization process, p1 and p2 are quantization constants, and k-bit represents the specific form of quantization, including FP32, INT4, and NF4; represents the initial weights using NF4 quantization, Represents a low-rank matrix using BF16 quantization. After the final quantization is completed, the digits are unified and expressed as , , .

4. The method for transferring and training a large medical model with multi-matrix aggregation and multi-dimensional features according to claim 1 is characterized in that: In step S22, adapting the quantized low-rank matrix to the linear layer and attention layer of the Baichuan large model means: First, search for all linear layers that meet the insertion criteria in the Baichuan model's feedforward network layer (FFN). Then, insert the five initialized low-rank matrices into the q, k, and v matrices of the Baichuan model's attention layer, and freeze the initial weights of the q, k, and v matrices. At the same time, insert the five low-rank matrices into all linear layers that meet the insertion criteria in the Baichuan model's feedforward network layer (FFN). Similarly, insert the five initialized low-rank matrices into the newly generated head layer of the Baichuan model to extract information that is more relevant to downstream medical tasks. The attention layer formula is as follows: （3） In formula (3), Q represents the query vector of the input data, K represents the key-value vector, V represents the value vector, and T represents the transposition operation of the K vector. It is the dimension of the key-value vector to prevent the dot product value from being too large, which may lead to the gradient disappearance problem. The softmax function converts it into a probability score, and finally obtains the attention score of V to Q. ; The formula of the feedforward network layer FFN is as follows, which represents the result after the input x is transformed by the feedforward network layer FFN: （4） In formula (4), and represents the weights and biases of the first layer, and Represents the weights and biases of the second layer.

5. The method for transferring and training a large medical model with multi-matrix aggregation and multi-dimensional features according to claim 3 is characterized in that: The specific steps of S23 are as follows: Next, fine-tuning is performed to extract multi-level knowledge. Based on the idea and training method of federated learning, the professional medical dataset and the quantized model weights and low-rank matrices are distributed to multiple servers for multiple rounds of efficient fine-tuning training. Each server node performs multiple rounds of training based on the local professional medical dataset, and sends the parameter matrix updated in each training to the central node. The central node updates the global weights based on the federated aggregation algorithm and sends them to the server nodes for training again. Through this step, three different dimensions of knowledge are extracted from the professional medical dataset: original knowledge, low-rank knowledge, and global complex knowledge.

6. The method for transferring and training a large medical model with multi-matrix aggregation and multi-dimensional features according to claim 5 is characterized in that: The three dimensions of knowledge extracted from professional medical datasets, namely original knowledge, low-rank knowledge, and global complex knowledge, are as follows: The process of extracting the original knowledge of the model is as follows: x（5） In formula (5), represents the extracted original knowledge, x represents the vector form of the instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing, 、 、 They are the increased dimension matrix, the matrix added by the optimization process and the reduced dimension matrix, which are further expressed as low-rank matrices , , , where r, d are the dimensions of the initial weight matrix of the Baichuan large model; The low-rank knowledge is extracted based on the original LoRA fine-tuning extraction, and the formula is as follows: （6） In formula (6), represents the extracted low-rank knowledge, x represents the form of instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing, and They are the dimensionality-raising and lowering matrices used for fine-tuning, which can be expressed as low-rank matrices , ; Extracting global complex knowledge is done by introducing a scaling operator to obtain global information content and further extracting it. The formula is as follows: x（7） In formula (7), represents the extracted global complex knowledge, x represents the form of the instruction data in the professional medical dataset constructed in step S1 after word segmentation and embedding processing, which can be represented as a low-rank matrix , , 、 Represents the scaling matrix used to extract complex deep knowledge, which can be expressed as a low-rank matrix , , where r1 is a hyperparameter that adjusts the dimension of the low-rank matrix.

7. The method for transferring and training a large medical model with multi-matrix aggregation and multi-dimensional features according to claim 3 is characterized in that: In step S24, the extracted knowledge of three different dimensions is fused and weighted, as follows: After each server has carried out several iterations of efficient fine-tuning training on the Baichuan model, a global model convergence is performed to fuse the three different dimensions of knowledge extracted, namely initial knowledge, low-rank knowledge, and global complex knowledge, to obtain , and further integrate the results of the calculation and pre-trained large model weights The sum is the final output, which is finally expressed as the weight of the pre-trained large model Based on this, we add the fusion calculation result Z(1+Y+XN)M to obtain the professional medical model Baichuan-pro, whose weight is , the formula is as follows: + + （8） （9） The weights and data involved in the entire efficient fine-tuning training process on the server side are all quantized in BF16 format to ensure the accuracy of the multi-level knowledge extracted during the fine-tuning process. The formula is as follows: （10） In formula (10), represents the input after BF16 quantization, and Represents the initial weight after using BF16 quantization form, , , , , They represent the low-rank matrix after BF16 quantization, and finally the quantized output uplink weight is obtained. .

Images