Federal learning system of medical big language model

Abstract

The embodiment of the invention relates to a federal learning system of a medical big language model. The system comprises a knowledge graph, a central network and a plurality of mechanism networks, the central network is used for configuring a global feature mapping rule, pre-training a base large language model for the first time, generating and issuing a corresponding version number + issuing model parameters + fine tuning layer numbers every time a version of central model parameters is generated, aggregating mechanism model parameters according to a mean value mode, and generating a new version of central model parameters based on the aggregated parameters; and the mechanism network is used for carrying out local training once after receiving a group of version numbers, issuing model parameters and finely tuning layer numbers, and uploading trained mechanism model parameters to the central network. According to the invention, the generalization ability of the medical big language model can be continuously improved under the condition that the medical data is not shared.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly relates to a federated learning system for a medical large language model. Background Art

[0002] Large Language Models (LLMs) can perform various Natural Language Processing (NLP) tasks, such as text generation tasks, text classification tasks, etc. Applying large language models in the medical field (referred to as medical large language models) can improve the classification accuracy of medical documents (such as electronic medical records, etc.). Before using a medical large language model, it needs to be trained. If the training data is insufficient, the generalization ability of the model will be reduced. Currently, a large amount of medical data is mainly distributed in various medical institutions (such as hospitals, medical teaching / research institutions, etc.). If a traditional centralized training mechanism is adopted, it is necessary to centrally collect the medical data of all institutions, which is very difficult to achieve in the actual situation.

[0003] Federated Learning technology is a technical framework for distributed model training. In principle, based on this technical framework, the generalization ability training of the model can be completed without sharing institutional data. That is to say, in principle, if a medical large language model is trained based on this technical framework, the generalization ability of the model can be continuously improved without sharing medical data. And how to train a medical large language model based on this technical framework is the technical problem to be solved by the present invention. Summary of the Invention

[0004] The objective of the present invention is to provide a federated learning system for a medical large language model in view of the deficiencies of the prior art. The system includes a knowledge graph, a central network, and multiple institutional networks. The central network includes a central server, a corpus, and a central model library. The institutional network includes an institutional server, a text set, and an institutional model library. Among them: 1) The central server is used to set a global feature mapping rule for the custom terms of all medical institutions; it is also used to perform a pre-training on the base large language model; it is also used to assign / set the corresponding version number, parameter directory, fine-tuning layer number, and distribute the model parameters for each version of the central model parameters generated, and store the current central model parameters and fine-tuning layer number in the corresponding directory, and send the current version number + distributed model parameters + fine-tuning layer number to each institutional server; it is also used to store the institutional model parameters sent by each institutional server in the corresponding directory and update the base model parameters according to each parameter directory. 2) The institutional server is used to construct a local fine-tuning framework and a local task framework according to the feature mapping rule, the knowledge graph, and the institutional model library; it is also used to perform a model parameter setting on the institutional large language model every time a set of version number + distributed model parameters + fine-tuning layer number is received, and perform a local training on the institutional large language model based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework, and send the latest institutional model parameters to the central server at the end of this training. The system of the present invention provides a federated learning solution for training a medical large language model. Through the present invention, the generalization ability of the model can be continuously improved under the condition of non-sharing of medical data.

[0005] To achieve the above objective, an embodiment of the present invention provides a federated learning system for a medical large language model. The system includes a knowledge graph, a central network, and multiple institutional networks;

[0006] The central network includes a central server, a corpus, and a central model library;

[0007] The central server is used to configure the one-to-one correspondence between the custom terms of all medical institutions and the nodes of the knowledge graph to obtain the corresponding feature mapping rule and send it to the institutional servers of all the institutional networks; and initialize the base model parameters;

[0008] The central server is also used to update the corpus through big data collection;

[0009] The central server is also used to, when the initialization of the base model parameters is completed, perform a model parameter setting on the base large language model in the central model library according to the base model parameters; construct a central pre-training framework according to the feature mapping rule, the knowledge graph, and the central model library; perform a first pre-training according to the corpus and the central pre-training framework; and use the current model parameters of the base large language model as the initial version of the central model parameters at the end of this training.

[0010] The central server is also used to, when generating each version of the central model parameters, assign a corresponding version number to the current central model parameters; assign a corresponding parameter directory to the current version number; set the corresponding fine-tuning layer number and distribute the model parameters based on the current version number; store the current central model parameters and the fine-tuning layer number in the current parameter directory; and send the current version number, the distributed model parameters, and the fine-tuning layer number to each institutional server; the fine-tuning layer number is a positive integer greater than or equal to 0.

[0011] The central server is also used to, when receiving the institutional model parameters and the version number sent from each institutional server, store the current institutional model parameters in the corresponding parameter directory.

[0012] The central server is also used to update the base model parameters according to each parameter directory to obtain a new version of the central model parameters.

[0013] The institutional network includes the institutional server, the text set, and the institutional model library.

[0014] The institutional server is used to update the text set through the data collection of this institution.

[0015] The institutional server is also used to construct a local fine-tuning framework and a local task framework according to the received feature mapping rule, knowledge graph, and institutional model library.

[0016] The institutional server is also used to, when receiving each group of the version number, the distributed model parameters, and the fine-tuning layer number, perform a model parameter setting on the institutional large language model in the institutional model library based on the fine-tuning layer number and the distributed model parameters; then perform a local training based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework; and at the end of this training, set the corresponding institutional model parameters based on the current model parameters of the institutional large language model and the fine-tuning layer number; and send the institutional model parameters and the version number to the central server.

[0017] Preferably, the knowledge graph consists of a first node set and a first edge set; the first node set includes multiple first nodes; each first node corresponds to a standard medical term; each first node corresponds to a set of node attributes, and the node attributes at least include a term name, a term code, a term type, and a data type; the term name is a medical term name; the term code is a standard code of a specified medical standard system, and the medical standard system at least includes the ICD-10 standard system; the term type at least includes diseases, drugs, diagnoses, tests, examinations, and genes; the first edge set includes multiple first edges; each first edge is a directed edge and is used to connect two first nodes; the edge attribute of each first edge is a type of medical term association relationship.

[0018] The corpus includes multiple first corpus texts.

[0019] The text set includes multiple first text records; the first text record includes a first medical text and a first classification label; the first classification label consists of multiple first label classification probabilities, and each first label classification probability corresponds to a type of preset disease type; only one of the first label classification probabilities in each first classification label is 1, and the rest of the first label classification probabilities are 0.

[0020] The inventory models in the central model library at least include the base large language model, the first feature projection model, the first generative adversarial network, and the first mask prediction model; the base large language model is a type of medical large language model.

[0021] The inventory models in the institutional model library at least include the institutional large language model, the second feature projection model, the second generative adversarial network, the second mask prediction model, and the institutional text classification model; the model structure of the institutional large language model is the same as that of the base large language model.

[0022] Preferably, the first feature projection model is implemented based on the MLP model structure; the model input and output vectors of the first feature projection model are the corresponding first text feature vectors and first text mapping vectors; the first text feature vector consists of multiple first word segmentation feature vectors; the first text mapping vector consists of multiple first word segmentation mapping vectors, and the first word segmentation mapping vector corresponds to the first word segmentation feature vector one by one. The vector feature dimension of the first word segmentation mapping vector is higher than that of the first word segmentation feature vector, and the vector feature dimension of the first word segmentation mapping vector is consistent with the input vector feature dimension of the base large language model.

[0023] The base large language model is implemented based on the Med-BERT model; the model input and output vectors of the base large language model are the corresponding first text mapping vectors and first text encoding vectors; the first text encoding vector is composed of multiple first token encoding vectors, and the first token encoding vector corresponds one-to-one with the first token mapping vector;

[0024] The first generative adversarial network is implemented based on the GAN model; the first generative adversarial network includes a first generator and a first discriminator; the model input and output of the first generator are the corresponding first text mapping vectors and first text scrambling vectors; the model input of the first discriminator is the corresponding first text mapping vector and the first text scrambling vector, and the model output is the corresponding first discrimination probability; the first text scrambling vector is composed of multiple first token scrambling vectors, and the first token scrambling vector corresponds one-to-one with the first token mapping vector; the first discrimination probability is a probability value between 0 and 1;

[0025] The first mask prediction model is implemented based on the task head network of the mask prediction task of the BERT model; the model input of the first mask prediction model is the corresponding first text encoding vector and first mask position index sequence; the first mask position index sequence is composed of one or more first mask position indexes, and each first mask position index corresponds to one of the first token encoding vectors in the first text encoding vector; the model output of the first mask prediction model is the corresponding first probability vector sequence; the first probability vector sequence is composed of one or more first probability vectors, and the first probability vector corresponds one-to-one with the first mask position index; the vector length of the first probability vector is the total number of tokens in the model vocabulary of the base large language model, and the first probability vector is composed of multiple first token probabilities, and the first token probability corresponds one-to-one with the tokens in the model vocabulary;

[0026] The second feature projection model is implemented based on the MLP model structure; the model input and output vectors of the second feature projection model are the corresponding second text feature vectors and second text mapping vectors;

[0027] The institutional large language model has the same model structure as the base large language model; the model input and output vectors of the institutional large language model are the corresponding second text mapping vectors and second text encoding vectors;

[0028] The second generative adversarial network is implemented based on the GAN model; the second generative adversarial network includes a second generator and a second discriminator; the model inputs and outputs of the second generator are the corresponding second text mapping vectors and second text scrambling vectors; the model input of the second discriminator is the corresponding second text mapping vector and the second text scrambling vector, and the model output is the corresponding second discrimination probability;

[0029] The second mask prediction model is implemented based on the task head network of the mask prediction task of the BERT model, and its model function is similar to that of the first mask prediction model; the model input of the second mask prediction model is the corresponding second text encoding vector and the second mask position index sequence; the second mask position index sequence consists of one or more second mask position indexes; the model output of the second mask prediction model is the corresponding second probability vector sequence; the second probability vector sequence consists of one or more second probability vectors, and the second probability vectors correspond one by one to the second mask position indexes; the vector length of the second probability vector is the total number of word segmentations of the model vocabulary of the institutional large language model, and the second probability vector consists of multiple second word segmentation probabilities, and the second word segmentation probabilities correspond one by one to the word segmentations of the model vocabulary; the model vocabulary of the institutional large language model is consistent with the model vocabulary of the base large language model;

[0030] The institutional text classification model is implemented based on a multi-classification model structure; the model inputs and outputs of the institutional text classification model are the corresponding second text encoding vectors and the first classification probability vector; the first classification probability vector consists of multiple first classification probabilities, and each first classification probability corresponds to a preset disease type; the institutional text classification model is used to perform classification prediction based on the second text encoding vector input to the model and output the corresponding first classification probability vector.

[0031] Preferably, the central pre-training framework includes a first preprocessing module, the first feature projection model, the base large language model, the first mask prediction model, and the first generative adversarial network;

[0032] The input end of the first preprocessing module is used to receive the first text, and the output end is connected to the input end of the first feature projection model; the output end of the first feature projection model is respectively connected to the input ends of the base large language model and the first generative adversarial network; the output end of the base large language model is connected to the first input end of the first mask prediction model; the second input end of the first mask prediction model is used to receive the first training index sequence, and the output end is used to output the corresponding first probability vector sequence; the output end of the first generative adversarial network is used to output the corresponding first discrimination probability; the first training index sequence consists of one or more first training indexes;

[0033] The first preprocessing module is used to take the received first text as the corresponding current text, preprocess the current text to obtain the corresponding first text feature vector, and send it to the first feature projection model. Specifically:

[0034] Perform word segmentation on the current text according to the word segmentation rules of the base large language model to obtain the corresponding first word segmentation sequence; and based on the feature mapping rule, record the first nodes corresponding to the respective first word segments in the first word segmentation sequence in the knowledge graph as the corresponding first word segment nodes; and identify whether each of the first word segment nodes is empty. If so, record the first word segment corresponding to the current first word segment node as a non-term word segment, and if not, record the first word segment corresponding to the current first word segment node as a term word segment; and use some or all of the node attributes in the knowledge graph as the corresponding node features; and form a corresponding first term feature vector from all the node features corresponding to each term word segment; and configure a zero vector for each non-term word segment as the corresponding first term feature vector; and perform encoding on each first word segment in the first word segmentation sequence according to the standard BERT model embedding method to obtain the corresponding first embedding coding vector; and perform vector concatenation on the first term feature vector and the first embedding coding vector corresponding to each first word segment according to the vector concatenation method to obtain the corresponding first word segment feature vector; and form the corresponding first text feature vector from all the obtained first word segment feature vectors and send it to the first feature projection model.

[0035] Preferably, the local fine-tuning framework includes a second preprocessing module, the second feature projection model, the institutional large language model, the second mask prediction model, and the second generative adversarial network;

[0036] The input end of the second preprocessing module is used to receive the second text, and the output end is connected to the input end of the second feature projection model; the output end of the second feature projection model is respectively connected to the input ends of the institutional large language model and the second generative adversarial network; the output end of the institutional large language model is connected to the first input end of the second mask prediction model; the second input end of the second mask prediction model is used to receive the second training index sequence, and the output end is used to output the corresponding second probability vector sequence; the output end of the second generative adversarial network is used to output the corresponding second discrimination probability; the second training index sequence is composed of one or more second training indexes;

[0037] The second preprocessing module is used to take the received second text as the corresponding current text, preprocess the current text to obtain the corresponding second text feature vector, and send it to the second feature projection model. Specifically:

[0038] Perform word segmentation on the current text according to the word segmentation rules of the institutional large language model to obtain the corresponding second word segmentation sequence; and based on the feature mapping rule, record the first nodes corresponding to each second word in the second word segmentation sequence in the knowledge graph as the corresponding second word nodes; and identify whether each second word node is empty. If so, record the second word corresponding to the current second word node as a non-term word, and if not, record the second word corresponding to the current second word node as a term word; and form a corresponding second term feature vector from all the node features corresponding to each term word; and configure a zero vector for each non-term word as the corresponding second term feature vector; and encode each second word in the second word segmentation sequence according to the standard BERT model embedding method to obtain the corresponding second embedding coding vector; and perform vector splicing on the second term feature vector and the second embedding coding vector corresponding to each second word according to the vector splicing method to obtain the corresponding second word feature vector; and form the corresponding second text feature vector from all the obtained second word feature vectors and send it to the second feature projection model; the word segmentation rules of the institutional large language model are consistent with the word segmentation rules of the base large language model.

[0039] Preferably, the local task framework includes the second preprocessing module, the second feature projection model, the institutional large language model, and the institutional text classification model;

[0040] The input end of the second preprocessing module is used to receive the third text, and the output end is connected to the input end of the second feature projection model; the output end of the second feature projection model is connected to the input end of the institutional large language model; the output end of the institutional large language model is connected to the input end of the institutional text classification model; the output end of the institutional text classification model is used to output the corresponding first classification probability vector.

[0041] Preferably, when the central server performs the first pre-training according to the corpus and the central pre-training framework:

[0042] Step 71, count the total number of the first corpus texts in the corpus to obtain the corresponding total number N a ; and record each first corpus text as the corresponding first text t i , 1 ≤ index i ≤ N a; and set the current training stage as the first stage; and according to the word segmentation rules of the base large language model for each of the first text t i Perform pre-segmentation processing to obtain the corresponding first pre-segmentation sequence sa i ;

[0043] Step 72, identify the current training stage; if the current training stage is the first stage, then the first pre-segmentation sequence sa is segmented according to the preset first mask rate and mask segmentation "[MASK]" i The word segmentation in is randomly masked and replaced to obtain the corresponding second pre-segmentation sequence sb i If the current training stage is the second stage, then the first pre-segmentation sequence sa is i The segmentation words of each specific disease type are masked and replaced to obtain the corresponding second pre-segmentation sequence sb i ;

[0044] Step 73: each of the second pre-segmented word sequences sb i The sequence index of each masked word in is used as a corresponding first training index s i,n ; and each of the second pre-segmentation sequences sb i All the corresponding first training indexes s i,n Form a corresponding first training index sequence S i ; and based on each of the first training indexes s i,n Set a corresponding first label vector y i,n ; and each of the first training index sequences S i All the corresponding first label vectors y i,n Form a corresponding first label vector sequence Y i ;

[0045] Among them, each of the first training index sequences S i The first training index s i,n The total number is recorded as N i , 1≤indexn≤N i ; The first label vector y i,n The vector length of y is the total number of word segments in the model vocabulary of the base large language model, and the total number of word segments in the model vocabulary of the base large language model is recorded as W; the first label vector y i,n By multiple first label probabilities ρ i,n,w The first label probability ρ i,n,w One-to-one correspondence with the word segmentation of the model vocabulary, 1≤index w≤W; the first label vector y i,n There is only one first label probability ρi,n,w is 1, and the rest are 0; the first label vector y i,n Specifically, the first label probability ρ is 1 i,n,w The corresponding word in the model vocabulary and the first training index s i,n In the first pre-segmentation sequence sa i The corresponding word matching in ;

[0046] Step 74: each of the first texts t i and the corresponding first training index sequence S i As the corresponding first text and the first training index sequence are input into the central pre-training framework for a forward reasoning; and the first probability vector sequence output by this reasoning is used as the corresponding first prediction vector sequence The first text feature vector output by the first preprocessing module in this inference process is recorded as the corresponding first feature vector x i ;

[0047] Among them, each of the first prediction vector sequences By the corresponding N i The first prediction vector The first prediction vector By W first predicted probabilities composition;

[0048] Step 75: each of the first label vector sequences Y i and its corresponding first prediction vector sequence Form a corresponding first data group And by N a The first eigenvector x i The first vector sequence {x i};

[0049] Step 76, the obtained N a The first data set Bring in the preset first model loss function L1 to calculate and obtain the corresponding first loss value;

[0050] Among them, the first model loss function L1 is:

[0051]

[0052] Step 77: Identify whether the first loss value meets a preset first loss value range; if the first loss value meets the first loss value range, then identify whether the current training stage is the first stage. If it is, reset the current training stage to the second stage and return to Step 72; if not, proceed to Step 78. If the first loss value does not meet the first loss value range, perform one round of modulation on the model parameters of the first feature projection model, the base large language model, and the first mask prediction model in the central pre-training framework based on a preset first model optimizer in the direction of minimizing the first model loss function L1, and return to Step 74 at the end of this round of modulation.

[0053] Among them, the first model optimizer at least includes the Adam optimizer and the SGD optimizer.

[0054] Step 78: Substitute the latest first vector sequence {x i} into a preset first discriminative loss L D1 and a first generative loss L G1 ; and perform one round of modulation on the model parameters of the first feature projection model in the central pre-training framework and the first discriminator of the first generative adversarial network based on a preset second model optimizer in the direction of minimizing the first discriminative loss L D1 ; and at the end of this round of modulation, perform one round of modulation on the model parameters of the first feature projection model and the first generator of the first generative adversarial network based on a preset third model optimizer in the direction of minimizing the first generative loss L G1 and confirm the end of this training at the end of this round of modulation.

[0055] Among them, the first discriminative loss L D1 is:

[0056]

[0057] θ 11 is the model parameter of the first feature projection model, and F 11 () is the inference function of the first feature projection model; F 11 (x i ; θ 11 ) is used to represent the text mapping vector output by the first feature projection model under the condition that the projection model parameter is θ 11 and the model input is the text feature vector x i .

[0058] θ 12 is the model parameter of the first discriminator of the first generative adversarial network, and F 12( ) is the inference function of the first discriminator; F 12 [F 11 (x i ; θ 11 ); θ 12 is used to represent that when the model parameters of the discriminator are θ 12 and the input of the discriminator is the text mapping vector F 11 (x i ; θ 11 ), the discrimination probability output by the first discriminator;

[0059] The first generation loss L G1 is:

[0060]

[0061] θ 13 is the model parameter of the first generator of the first generative adversarial network, F 13 ( ) is the inference function of the first generator; F 13 [F 11 (x i ; θ 11 ), θ 13 is used to represent that when the model parameters of the generator are θ 13 and the model input is the text mapping vector F 11 (x i ; θ 11 ), the text scrambling vector output by the first generator; F 12 {F 13 [F 11 (x i ; θ 11 ), θ 13 ; θ 12} is used to represent that when the model parameters of the discriminator are θ 12 and the input of the discriminator is the text scrambling vector F 13 [F 11 (x i ; θ 11 ), θ 13 , the discrimination probability output by the first discriminator;

[0062] The second model optimizer includes at least Adam optimizer and SGD optimizer;

[0063] The third model optimizer includes at least Adam optimizer and SGD optimizer.

[0064] Preferably, the central server is specifically used for setting the corresponding fine-tuning layer number and distributing model parameters based on the current version number:

[0065] Identify whether the version number matches the preset initial version number; if it matches, set the fine-tuning layer number to 0, and use the central model parameters corresponding to the current version number as the corresponding issued model parameters; if it does not match, use the layer number corresponding to the last Encoder layer of the base large language model as the fine-tuning layer number, and use the model parameters corresponding to the last Encoder layer in the central model parameters corresponding to the current version number as the corresponding issued model parameters.

[0066] Preferably, the central server is specifically configured to, when updating the base model parameters according to each parameter directory to obtain a new version of the central model parameters:

[0067] Take each parameter directory as the corresponding current directory; and take the fine-tuning layer number and the central model parameters stored in the current directory as the corresponding current number and current central model parameters; and identify the current number; if the current number is 0, take all the parameters of the current central model parameters as the current reference parameters; if the current number is greater than 0, take the model layer parameters corresponding to the current number in the current central model parameters as the current reference parameters; and identify whether the total number of the institutional model parameters stored in the current directory matches the total number of the institutional networks, if so, perform model parameter fusion on all the institutional model parameters in the current directory to obtain the corresponding current fusion parameters, and replace some or all of the parameters in the current central model parameters corresponding to the current reference parameters based on the current fusion parameters to obtain a new version of the central model parameters;

[0068] Among them, the model parameter fusion is specifically as follows:

[0069]

[0070] 1 ≤ institutional index k ≤ N org ,

[0071] θ k is the k-th institutional model parameter in the current directory,

[0072] θ R is the current fusion parameter.

[0073] Preferably, the institutional server is specifically configured to, when performing one local training based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework:

[0074] Identify the current fine-tuning layer number;

[0075] If the current fine-tuning layer number is 0, all model parameters of the second feature projection model, the institutional large language model, the second mask prediction model, and the second generative adversarial network form a corresponding first adjustable parameter set; and all model parameters of the institutional large language model and the institutional text classification model form a corresponding second adjustable parameter set;

[0076] If the current fine-tuning layer number is greater than 1, only the overall mode parameters of the second feature projection model and the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the institutional large language model form the corresponding first adjustable parameter set; and only the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the institutional large language model and the overall model parameters of the institutional text classification model form the corresponding second adjustable parameter set;

[0077] And perform a pre-training based on the text set, the local fine-tuning framework, and the first adjustable parameter set; and after the end of this pre-training, perform a task training based on the text set, the local task framework, and the second adjustable parameter set; and at the end of this task training, confirm the end of the current local training.

[0078] Furthermore, the institutional server is specifically used for, when performing a pre-training based on the text set, the local fine-tuning framework, and the first adjustable parameter set:

[0079] Step 111, count the total number of the first medical texts in the text set to obtain a corresponding total number N b ; and denote each of the first medical texts as a corresponding second text t j , 1 ≤ index j ≤ N b ; and perform pre-tokenization processing on each of the second texts t j according to the tokenization rule of the institutional large language model to obtain a corresponding third pre-tokenization sequence sc j ; and randomly mask and replace the tokens in each of the third pre-tokenization sequences sc j according to a preset second masking rate and the masking token "[MASK]" to obtain a corresponding fourth pre-tokenization sequence sd j ;

[0080] Step 112, use the sequence index of each masked token in each of the fourth pre-tokenization sequences sd j as a corresponding second training index s j,m ; and form a corresponding second training index sequence S from all the second training indices s j corresponding to each of the fourth pre-tokenization sequences sd j,m ; j; and based on each of the second training indexes s j,m Set a corresponding second label vector y j,m ; and each of the second training index sequences S j All the corresponding second label vectors y j,m Form a corresponding second label vector sequence Y j ;

[0081] Among them, each of the second training index sequences S j The second training index s j,m The total number is recorded as N j , 1≤indexm≤N j ; The second label vector y j,m The vector length of y is the total number of word segments in the model vocabulary of the large language model of the organization, which is consistent with the total number of word segments W in the model vocabulary of the large language model of the base; the second label vector y j,m By multiple second label probabilities ρ j,m,w The second label probability ρ j,m,w One-to-one correspondence with the word segmentation of the model vocabulary, 1≤index w≤W; the second label vector y j,m There is only one second label probability ρ j,m,w is 1, and the rest are 0; the second label vector y j,m The second label probability ρ is specifically 1 j,m,w The corresponding word in the model vocabulary and the second training index s j,m In the third pre-segmentation sequence sc j The corresponding word matching in ;

[0082] Step 113: each of the second texts t j and the corresponding second training index sequence S j The second text and the second training index sequence are input into the local fine-tuning framework for a forward inference; and the second probability vector sequence output this time is used as the corresponding second prediction vector sequence The second text feature vector output by the second preprocessing module in this inference process is recorded as the corresponding second feature vector x j ; And the second text encoding vector output by the language model of the organization during this reasoning process is recorded as the corresponding first encoding vector e j ;

[0083] Among them, each of the second prediction vector sequences By the corresponding N j The second prediction vector The second prediction vector Consisting of W second prediction probabilities ;

[0084] Step 114, form a corresponding second data group from each of the second label vector sequences Y j and their corresponding second prediction vector sequences and form a corresponding second vector sequence {x and consist of N b second feature vectors x j}; j}

[0085] Step 115, take each of the first encoding vectors e j as the corresponding current encoding vector, add noise to the current encoding vector to obtain the corresponding positive sample vector and denote the remaining N b - 1 first encoding vectors e j other than the current encoding vector as the corresponding negative sample vectors 1 ≤ index u ≤ N b - 1; and form a corresponding third data group from each of the first encoding vectors e j and their corresponding one positive sample vector and N b - 1 negative sample vectors ;

[0086] Step 116, denote the parameter part of the institutional large language model in the first adjustable parameter set as parameter θ l , and denote the corresponding issued model parameter of the parameter θ L as parameter θ C ; and substitute the obtained N a second and third data groups into the preset second model loss function L2 for calculation to obtain the corresponding second loss value;

[0087] wherein, the second model loss function L2 is:

[0088] L2 = L 21 + αL 22 + β||θ l - θ C || 2 ,

[0089]

[0090] α and β are two preset weight parameters, τ are two preset temperature coefficients, and the loss function L 21Implemented based on the cross-entropy loss function, the loss function L 22 Implemented based on the contrastive loss function; ||θ l -θ C || 2 Is the latest parameter θ l And the parameter θ C Of the L2 regularization; sim() is the cosine similarity function of vectors; For each of the first encoded vectors e j And its corresponding positive sample vector Of the cosine similarity, Is the first encoded vector e j And its corresponding negative sample vectors Of the cosine similarity;

[0091] Step 117, identify whether the second loss value satisfies a preset second loss value range; if the second loss value satisfies the second loss value range, go to step 118; if the second loss value does not satisfy the second loss value range, then based on a preset fourth model optimizer, perform a round of modulation on the model parameters in the first adjustable parameter set in the direction of minimizing the second model loss function L2, and return to step 113 at the end of this round of modulation;

[0092] Among them, the fourth model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0093] Step 118, bring the latest second vector sequence {x j} into the preset second discriminative loss L D2 And the second generation loss L G2 ; and based on a preset fifth model optimizer, perform a round of modulation on the model parameters in the first adjustable parameter set in the direction of minimizing the second discriminative loss L D2 ; and at the end of this round of modulation, then based on a preset sixth model optimizer, perform a round of modulation on the model parameters in the first adjustable parameter set in the direction of minimizing the second generation loss L G2 ; and confirm the end of this training at the end of this round of modulation;

[0094] Among them, the second discriminative loss L D2 Is:

[0095]

[0096] G is the first adjustable parameter set;

[0097] θ 21 Is the model parameter of the second feature projection model; θ21 ∩G is the adjustable parameter part corresponding to the second feature projection model in the first adjustable parameter set; F 21 () is the inference function of the second feature projection model; F 21 (x j ; θ 21 ) is used to represent the text mapping vector output by the second feature projection model under the condition that the projection model parameter is θ 21 and the model input is the text feature vector x j ;

[0098] θ 22 is the model parameter of the second discriminator of the second adversarial network; θ 22 ∩G is the adjustable parameter part corresponding to the second discriminator in the first adjustable parameter set; F 22 () is the inference function of the second discriminator; F 22 [F 21 (x j ; θ 21 ); θ 22 is used to represent the discrimination probability output by the second discriminator under the condition that the discriminator model parameter is θ 22 and the discriminator input is the text mapping vector F 21 (x j ; θ 21 );

[0099] The second generation loss L G2 is:

[0100]

[0101] θ 23 is the model parameter of the second generator of the second generative adversarial network; θ 23 ∩G is the adjustable parameter part corresponding to the second generator in the first adjustable parameter set; F 23 () is the inference function of the second generator; F 23 [F 21 (x j ; θ 21 ), θ 23 is used to represent the text scrambling vector output by the second generator under the condition that the generator model parameter is θ 23 and the model input is the text mapping vector F 21 (x j ; θ 21 ); F 22 {F 23 [F 21 (x j ; θ 21), θ 23 ; θ 22} used to represent under the condition that the discriminator model parameters are θ 22 and the discriminator input is the text scrambling vector F 23 [F 21 (x j ; θ 21 ), θ 23 , the discrimination probability output by the second discriminator;

[0102] The fifth model optimizer includes at least Adam optimizer and SGD optimizer;

[0103] The sixth model optimizer includes at least Adam optimizer and SGD optimizer.

[0104] Furthermore, the institutional server is specifically used for performing a task training based on the text set, the local task framework, and the second adjustable parameter set:

[0105] Step 121, count the total number of the first medical texts in the text set to obtain the corresponding total number N c ; and record each of the first medical texts as the corresponding third text t h , 1 ≤ index h ≤ N c ; and perform pre-tokenization processing on each of the third texts t h according to the tokenization rules of the institutional large language model to obtain the corresponding fifth pre-tokenization sequence se h ; and record the first classification label corresponding to each of the first medical texts as the corresponding label vector Y h ;

[0106] Among them, the label vector Y h includes multiple label probabilities ρ h,v , 1 ≤ index v ≤ N d , N d is the total number of preset disease types;

[0107] Step 122, use each of the third texts t h as the corresponding third text to input into the local task framework for a forward inference; and use the first classification probability vector output this time as the corresponding prediction vector and record the second text mapping vector output by the second feature projection model during this inference process as the corresponding mapping vector X h ;

[0108] Among them, the prediction vector includes multiple prediction probabilities

[0109] Step 123, from each of the label vectors Y h and its corresponding prediction vector to form a corresponding fourth data set and set a standard normal distribution space with a feature dimension consistent with the mapping vector X h and randomly sample N e space vectors from the standard normal distribution space and denote them as X g ; and from the mapping vector X h and the space vector P g pairwise combination to obtain N c ×N e fifth data sets (X h , P g );

[0110] wherein, N e is a positive integer, and N e > N c ; 1 ≤ index g ≤ N e ;

[0111] Step 124, denote the overall model parameters of the second feature projection model as parameter θ proj ; and substitute the parameter θ proj , N c fourth data sets N c ×N e fifth data sets (X h , P g ) into a preset third model loss function L3 for calculation to obtain a corresponding third loss value;

[0112] wherein, the third model loss function L3 is:

[0113]

[0114] γ, λ are two preset weight parameters; 1 ≤ index h * ≤ N c , 1 ≤ index g * ≤ N e ; the loss function L 31 is implemented based on the cross-entropy loss function, and the loss function L 32 is implemented based on the maximum mean discrepancy loss function, is the Frobenius norm regularizer of the latest parameter θ proj ; f kernel () is a preset kernel function with two input quantities;

[0115] Step 125: Identify whether the third loss value meets a preset third loss value range. If the third loss value does not meet the third loss value range, perform one round of modulation on the model parameters in the second adjustable parameter set in the direction of minimizing the third model loss function L3 based on a preset seventh model optimizer, and return to Step 122 at the end of this round of modulation. If the third loss value meets the third loss value range, stop training and confirm the end of the current task training.

[0116] The embodiment of the present invention provides a federated learning system for a medical large language model. The system includes: a knowledge graph, a central network, and multiple institutional networks; the central network includes a central server, a corpus, and a central model library; the institutional network includes an institutional server, a text set, and an institutional model library. Among them: 1) The central server is used to set a global feature mapping rule for the custom terms of all medical institutions; is also used to perform a pre-training on the base large language model; is also used to assign / set a corresponding version number, parameter directory, fine-tuning layer number, and distribute the model parameters for each generated version of the central model parameters, store the current central model parameters and fine-tuning layer number in the corresponding directory, and distribute the current version number + distributed model parameters + fine-tuning layer number to each institutional server; is also used to store the institutional model parameters sent up by each institutional server in the corresponding directory and update the base model parameters according to each parameter directory. 2) The institutional server is used to construct a local fine-tuning framework and a local task framework according to the feature mapping rule, the knowledge graph, and the institutional model library; is also used to perform a model parameter setting on the institutional large language model for each received set of version number + distributed model parameters + fine-tuning layer number, perform a local training on the institutional large language model based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework, and send up the latest institutional model parameters to the central server at the end of this training. The embodiment of the present invention provides a federated learning solution for training a medical large language model, and through the embodiment of the present invention, the generalization ability of the model can be continuously improved under the condition of non-sharing of medical data. Description of the Drawings

[0117] Figure 1 It is a module structure diagram of a federated learning system for a medical large language model provided by an embodiment of the present invention;

[0118] Figure 2 It is a module structure diagram of a central pre-training framework provided by an embodiment of the present invention;

[0119] Figure 3 It is a module structure diagram of a local fine-tuning framework provided by an embodiment of the present invention;

[0120] Figure 4This is the module structure diagram of the local task framework provided by the embodiments of the present invention. Detailed implementation manners

[0121] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Apparently, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0122] A federated learning system for a medical large language model provided by an embodiment of the present invention, as Figure 1 shown in the module structure diagram of a federated learning system for a medical large language model provided by an embodiment of the present invention, mainly includes: a knowledge graph 1, a central network 2, and multiple institutional networks 3.

[0123] (1) Knowledge graph 1:

[0124] The knowledge graph 1 of the embodiment of the present invention includes a first node set and a first edge set.

[0125] Among them, the first node set includes multiple first nodes; each first node corresponds to a standard medical term; each first node corresponds to a set of node attributes, and the node attributes at least include a term name, a term code, a term type, and a data type; the term name is a medical term name; the term code is a standard code of a specified medical standard system, and the medical standard system at least includes the ICD-10 standard system; the term type at least includes diseases, drugs, diagnoses, tests, examinations, and genes; the first edge set includes multiple first edges; each first edge is a directed edge and is used to connect two first nodes; the edge attribute of each first edge is a type of medical term association relationship.

[0126] (2) Central network 2:

[0127] The central network 2 of the embodiment of the present invention includes a central server 21, a corpus 22, and a central model library 23. The central server 21 is respectively connected to the knowledge graph 1, the corpus 22, the central model library, and the institutional servers 31 of each institutional network 3.

[0128] 1) Central server 21:

[0129] The central server 21 of the embodiment of the present invention is used to configure the one-to-one correspondence between the custom terms of all medical institutions and the nodes of the knowledge graph 1 to obtain corresponding feature mapping rules and send them to the institutional servers 31 of all institutional networks 3; and initialize the base model parameters.

[0130] The central server 21 is also used to update the corpus 22 through big data collection.

[0131] The central server 21 is also used to, when the initialization of the base model parameters is completed, perform a model parameter setting on the base large language model in the central model library 23 according to the base model parameters; construct a central pre-training framework according to the feature mapping rules, the knowledge graph 1, and the central model library 23; perform a pre-training according to the corpus 22 and the central pre-training framework; and use the current model parameters of the base large language model as the initial version of the central model parameters at the end of this training.

[0132] The central server 21 is also used to, when generating each version of the central model parameters, assign a corresponding version number to the current central model parameters; assign a corresponding parameter directory to the current version number; set the corresponding fine-tuning layer number and distribute the model parameters based on the current version number; store the current central model parameters and the fine-tuning layer number in the current parameter directory; and distribute the current version number, the distributed model parameters, and the fine-tuning layer number to each institutional server 31. Here, the fine-tuning layer number is a positive integer greater than or equal to 0.

[0133] The central server 21 is also used to, when receiving the institutional model parameters and the version number sent from each institutional server 31, store the current institutional model parameters in the corresponding parameter directory.

[0134] The central server 21 is also used to update the base model parameters according to each parameter directory to obtain a new version of the central model parameters.

[0135] 2) Corpus 22:

[0136] The corpus 22 of the embodiment of the present invention includes multiple first corpus texts.

[0137] 3) Central model library 23:

[0138] The inventory models in the central model library 23 of the embodiment of the present invention at least include a base large language model, a first feature projection model, a first generative adversarial network, and a first mask prediction model; the base large language model is a type of medical large language model.

[0139] A. The first feature projection model of the embodiment of the present invention is implemented based on the MLP model structure. The model input and output vectors of the first feature projection model are the corresponding first text feature vector and the first text mapping vector; among them, the first text feature vector is composed of multiple first word segmentation feature vectors; the first text mapping vector is composed of multiple first word segmentation mapping vectors, and the first word segmentation mapping vector corresponds one-to-one with the first word segmentation feature vector. The vector feature dimension of the first word segmentation mapping vector is higher than that of the first word segmentation feature vector, and the vector feature dimension of the first word segmentation mapping vector is consistent with the input vector feature dimension of the base large language model. The first feature projection model is used to perform high-dimensional vector mapping on the input first text feature vector and output the corresponding first text mapping vector.

[0140] B. The base large language model of the embodiment of the present invention is implemented based on the Med-BERT model. The model input and output vectors of the base large language model are the corresponding first text mapping vector and the first text encoding vector; the first text encoding vector is composed of multiple first word segmentation encoding vectors, and the first word segmentation encoding vector corresponds one-to-one with the first word segmentation mapping vector. The base large language model is used to perform feature encoding processing on the input first text mapping vector and output the corresponding first text encoding vector.

[0141] C. The first generative adversarial network of the embodiment of the present invention is implemented based on the GAN model. The first generative adversarial network includes a first generator and a first discriminator. The model input and output of the first generator are the corresponding first text mapping vector and the first text scrambling vector; the model input of the first discriminator is the corresponding first text mapping vector and the first text scrambling vector, and the model output is the corresponding first discrimination probability; the first text scrambling vector is composed of multiple first word segmentation scrambling vectors, and the first word segmentation scrambling vector corresponds one-to-one with the first word segmentation mapping vector; the first discrimination probability is a probability value between 0 and 1. The first generator is used to add random noise to each first word segmentation mapping vector of the first text mapping vector to obtain the corresponding first word segmentation scrambling vector, and all the first word segmentation scrambling vectors form the corresponding first text scrambling vector; the first discriminator is used to predict the true data probability of the input first text scrambling vector according to the input first text mapping vector and use the obtained prediction probability as the corresponding first discrimination probability.

[0142] D. The first mask prediction model is implemented based on the task head network of the mask prediction task of the BERT model. The model input of the first mask prediction model is the corresponding first text encoding vector and the first mask position index sequence; the first mask position index sequence is composed of one or more first mask position indexes, and each first mask position index corresponds to a first token encoding vector in the first text encoding vector; the model output of the first mask prediction model is the corresponding first probability vector sequence; the first probability vector sequence is composed of one or more first probability vectors, and the first probability vectors are in one-to-one correspondence with the first mask position indexes; the vector length of the first probability vector is the total number of tokens in the model vocabulary of the base large language model, and the first probability vector is composed of multiple first token probabilities, and the first token probabilities are in one-to-one correspondence with the tokens in the model vocabulary. The first mask prediction model is used to predict the token probability of the masked token corresponding to the current first mask position index according to the first token encoding vector corresponding to each first mask position index in the first text encoding vector to obtain the corresponding first probability vector, and the obtained first probability vectors form the corresponding first probability vector sequence.

[0143] As Figure 2 As shown in the module structure diagram of the central pre-training framework provided by the embodiments of the present invention, the central pre-training framework of the embodiments of the present invention includes a first preprocessing module, a first feature projection model, a base large language model, a first mask prediction model, and a first generative adversarial network. Among them, the input end of the first preprocessing module is used to receive the first text, and the output end is connected to the input end of the first feature projection model; the output end of the first feature projection model is respectively connected to the input ends of the base large language model and the first generative adversarial network; the output end of the base large language model is connected to the first input end of the first mask prediction model; the second input end of the first mask prediction model is used to receive the first training index sequence, and the output end is used to output the corresponding first probability vector sequence; the output end of the first generative adversarial network is used to output the corresponding first discrimination probability; the first training index sequence is composed of one or more first training indexes.

[0144] The first preprocessing module of the central pre-training framework is used to use the received first text as the corresponding current text and preprocess the current text to obtain the corresponding first text feature vector and send it to the first feature projection model;

[0145] Specifically: perform word segmentation on the current text according to the word segmentation rules of the base large language model to obtain the corresponding first word segmentation sequence; and based on the feature mapping rules, record the first nodes corresponding to each first word in the knowledge graph 1 as the corresponding first word nodes; and identify whether each first word node is empty. If so, record the first word corresponding to the current first word node as a non-term word, and if not, record the first word corresponding to the current first word node as a term word; and use some or all of the node attributes in the knowledge graph 1 as the corresponding node features; and form a corresponding first term feature vector from all the node features corresponding to each term word; and configure a zero vector for each non-term word as the corresponding first term feature vector; and encode each first word in the first word segmentation sequence according to the standard BERT model embedding method to obtain the corresponding first embedding encoding vector; and perform vector splicing on the first term feature vector and the first embedding encoding vector corresponding to each first word according to the vector splicing method to obtain the corresponding first word feature vector; and form the corresponding first text feature vector from all the obtained first word feature vectors and send it to the first feature projection model.

[0146] The first feature projection model of the central pre-training framework performs high-dimensional vector mapping on the first text feature vector and sends the obtained first text mapping vector to the base large language model and the first generative adversarial network.

[0147] The base large language model of the central pre-training framework performs feature encoding on the first text mapping vector and sends the obtained first text encoding vector to the first mask prediction model.

[0148] The first mask prediction model of the central pre-training framework uses the first training index sequence as the corresponding first mask position index sequence, and performs mask word segmentation probability prediction according to the current first mask position index sequence and the first text encoding vector and outputs the corresponding first probability vector sequence.

[0149] The first generative adversarial network of the central pre-training framework adds random noise to the first text mapping vector by the first generator to obtain the corresponding first text scrambled vector, and the first discriminator predicts the true data probability of the first text scrambled vector and outputs the corresponding first discriminant probability.

[0150] In a specific implementation manner of the embodiment of the present invention, the central server 21 is specifically configured to, when updating the corpus 22 through big data collection: obtain a plurality of corresponding first collected documents from the publicly available medical literature and clinical medical documents in the open field as a large data set; segment each of the first collected documents to obtain a plurality of first text segments; perform text cleaning on each of the first text segments according to the preset text cleaning rules to obtain the corresponding first cleaned text; perform stop word removal processing on each of the first cleaned texts to obtain the corresponding second cleaned text; perform deduplication processing on all the obtained second cleaned texts; and after the deduplication processing is completed, add each remaining second cleaned text as a corresponding first corpus text to the corpus 22. Here, the text cleaning rules of the embodiment of the present invention are conventional text cleaning rules, such as removing punctuation marks, numbers / letters, special symbols, correcting spelling / writing errors, etc.

[0151] In another specific implementation manner of the embodiment of the present invention, the central server 21 is specifically configured to, when performing a first pre-training according to the corpus 22 and the central pre-training framework:

[0152] Step A1, count the total number of the first corpus texts in the corpus 22 to obtain the corresponding total number N a ; and denote each of the first corpus texts as the corresponding first text t i , 1 ≤ index i ≤ N a ; set the current training stage as the first stage; and perform pre-tokenization processing on each of the first texts t i according to the tokenization rules of the base large language model to obtain the corresponding first pre-tokenization sequence sa i ;

[0153] Step A2, identify the current training stage; if the current training stage is the first stage, randomly mask and replace the tokens in each of the first pre-tokenization sequences sa i with the mask token "[MASK]" according to the preset first mask rate to obtain the corresponding second pre-tokenization sequence sb i ; if the current training stage is the second stage, mask and replace the tokens in each of the first pre-tokenization sequences sa i that are specifically of a certain disease type with the mask token "[MASK]" to obtain the corresponding second pre-tokenization sequence sb i ;

[0154] Here, the first mask rate is a preset ratio parameter, such as 15%, 35%, 88%, etc.;

[0155] Step A3, use the sequence index of each mask token in each of the second pre-tokenization sequences sb i as a corresponding first training index s i,n; and each second pre-segmentation sequence sb i All corresponding first training indexes s i,n Form a corresponding first training index sequence S i ; and based on each first training index s i,n Set a corresponding first label vector y i,n ; and each first training index sequence S i All corresponding first label vectors y i,n Form a corresponding first label vector sequence Y i ;

[0156] Among them, each first training index sequence S i The first training index s i,n The total number is recorded as N i , 1≤indexn≤N i ; First label vector y i,n The vector length of y is the total number of word segments in the model vocabulary of the base large language model, and the total number of word segments in the model vocabulary of the base large language model is recorded as W; the first label vector y i,n By multiple first label probabilities ρ i,n,w Composition, the first label probability ρ i,n,w One-to-one correspondence with the word segmentation of the model vocabulary, 1≤index w≤W; the first label vector y i,n There is only one first label probability ρ i,n,w is 1, and the rest are 0; the first label vector y i,n Specifically, the first label probability ρ is 1 i,n,w The corresponding word in the model vocabulary and the first training index s i,n In the first pre-segmentation sequence sa i The corresponding word matching in ;

[0157] Step A4: each first text t i and the corresponding first training index sequence S i As the corresponding first text and first training index sequence, input the central pre-training framework to perform a forward reasoning; and the first probability vector sequence output by this reasoning is used as the corresponding first prediction vector sequence The first text feature vector output by the first preprocessing module in this inference process is recorded as the corresponding first feature vector x i ;

[0158] Among them, each first prediction vector sequence By the corresponding N i The first prediction vector Composition; first prediction vector By W first predicted probabilities composition;

[0159] Step A5: From each first label vector sequence Y i and its corresponding first prediction vector sequence to form a corresponding first data group and from N a first feature vectors x i to form a corresponding first vector sequence {x i};

[0160] Step A6: Substitute the obtained N a first data groups into the preset first model loss function L1 for calculation to obtain the corresponding first loss value;

[0161] Among them, the first model loss function L1 is:

[0162]

[0163] Step A7: Identify whether the first loss value meets the preset first loss value range; if the first loss value meets the first loss value range, identify whether the current training stage is the first stage. If so, reset the current training stage to the second stage and return to Step A2. If not, go to Step A8; if the first loss value does not meet the first loss value range, based on the preset first model optimizer, modulate the model parameters of the first feature projection model, the base large language model, and the first mask prediction model in the central pre-training framework in the direction of minimizing the first model loss function L1, and return to Step A4 at the end of this round of modulation;

[0164] Among them, the first loss value range is a preset numerical range; the first model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0165] Step A8: Substitute the latest first vector sequence {x i} into the preset first discriminant loss L D1 and the first generation loss L G1 ; and based on the preset second model optimizer, modulate the model parameters of the first feature projection model and the first discriminator of the first generative adversarial network in the direction of minimizing the first discriminant loss L D1 ; and at the end of this round of modulation, based on the preset third model optimizer, modulate the model parameters of the first feature projection model and the first generator of the first generative adversarial network in the direction of minimizing the first generation loss L G1 and confirm the end of this training at the end of this round of modulation;

[0166] Among them, the first discrimination loss \(L\) D1 is:

[0167]

[0168] \(\theta\) 11 is the model parameter of the first feature projection model, and \(F\) 11 () is the inference function of the first feature projection model; \(F\) 11 (\(x\) i ;\(\theta\) 11 ) is used to represent the text mapping vector output by the first feature projection model under the condition that the projection model parameter is \(\theta\) 11 and the model input is the text feature vector \(x\) i ;

[0169] \(\theta\) 12 is the model parameter of the first discriminator of the first generative adversarial network, and \(F\) 12 () is the inference function of the first discriminator; \(F\) 12 [\(F\) 11 (\(x\) i ;\(\theta\) 11 );\(\theta\) 12 is used to represent the discrimination probability output by the first discriminator under the condition that the discriminator model parameter is \(\theta\) 12 and the discriminator input is the text mapping vector \(F\) 11 (\(x\) i ;\(\theta\) 11 );

[0170] The first generation loss \(L\) G1 is:

[0171]

[0172] \(\theta\) 13 is the model parameter of the first generator of the first generative adversarial network, and \(F\) 13 () is the inference function of the first generator; \(F\) 13 [\(F\) 11 (\(x\) i ;\(\theta\) 11 ),\(\theta\) 13 is used to represent the text scrambling vector output by the first generator under the condition that the generator model parameter is \(\theta\) 13 and the model input is the text mapping vector \(F\) 11 (\(x\) i ;\(\theta\) 11 ); \(F\) 12 {\(F\) 13 [\(F\) 11 (\(x\) i ;\(\theta\) 11 ),\(\theta\) 13 ;\(\theta\) 12} Used to represent when the model parameters of the discriminator are θ 12 and the input of the discriminator is the text scrambling vector F 13 [F 11 (x i ; θ 11 ), θ 13 , the discrimination probability output by the first discriminator under the condition;

[0173] The second model optimizer at least includes the Adam optimizer and the SGD optimizer;

[0174] The third model optimizer at least includes the Adam optimizer and the SGD optimizer.

[0175] In another specific implementation manner of the embodiment of the present invention, the central server 21 is specifically used for: when setting the corresponding fine-tuning layer number and distributing model parameters based on the current version number, identifying whether the version number matches the preset initial version number; if it matches, setting the fine-tuning layer number to 0 and using the central model parameters corresponding to the current version number as the corresponding distributed model parameters; if it does not match, using the layer number corresponding to the last Encoder layer of the base large language model as the fine-tuning layer number and using the model parameters corresponding to the last Encoder layer in the central model parameters corresponding to the current version number as the corresponding distributed model parameters. Here, the initial version number is a preset version number value.

[0176] In another specific implementation manner of the embodiment of the present invention, the central server 21 is specifically used for: when updating the base model parameters according to each parameter directory to obtain a new version of the central model parameters, using each parameter directory as the corresponding current directory; and using the fine-tuning layer number and central model parameters stored in the current directory as the corresponding current number and current central model parameters; and identifying the current number; if the current number is 0, using all the parameters of the current central model parameters as the current reference parameters; if the current number is greater than 0, using the model layer parameters corresponding to the current number in the current central model parameters as the current reference parameters; and identifying whether the total number of institutional model parameters stored in the current directory matches the total number of the institutional network 3. If so, performing model parameter fusion on all the institutional model parameters in the current directory to obtain the corresponding current fusion parameters, and replacing some or all of the parameters in the current central model parameters corresponding to the current reference parameters based on the current fusion parameters to obtain a new version of the central model parameters;

[0177] Among them, the model parameter fusion is specifically:

[0178] 1 ≤ institutional index k ≤ N org , θ k is the kth institutional model parameter in the current directory, θR is the current fusion parameter.

[0179] (3) Institution Network 3:

[0180] The institution network 3 of the embodiment of the present invention includes an institution server 31, a text set 32, and an institution model library 33. The institution server 31 is respectively connected to the knowledge graph 1, the text set 32, and the institution model library 33; each institution network 3 corresponds to a medical institution.

[0181] 1) Institution server 31:

[0182] The institution server 31 of the embodiment of the present invention is used to update the text set 32 through the data collection of this institution.

[0183] The institution server 31 is also used to construct a local fine-tuning framework and a local task framework according to the received feature mapping rules, the knowledge graph 1, and the institution model library 33.

[0184] The institution server 31 is also used to, when receiving a set of version numbers, downloaded model parameters, and fine-tuning layer numbers each time, perform a model parameter setting on the institution large language model in the institution model library 33 based on the fine-tuning layer number and the downloaded model parameters; then perform a local training based on the fine-tuning layer number, the text set 32, the local fine-tuning framework, and the local task framework; and at the end of this training, set the corresponding institution model parameters based on the current model parameters and the fine-tuning layer number of the institution large language model; and upload the institution model parameters and the version number to the central server 21.

[0185] 2) Text set 32:

[0186] The text set 32 of the embodiment of the present invention includes multiple first text records; the first text record includes a first medical text and a first classification label. The first classification label consists of multiple first label classification probabilities, and each first label classification probability corresponds to a preset disease type; only one first label classification probability in each first classification label is 1, and the rest of the first label classification probabilities are 0.

[0187] 3) Institution model library 33:

[0188] The inventory models in the institution model library 33 of the embodiment of the present invention at least include an institution large language model, a second feature projection model, a second generative adversarial network, a second mask prediction model, and an institution text classification model; the model structure of the institution large language model is the same as that of the base large language model.

[0189] A. The second feature projection model of the embodiment of the present invention is implemented based on the MLP model structure, and its model function is similar to that of the first feature projection model. The model input and output vectors of the second feature projection model are the corresponding second text feature vectors and second text mapping vectors; the second feature projection model is used to perform high-dimensional vector mapping on the input second text feature vectors and output the corresponding second text mapping vectors.

[0190] B. The institutional large language model of the embodiment of the present invention has the same model structure as the base large language model. The model input and output vectors of the institutional large language model are the corresponding second text mapping vectors and second text encoding vectors; the base large language model is used to perform feature encoding processing on the input second text mapping vectors and output the corresponding second text encoding vectors.

[0191] C. The second generative adversarial network of the embodiment of the present invention is implemented based on the GAN model, and its model function is similar to that of the first generative adversarial network. The second generative adversarial network includes a second generator and a second discriminator; the model input and output of the second generator are the corresponding second text mapping vectors and second text scrambled vectors; the model input of the second discriminator is the corresponding second text mapping vector and second text scrambled vector, and the model output is the corresponding second discrimination probability. The second generator is used to add random noise to each second token mapping vector of the second text mapping vector to obtain the corresponding second token scrambled vector, and all the second token scrambled vectors form the corresponding second text scrambled vector; the second discriminator is used to predict the true data probability of the input second text scrambled vector according to the input second text mapping vector and use the obtained prediction probability as the corresponding second discrimination probability.

[0192] D. The second mask prediction model is implemented based on the task head network of the mask prediction task of the BERT model, and its model function is similar to that of the first mask prediction model. The model input of the second mask prediction model is the corresponding second text encoding vector and second mask position index sequence; the second mask position index sequence is composed of one or more second mask position indexes; the model output of the second mask prediction model is the corresponding second probability vector sequence; the second probability vector sequence is composed of one or more second probability vectors, and the second probability vectors correspond one by one to the second mask position indexes; the vector length of the second probability vector is the total number of tokens in the model vocabulary of the institutional large language model, and the second probability vector is composed of multiple second token probabilities, and the second token probabilities correspond one by one to the tokens in the model vocabulary; the model vocabulary of the institutional large language model is consistent with the model vocabulary of the base large language model. The second mask prediction model is used to predict the token probability of the masked token corresponding to the current second mask position index according to the second token encoding vector corresponding to each second mask position index in the second text encoding vector to obtain the corresponding second probability vector, and all the obtained second probability vectors form the corresponding second probability vector sequence.

[0193] E. The institutional text classification model of the embodiments of the present invention is based on a multi-classification model structure. The model input and output of the institutional text classification model are corresponding second text encoding vectors and first classification probability vectors; the first classification probability vector consists of multiple first classification probabilities, and each first classification probability corresponds to a preset disease type. The institutional text classification model is used to perform classification prediction based on the second text encoding vector input by the model and output the corresponding first classification probability vector.

[0194] As Figure 3 As shown in the module structure diagram of the local fine-tuning framework provided by the embodiments of the present invention, the local fine-tuning framework of the embodiments of the present invention includes a second preprocessing module, a second feature projection model, an institutional large language model, a second mask prediction model, and a second generative adversarial network. Among them, the input end of the second preprocessing module is used to receive the second text, and the output end is connected to the input end of the second feature projection model; the output end of the second feature projection model is respectively connected to the input ends of the institutional large language model and the second generative adversarial network; the output end of the institutional large language model is connected to the first input end of the second mask prediction model; the second input end of the second mask prediction model is used to receive the second training index sequence, and the output end is used to output the corresponding second probability vector sequence; the output end of the second generative adversarial network is used to output the corresponding second discrimination probability; the second training index sequence consists of one or more second training indexes.

[0195] The second preprocessing module of the local fine-tuning framework is used to take the received second text as the corresponding current text and preprocess the current text to obtain the corresponding second text feature vector and send it to the second feature projection model;

[0196] Specifically: segment the current text according to the tokenization rules of the institutional large language model to obtain the corresponding second token sequence; based on the feature mapping rules, record the first nodes corresponding to each second token in the second token sequence in Knowledge Graph 1 as the corresponding second token nodes; identify whether each second token node is empty, if so, record the second token corresponding to the current second token node as a non-term token, otherwise record the second token corresponding to the current second token node as a term token; form a corresponding second term feature vector from all node features corresponding to each term token; configure a zero vector for each non-term token as the corresponding second term feature vector; encode each second token in the second token sequence according to the standard BERT model embedding method to obtain the corresponding second embedding encoding vector; concatenate the second term feature vectors and the second embedding encoding vectors corresponding to each second token in a vector concatenation manner to obtain the corresponding second token feature vector; form the corresponding second text feature vector from all the obtained second token feature vectors and send it to the second feature projection model. Here, the tokenization rules of the institutional large language model are consistent with those of the base large language model.

[0197] The second feature projection model of the local fine-tuning framework performs high-dimensional vector mapping on the second text feature vector and sends the obtained second text mapping vector to the institutional large language model and the second generative adversarial network.

[0198] The institutional large language model of the local fine-tuning framework performs feature encoding on the second text mapping vector and sends the obtained second text encoding vector to the second mask prediction model.

[0199] The second mask prediction model of the local fine-tuning framework takes the second training index sequence as the corresponding second mask position index sequence, and predicts the mask token probability according to the current second mask position index sequence and the second text encoding vector and outputs the corresponding second probability vector sequence.

[0200] The second generative adversarial network of the local fine-tuning framework adds random noise to the second text mapping vector by the second generator to obtain the corresponding second text scrambled vector, and the second discriminator predicts the real data probability of the second text scrambled vector and outputs the corresponding second discriminant probability.

[0201] Such as Figure 4As shown in the module structure diagram of the local task framework provided by the embodiments of the present invention, the local task framework of the embodiments of the present invention includes a second preprocessing module, a second feature projection model, an institutional large language model, and an institutional text classification model. Among them, the input end of the second preprocessing module is used to receive the third text, and the output end is connected to the input end of the second feature projection model; the output end of the second feature projection model is connected to the input end of the institutional large language model; the output end of the institutional large language model is connected to the input end of the institutional text classification model; the output end of the institutional text classification model is used to output the corresponding first classification probability vector.

[0202] The second preprocessing module of the local task framework takes the received third text as the corresponding current text, preprocesses the current text to obtain the corresponding second text feature vector, and sends it to the second feature projection model.

[0203] The second feature projection model of the local task framework performs high-dimensional vector mapping on the second text feature vector, and sends the obtained second text mapping vector to the institutional large language model.

[0204] The institutional large language model of the local task framework performs feature encoding on the second text mapping vector, and sends the obtained second text encoding vector to the institutional text classification model.

[0205] The institutional text classification model of the local task framework performs classification prediction based on the second text encoding vector and outputs the corresponding first classification probability vector.

[0206] In another specific implementation manner of the embodiments of the present invention, the institutional server 31 is specifically used for, when updating the text set 32 through the data collection of this institution: regularly collecting data of newly added electronic medical record documents within this medical institution to obtain a plurality of first medical record documents; performing de-identification processing on the patient privacy information of each first medical record document to obtain the corresponding first de-identified document; performing segmentation processing on each first de-identified document to obtain a plurality of second text segments; performing text cleaning on each second text segment according to the preset text cleaning rules to obtain the corresponding third cleaned text; performing stop word removal processing on each third cleaned text to obtain the corresponding fourth cleaned text; performing duplicate removal processing on all the obtained fourth cleaned texts; and after the duplicate removal processing is completed, taking each remaining fourth cleaned text as a corresponding first medical text; confirming the disease type corresponding to each first medical record document to obtain the corresponding first disease type; setting a corresponding first classification label according to each first disease type; forming a corresponding first text record from each first medical text and its corresponding first classification label; and adding all the obtained first text records to the text set 32;

[0207] Among them, the first classification label consists of multiple first label classification probabilities, and each first label classification probability corresponds to a preset disease type; among all the first classification labels, only the first label classification probability corresponding to the corresponding first disease type is 1, and the rest of the first label classification probabilities are 0.

[0208] In another specific implementation manner of the embodiment of the present invention, the institution server 31 is specifically configured to, when performing a first model parameter setting on the institution large language model in the institution model library 33 based on the fine-tuning layer number and the issued model parameters: identify the current fine-tuning layer number; if the current fine-tuning layer number is 0, set the overall model parameters of the institution large language model based on the issued model parameters; if the current fine-tuning layer number is greater than 1, set the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the institution large language model based on the issued model parameters.

[0209] In another specific implementation manner of the embodiment of the present invention, the institution server 31 is specifically configured to, when performing a first local training based on the fine-tuning layer number, the text set 32, the local fine-tuning framework, and the local task framework:

[0210] Step B1, identify the current fine-tuning layer number;

[0211] Step B2, if the current fine-tuning layer number is 0, form a corresponding first adjustable parameter set from all the model parameters of the second feature projection model, the institution large language model, the second mask prediction model, and the second generative adversarial network; and form a corresponding second adjustable parameter set from all the model parameters of the institution large language model and the institution text classification model;

[0212] Step B3, if the current fine-tuning layer number is greater than 1, only form a corresponding first adjustable parameter set from the overall mode parameters of the second feature projection model and the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the institution large language model; and only form a corresponding second adjustable parameter set from the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the institution large language model and the overall model parameters of the institution text classification model;

[0213] Step B4, and perform a first pre-training based on the text set 32, the local fine-tuning framework, and the first adjustable parameter set; and after the end of this pre-training, perform a task training based on the text set 32, the local task framework, and the second adjustable parameter set; and at the end of this task training, confirm the end of this local training.

[0214] In another specific implementation manner of the embodiment of the present invention, the institution server 31 is specifically configured to, when performing a first pre-training based on the text set 32, the local fine-tuning framework, and the first adjustable parameter set:

[0215] Step C1: Count the total number of the first medical texts in the text set 32 to obtain the corresponding total number N b ; and record each first medical text as the corresponding second text t j , 1≤indexj≤N b ; and segment each second text t according to the word segmentation rules of the organization's large language model j Perform pre-segmentation processing to obtain the corresponding third pre-segmentation sequence sc j ; And according to the preset second mask rate and mask segmentation "[MASK]" for each third pre-segmentation sequence sc j The word segmentation in is randomly masked and replaced to obtain the corresponding fourth pre-segmentation sequence sd j ;

[0216] Here, the second mask rate is a preset ratio parameter. The embodiment of the present invention requires that the second mask rate should not be less than the first mask rate.

[0217] Step C2: each fourth pre-segmentation sequence sd j The sequence index of each masked word in is used as a corresponding second training index s j,m ; and each fourth pre-segmentation sequence sd j All corresponding second training indexes s j,m Form a corresponding second training index sequence S j ; and based on each second training index s j,m Set a corresponding second label vector y j,m ; and each second training index sequence S j All corresponding second label vectors y j,m Form a corresponding second label vector sequence Y j ;

[0218] Among them, each second training index sequence S j The second training index s j,m The total number is recorded as N j , 1≤indexm≤N j ; The second label vector y j,m The vector length of y is the total number of word segments in the model vocabulary of the organization's large language model, which is consistent with the total number of word segments W in the model vocabulary of the base large language model; the second label vector y j,m By multiple second label probabilities ρ j,m,w Composition, the second label probability ρ j,m,w One-to-one correspondence with the word segmentation of the model vocabulary, 1≤index w≤W; the second label vector y j,m There is only one second label probability ρ j,m,w is 1, and the rest are 0; the second label vector y j,m The second label probability ρ is specifically 1j,m,w The corresponding word segmentation in the model vocabulary and the second training index s j,m In the third pre-word-segmented sequence sc j The corresponding word segmentations match;

[0219] Step C3: Input each second text t j And the corresponding second training index sequence S j As the corresponding second text and second training index sequence into the local fine-tuning framework for a forward inference; And use the second probability vector sequence output this time as the corresponding second prediction vector sequence And record the second text feature vector output by the second preprocessing module during this inference process as the corresponding second feature vector x j ; And record the second text encoding vector output by the institutional large language model during this inference process as the corresponding first encoding vector e j ;

[0220] Among them, each second prediction vector sequence Consists of the corresponding N j Second prediction vectors Compose; The second prediction vector Consists of W second prediction probabilities Compose;

[0221] Step C4: Combine each second label vector sequence Y j And its corresponding second prediction vector sequence To form a corresponding second data group And use N b Second feature vectors x j To form the corresponding second vector sequence {x j};

[0222] Step C5: Use each first encoding vector e j As the corresponding current encoding vector, and add noise to the current encoding vector to obtain the corresponding positive sample vector And record the remaining N b - 1 first encoding vectors e j All as the corresponding negative sample vectors 1 ≤ index u ≤ N b - 1; And use each first encoding vector e j And its corresponding one positive sample vector And N b - 1 negative sample vectors To form a corresponding third data group

[0223] Step C6, denote the parameter part of the first adjustable parameter set large language model as parameter θ l , and denote the corresponding downloaded model parameter of parameter θ L as parameter θ C ; and bring the obtained N a second and third data groups into the preset second model loss function L2 for calculation to obtain the corresponding second loss value;

[0224] Among them, the second model loss function L2 is:

[0225] L2 = L 21 + αL 22 + β||θ l - θ C || 2 ,

[0226]

[0227] α and β are two preset weight parameters, τ is two preset temperature coefficients, the loss function L 21 is implemented based on the cross-entropy loss function, and the loss function L 22 is implemented based on the contrastive loss function; ||θ l - θ C || 2 is the L2 regularization quantity of the latest parameter θ l and parameter θ C ; sim() is the cosine similarity function of vectors; is the cosine similarity between each first encoded vector e j and its corresponding positive sample vector , is the cosine similarity between the first encoded vector e j and its corresponding negative sample vectors ;

[0228] Step C7, identify whether the second loss value satisfies the preset second loss value range; if the second loss value satisfies the second loss value range, go to step C8; if the second loss value does not satisfy the second loss value range, based on the preset fourth model optimizer, perform a round of modulation on the model parameters in the first adjustable parameter set in the direction of minimizing the second model loss function L2, and return to step C3 at the end of this round of modulation;

[0229] Among them, the second loss value range is a preset numerical range; the fourth model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0230] Step C8, take the latest second vector sequence {x j}Introduce the preset second discriminant loss L D2 and the second generation loss L G2 ; and based on the preset fifth model optimizer towards making the second discriminant loss L D2 A round of modulation is performed on the model parameters in the first adjustable parameter set in the direction of reaching the minimum value; and at the end of this round of modulation, based on the preset sixth model optimizer, the model parameters in the first adjustable parameter set are modulated in the direction of reaching the minimum value; and at the end of this round of modulation ... G2 Perform a round of modulation on the model parameters in the first adjustable parameter set in the direction of reaching the minimum value; and confirm the completion of this training at the end of this round of modulation;

[0231] Among them, the second discriminant loss L D2 for:

[0232]

[0233] G is the first adjustable parameter set;

[0234] θ 21 is the model parameter of the second feature projection model; θ 21 ∩G is the adjustable parameter part corresponding to the second feature projection model in the first adjustable parameter set; F 21 () is the inference function of the second feature projection model; F 21 (x j θ 21 ) is used to represent the projection model parameter θ 21 And the model input is the text feature vector x j Under the condition of , the text mapping vector output by the second feature projection model;

[0235] θ 22 is the model parameter of the second discriminator of the second adversarial network; θ 22 ∩G is the adjustable parameter part corresponding to the second discriminator in the first adjustable parameter set; F 22 () is the inference function of the second discriminator; F 22 [F 21 (x j θ 21 );θ 22 ] is used to indicate that the model parameters of the discriminator are θ 22 And the discriminator input is the text mapping vector F 21 (x j θ 21 ) under the condition of, the discrimination probability output by the second discriminator;

[0236] Second generation loss L G2 for:

[0237]

[0238] θ 23 are the model parameters of the second generator of the second generative adversarial network; θ 23 ∩G is the adjustable parameter part corresponding to the second generator in the first adjustable parameter set; F 23 () is the inference function of the second generator; F 23 [F 21 (x j ; θ 21 ), θ 23 is used to represent the text scrambling vector output by the second generator under the condition that the generator model parameters are θ 23 and the model input is the text mapping vector F 21 (x j ; θ 21 ); F 22 {F 23 [F 21 (x j ; θ 21 ), θ 23 ; θ 22} is used to represent the discrimination probability output by the second discriminator under the condition that the discriminator model parameters are θ 22 and the discriminator input is the text scrambling vector F 23 [F 21 (x j ; θ 21 ), θ 23 ;

[0239] The fifth model optimizer includes at least Adam optimizer and SGD optimizer;

[0240] The sixth model optimizer includes at least Adam optimizer and SGD optimizer.

[0241] In another specific implementation manner of the embodiment of the present invention, the institution server 31 is specifically used for: when performing a task training based on the text set 32, the local task framework, and the second adjustable parameter set:

[0242] Step D1, count the total number of the first medical texts in the text set 32 to obtain the corresponding total number N c ; and denote each first medical text as the corresponding third text t h , 1 ≤ index h ≤ N c ; and perform pre-tokenization processing on each third text t h according to the tokenization rules of the institution large language model to obtain the corresponding fifth pre-tokenization sequence se h ; and denote the first classification label corresponding to each first medical text as the corresponding label vector Y h ;

[0243] Among them, the label vector Yh including multiple tag probabilities ρ h,v , 1 ≤ index v ≤ N d , N d being the total number of preset disease types;

[0244] Step D2: Take each third text t h as the corresponding third text and input it into the local task framework for a forward inference; and use the first classification probability vector output this time as the corresponding prediction vector and denote the second text mapping vector output by the second feature projection model during this inference process as the corresponding mapping vector X h ;

[0245] Among them, the prediction vector includes multiple prediction probabilities

[0246] Step D3: Combine each label vector Y h and its corresponding prediction vector to form a corresponding fourth data group and set a standard normal distribution space with the same feature dimension as the mapping vector X h and randomly sample N e space vectors from the standard normal distribution space and denote them as X g ; and pairwise combine the mapping vector X h and the space vector P g to obtain N c ×N e fifth data groups (X h , P g );

[0247] Among them, N e is a positive integer, and N e > N c ; 1 ≤ index g ≤ N e ;

[0248] Step D4: Denote the overall model parameters of the second feature projection model as parameter θ proj ; and substitute parameter θ proj , N c fourth data groups N c ×N e fifth data groups (X h , P g ) into the preset third model loss function L3 for calculation to obtain the corresponding third loss value;

[0249] Among them, the third model loss function L3 is:

[0250]

[0251] γ and λ are two preset weight parameters; 1 ≤ index h * ≤ N c and 1 ≤ index g * ≤ N e ; The loss function L 31 is implemented based on the cross-entropy loss function, and the loss function L 32 is implemented based on the maximum mean discrepancy loss function. is the Frobenius norm regularization term of the latest parameter θ proj ; f kernel () is a preset kernel function with two input quantities;

[0252] Step D5: Identify whether the third loss value meets the preset third loss value range; if the third loss value does not meet the third loss value range, then based on the preset seventh model optimizer, perform one round of modulation on the model parameters in the second adjustable parameter set in the direction of minimizing the third model loss function L3, and return to step D2 at the end of this round of modulation; if the third loss value meets the third loss value range, stop training and confirm that the training of this task is completed. Here, the third loss value range is a preset numerical range.

[0253] In another specific implementation manner of the embodiment of the present invention, the institution server 31 is specifically used for setting the corresponding institution model parameters based on the current model parameters of the institution large language model and the fine-tuning layer number: identifying the current fine-tuning layer number; if the current fine-tuning layer number is 0, then using the overall model parameters of the institution large language model as the corresponding institution model parameters; if the current fine-tuning layer number is greater than 1, then only using the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the institution large language model as the corresponding institution model parameters.

[0254] It should be noted that it should be understood that the division of each module of the above system is only a division of logical functions. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; they can also be partially implemented in the form of software called by processing elements and partially in the form of hardware. For example, the central server can be a separately established processing element, or a chip integrated in a device, equipment or server. In addition, it can also be stored in the memory of the device, equipment or server in the form of program code and called by a certain processing element of the device, equipment or server to implement the corresponding processing end function. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together or can be independently implemented. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, the processing steps of each module of the foregoing system can be completed by the integrated logic circuit in the processor element or the instructions in software form.

[0255] For example, these modules of the above system can be one or more integrated circuits configured to implement the foregoing processing steps. For example: one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), etc. Again, when a certain module of the above system is implemented in the form of a processing element scheduling program code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. Again, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0256] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the foregoing method embodiments are generated in whole or in part. The foregoing computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The foregoing computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the foregoing computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wirelessly (such as infrared, wireless, Bluetooth, microwave, etc.). The foregoing computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The foregoing available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0257] An embodiment of the present invention provides a federated learning system for a medical large language model. The system includes: a knowledge graph, a central network, and multiple institutional networks; the central network includes a central server, a corpus, and a central model library; the institutional network includes an institutional server, a text set, and an institutional model library. Among them: 1) The central server is used to set a global feature mapping rule for the custom terms of all medical institutions; it is also used to perform a pre-training on the base large language model; it is also used to assign / set a corresponding version number, parameter directory, fine-tuning layer number, and distribute model parameters for each version of the central model parameters generated, and store the current central model parameters and fine-tuning layer number in the corresponding directory, and send the current version number + distributed model parameters + fine-tuning layer number to each institutional server; it is also used to store the institutional model parameters sent by each institutional server in the corresponding directory and update the base model parameters according to each parameter directory. 2) The institutional server is used to construct a local fine-tuning framework and a local task framework according to the feature mapping rule, the knowledge graph, and the institutional model library; it is also used to perform a model parameter setting on the institutional large language model every time a set of version number + distributed model parameters + fine-tuning layer number is received, and perform a local training on the institutional large language model based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework, and send the latest institutional model parameters to the central server at the end of this training. The embodiment of the present invention provides a federated learning solution for training a medical large language model. Through the embodiment of the present invention, the generalization ability of the model can be continuously improved under the condition of non-sharing of medical data.

[0258] Those skilled in the art should also be able to further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0259] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0260] The specific embodiments described above further elaborate on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A federated learning system for a medical large language model, characterized in that, The system includes: a knowledge graph, a central network, and multiple institutional networks; The central network includes a central server, a corpus, and a central model library; The central server is used to configure the one-to-one correspondence between the custom terms of all medical institutions and the nodes of the knowledge graph to obtain the corresponding feature mapping rules and send them to the institutional servers of all the institutional networks; and initialize the base model parameters; The central server is also used to update the corpus through big data collection; The central server is also used to, when the initialization of the base model parameters is completed, perform a first model parameter setting on the base large language model of the central model library according to the base model parameters; and construct a central pre-training framework according to the feature mapping rules, the knowledge graph, and the central model library; and perform a first pre-training according to the corpus and the central pre-training framework; and use the current model parameters of the base large language model as the initial version of the central model parameters at the end of this training; The central server is also used to, when generating each version of the central model parameters, assign a corresponding version number to the current central model parameters; and assign a corresponding parameter directory to the current version number; and set the corresponding fine-tuning layer number and send the model parameters based on the current version number; and store the current central model parameters and the fine-tuning layer number in the current parameter directory; and send the current version number, the sent model parameters, and the fine-tuning layer number to each institutional server; the fine-tuning layer number is a positive integer greater than or equal to 0; The central server is also used to, when receiving the institutional model parameters and the version number sent by each institutional server, store the current institutional model parameters in the corresponding parameter directory; The central server is also used to update the base model parameters according to each parameter directory to obtain a new version of the central model parameters; The institutional network includes the institutional server, a text set, and an institutional model library; The institutional server is used to update the text set through the data collection of this institution; The institutional server is also used to construct a local fine-tuning framework and a local task framework according to the received feature mapping rules, the knowledge graph, and the institutional model library; The institutional server is also used to, when receiving each set of the version number, the sent model parameters, and the fine-tuning layer number, perform a first model parameter setting on the institutional large language model of the institutional model library based on the fine-tuning layer number and the sent model parameters; then perform a first local training based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework; and at the end of this training, set the corresponding institutional model parameters based on the current model parameters of the institutional large language model and the fine-tuning layer number; and send the institutional model parameters and the version number to the central server.

2. The federated learning system of the medical large language model according to claim 1, wherein The knowledge graph consists of a first node set and a first edge set; the first node set includes a plurality of first nodes; each of the first nodes corresponds to a standard medical term; each of the first nodes corresponds to a set of node attributes, and the node attributes at least include a term name, a term code, a term type, and a data type; the term name is a medical term name; the term code is a standard code of a specified medical standard system, and the medical standard system at least includes the ICD-10 standard system; the term type at least includes diseases, drugs, diagnoses, tests, examinations, and genes; the first edge set includes a plurality of first edges; each of the first edges is a directed edge for connecting two of the first nodes; the edge attribute of each of the first edges is a type of medical term association relationship. The corpus includes a plurality of first corpus texts. The text set includes a plurality of first text records; the first text record includes a first medical text and a first classification label; the first classification label consists of a plurality of first label classification probabilities, and each of the first label classification probabilities corresponds to a type of preset disease type; only one of the first label classification probabilities in each of the first classification labels is 1, and the rest of the first label classification probabilities are all 0. The inventory models in the central model library at least include the base large language model, the first feature projection model, the first generative adversarial network, and the first mask prediction model. The base large language model is a type of medical large language model. The inventory models in the institutional model library at least include the institutional large language model, the second feature projection model, the second generative adversarial network, the second mask prediction model, and the institutional text classification model; the model structure of the institutional large language model is the same as that of the base large language model.

3. The federated learning system of the medical large language model according to claim 2, wherein The first feature projection model is implemented based on the MLP model structure; the model input and output vectors of the first feature projection model are the corresponding first text feature vectors and first text mapping vectors; the first text feature vectors are composed of a plurality of first tokenization feature vectors; the first text mapping vectors are composed of a plurality of first tokenization mapping vectors, and the first tokenization mapping vectors correspond one-to-one with the first tokenization feature vectors. The vector feature dimension of the first tokenization mapping vector is higher than that of the first tokenization feature vector, and the vector feature dimension of the first tokenization mapping vector is consistent with the input vector feature dimension of the base large language model. The base large language model is implemented based on the Med-BERT model; the model input and output vectors of the base large language model are the corresponding first text mapping vectors and first text encoding vectors; the first text encoding vectors are composed of a plurality of first tokenization encoding vectors, and the first tokenization encoding vectors correspond one-to-one with the first tokenization mapping vectors. The first generative adversarial network is implemented based on the GAN model; the first generative adversarial network includes a first generator and a first discriminator; the model input and output of the first generator are the corresponding first text mapping vectors and first text scrambling vectors; the model input of the first discriminator is the corresponding first text mapping vector and the first text scrambling vector, and the model output is the corresponding first discrimination probability; the first text scrambling vector is composed of a plurality of first word segmentation scrambling vectors, and the first word segmentation scrambling vector corresponds one-to-one with the first word segmentation mapping vector; the first discrimination probability is a probability value between 0 and 1; The first mask prediction model is implemented based on the task head network of the mask prediction task of the BERT model; the model input of the first mask prediction model is the corresponding first text encoding vector and the first mask position index sequence; the first mask position index sequence is composed of one or more first mask position indexes, and each first mask position index corresponds to one first word segmentation encoding vector in the first text encoding vector; the model output of the first mask prediction model is the corresponding first probability vector sequence; the first probability vector sequence is composed of one or more first probability vectors, and the first probability vector corresponds one-to-one with the first mask position index; the vector length of the first probability vector is the total number of word segmentations of the model vocabulary of the base large language model, and the first probability vector is composed of a plurality of first word segmentation probabilities, and the first word segmentation probability corresponds one-to-one with the word segmentation of the model vocabulary; The second feature projection model is implemented based on the MLP model structure; the model input and output vectors of the second feature projection model are the corresponding second text feature vectors and second text mapping vectors; The institutional large language model has the same model structure as the base large language model; the model input and output vectors of the institutional large language model are the corresponding second text mapping vectors and second text encoding vectors; The second generative adversarial network is implemented based on the GAN model; the second generative adversarial network includes a second generator and a second discriminator; the model input and output of the second generator are the corresponding second text mapping vectors and second text scrambling vectors; the model input of the second discriminator is the corresponding second text mapping vector and the second text scrambling vector, and the model output is the corresponding second discrimination probability; The second mask prediction model is implemented based on the task head network of the mask prediction task of the BERT model, and its model function is similar to that of the first mask prediction model; the model input of the second mask prediction model is the corresponding second text encoding vector and the second mask position index sequence; the second mask position index sequence is composed of one or more second mask position indexes; the model output of the second mask prediction model is the corresponding second probability vector sequence; the second probability vector sequence is composed of one or more second probability vectors, and the second probability vector corresponds to the second mask position index one by one; the vector length of the second probability vector is the total number of word segmentations of the model vocabulary of the institutional large language model, and the second probability vector is composed of multiple second word segmentation probabilities, and the second word segmentation probability corresponds to the word segmentation of the model vocabulary one by one; the model vocabulary of the institutional large language model is consistent with the model vocabulary of the base large language model; The institutional text classification model is implemented based on a multi-classification model structure; the model input and output of the institutional text classification model are the corresponding second text encoding vector and the first classification probability vector; the first classification probability vector is composed of multiple first classification probabilities, and each first classification probability corresponds to a preset disease type; the institutional text classification model is used to perform classification prediction according to the second text encoding vector input by the model and output the corresponding first classification probability vector.

4. The federated learning system of the medical large language model according to claim 3, wherein The central pre-training framework includes a first preprocessing module, the first feature projection model, the base large language model, the first mask prediction model, and the first generative adversarial network; The input end of the first preprocessing module is used to receive the first text, and the output end is connected to the input end of the first feature projection model; the output end of the first feature projection model is respectively connected to the input ends of the base large language model and the first generative adversarial network; the output end of the base large language model is connected to the first input end of the first mask prediction model; the second input end of the first mask prediction model is used to receive the first training index sequence, and the output end is used to output the corresponding first probability vector sequence; the output end of the first generative adversarial network is used to output the corresponding first discrimination probability; The first training index sequence is composed of one or more first training indexes; The first preprocessing module is used to use the received first text as the corresponding current text and preprocess the current text to obtain the corresponding first text feature vector and send it to the first feature projection model, specifically: Perform word segmentation on the current text according to the word segmentation rules of the base large language model to obtain a corresponding first word segmentation sequence; and based on the feature mapping rule, record the first nodes corresponding to each first word in the first word segmentation sequence in the knowledge graph as corresponding first word segmentation nodes; and identify whether each of the first word segmentation nodes is empty, if so, record the first word corresponding to the current first word segmentation node as a non-term word, and if not, record the first word corresponding to the current first word segmentation node as a term word; and use some or all of the node attributes in the node attributes of the knowledge graph as corresponding node features; And form a corresponding first term feature vector from all the node features corresponding to each of the term words; and configure a zero vector for each of the non-term words as the corresponding first term feature vector; and encode each of the first words in the first word segmentation sequence according to the standard BERT model embedding method to obtain a corresponding first embedding coding vector; and perform vector splicing on the first term feature vector and the first embedding coding vector corresponding to each of the first words according to the vector splicing method to obtain a corresponding first word feature vector; and form a corresponding first text feature vector from all the obtained first word feature vectors and send it to the first feature projection model.

5. The federated learning system of the medical large language model according to claim 4, wherein The local fine-tuning framework includes a second preprocessing module, the second feature projection model, the institutional large language model, the second mask prediction model, and the second generative adversarial network; The input end of the second preprocessing module is used to receive a second text, and the output end is connected to the input end of the second feature projection model; the output end of the second feature projection model is respectively connected to the input ends of the institutional large language model and the second generative adversarial network; the output end of the institutional large language model is connected to the first input end of the second mask prediction model; the second input end of the second mask prediction model is used to receive a second training index sequence, and the output end is used to output a corresponding second probability vector sequence; the output end of the second generative adversarial network is used to output a corresponding second discrimination probability; the second training index sequence is composed of one or more second training indexes; The second preprocessing module is used to use the received second text as the corresponding current text and perform preprocessing on the current text to obtain a corresponding second text feature vector and send it to the second feature projection model, specifically: Tokenize the current text according to the tokenization rules of the institutional large language model to obtain a corresponding second token sequence; and based on the feature mapping rule, record the first nodes corresponding to the respective second tokens in the second token sequence in the knowledge graph as corresponding second token nodes; and identify whether each of the second token nodes is empty, if so, record the second token corresponding to the current second token node as a non-term token, otherwise record the second token corresponding to the current second token node as a term token; and form a corresponding second term feature vector from all the node features corresponding to each of the term tokens; and configure a zero vector for each of the non-term tokens as the corresponding second term feature vector; and encode each of the second tokens in the second token sequence according to the standard BERT model embedding method to obtain a corresponding second embedding coding vector; and perform vector concatenation on the second term feature vector and the second embedding coding vector corresponding to each of the second tokens according to the vector concatenation method to obtain a corresponding second token feature vector; and form a corresponding second text feature vector from all the obtained second token feature vectors and send it to the second feature projection model; the tokenization rules of the institutional large language model are consistent with the tokenization rules of the base large language model.

6. The federated learning system of the medical large language model according to claim 5, wherein the local task framework includes the second preprocessing module, the second feature projection model, the institutional large language model, and the institutional text classification model; the input end of the second preprocessing module is used to receive a third text, and the output end is connected to the input end of the second feature projection model; the output end of the second feature projection model is connected to the input end of the institutional large language model; the output end of the institutional large language model is connected to the input end of the institutional text classification model; the output end of the institutional text classification model is used to output a corresponding first classification probability vector.

7. The federated learning system of the medical large language model according to claim 4, characterized in that, The central server is specifically used for, when performing a first pre-training according to the corpus and the central pre-training framework: Step 71, count the total number of the first corpus texts in the corpus to obtain the corresponding total number N a ; and denote each of the first corpus texts as the corresponding first text t i , 1 ≤ index i ≤ N a ; and set the current training phase as the first phase; and perform pre-tokenization processing on each of the first texts t i according to the tokenization rule of the base large language model to obtain the corresponding first pre-tokenization sequence sa i ; Step 72, identify the current training stage; if the current training stage is the first stage, randomly mask and replace the word segments in each of the first pre-word-segmented sequences sa i with the corresponding second pre-word-segmented sequences sb according to a preset first masking rate and the masking token "[MASK]" i ; if the current training stage is the second stage, mask and replace the word segments in each of the first pre-word-segmented sequences sa i that are specifically of a certain disease type with the corresponding second pre-word-segmented sequences sb i ; Step 73, for each of the second pre-tokenized sequences sb i use the sequence index of each masked token as a corresponding first training index s i,n ; and from all the first training indices s i corresponding to each of the second pre-tokenized sequences sb i,n form a corresponding first training index sequence S i ; and based on each of the first training indices s i,n set a corresponding first label vector y i,n ; and from all the first label vectors y i corresponding to each of the first training index sequences S i,n form a corresponding first label vector sequence Y i ; Among them, each of the first training index sequences S i of the first training index s i,n has a total number denoted as N i , where 1 ≤ index n ≤ N i ; the vector length of the first label vector y i,n is the total number of word segments in the model vocabulary of the base large language model, and the total number of word segments in the model vocabulary of the base large language model is denoted as W; the first label vector y i,n consists of multiple first label probabilities ρ i,n,w , and the first label probability ρ i,n,w corresponds one-to-one with the word segments in the model vocabulary, where 1 ≤ index w ≤ W; only one of the first label probabilities ρ i,n in the first label vector y i,n,w is 1 and the rest are 0; the specific first label probability ρ i,n that is 1 in the first label vector y i,n,w has a corresponding word segment in the model vocabulary that matches the corresponding word segment of the first training index s i,n in the first pre-tokenized sequence sa i . Step 74, take each of the first texts t i and the corresponding first training index sequence S i as the corresponding first text and the first training index sequence and input them into the central pre-training framework for a forward inference; and take the first probability vector sequence output by this inference as the corresponding first prediction vector sequence and denote the first text feature vector output by the first preprocessing module during this inference process as the corresponding first feature vector x i ; Among them, each of the first prediction vector sequences is composed of the corresponding N i first prediction vectors ; the first prediction vector is composed of W first prediction probabilities . Step 75, from each of the first label vector sequences Y i and its corresponding first prediction vector sequence to form a corresponding first data group and by N a of the first feature vectors x i to form a corresponding first vector sequence {x i}; Step 76, taking the obtained N a first data groups and substituting them into a preset first model loss function L1 for calculation to obtain corresponding first loss values; wherein, the first model loss function L1 is: Step 77, identify whether the first loss value satisfies a preset first loss value range; if the first loss value satisfies the first loss value range, identify whether the current training stage is the first stage, if so, reset the current training stage to the second stage and return to step 72, if not, go to step 78; if the first loss value does not satisfy the first loss value range, based on a preset first model optimizer, modulate the model parameters of the first feature projection model, the base large language model, and the first mask prediction model in the central pre-training framework in the direction of minimizing the first model loss function L1 for one round, and return to step 74 at the end of this round of modulation; wherein, the first model optimizer includes at least an Adam optimizer and an SGD optimizer; Step 78, input the latest first vector sequence {x i} into the preset first discriminative loss L D1 and the first generative loss L G1 ; and based on the preset second model optimizer, perform one round of modulation on the model parameters of the first feature projection model in the center pre-training framework and the first discriminator of the first generative adversarial network in the direction of minimizing the first discriminative loss L D1 ; and at the end of this round of modulation, based on the preset third model optimizer, perform one round of modulation on the model parameters of the first feature projection model and the first generator of the first generative adversarial network in the direction of minimizing the first generative loss L G1 and confirm the end of this training at the end of this round of modulation; Among them, the first discriminant loss L D1 is: θ 11 is the model parameter of the first feature projection model, F 11 () is the inference function of the first feature projection model; F 11 (x i ; θ 11 ) is used to represent the text mapping vector output by the first feature projection model under the condition that the projection model parameter is θ 11 and the model input is the text feature vector x i ; θ 12 is the model parameter of the first discriminator of the first generative adversarial network, F 12 () is the inference function of the first discriminator; F 12 [F 11 (x i ; θ 11 ); θ 12 is used to represent that the model parameter of the discriminator is θ 12 and the discriminator input is the text mapping vector F 11 (x i ; θ 11 ), the discrimination probability output by the first discriminator; The first generation loss L G1 is as follows: θ 13 is the model parameter of the first generator of the first generative adversarial network, and F 13 () is the inference function of the first generator; F 13 [F 11 (x i ; θ 11 ), θ 13 is used to represent the text scrambling vector output by the first generator under the condition that the model parameter of the generator is θ 13 and the model input is the text mapping vector F 11 (x i ; θ 11 ); F 12 {F 13 [F 11 (x i ; θ 11 ), θ 13 ; θ 12} is used to represent the discrimination probability output by the first discriminator under the condition that the model parameter of the discriminator is θ 12 and the discriminator input is the text scrambling vector F 13 [F 11 (x i ; θ 11 ), θ 13 ; The second model optimizer at least includes an Adam optimizer and an SGD optimizer; The third model optimizer at least includes an Adam optimizer and an SGD optimizer.

8. The federated learning system of the medical large language model according to claim 1, characterized in that, The central server is specifically configured to, when setting the corresponding fine-tuning layer number and distributing model parameters based on the current version number: Identify whether the version number matches a preset initial version number; If it matches, set the fine-tuning layer number to 0, and use the central model parameters corresponding to the current version number as the corresponding distributed model parameters; if it does not match, use the layer number corresponding to the last Encoder layer of the base large language model as the fine-tuning layer number, and use the model parameters corresponding to the last Encoder layer in the central model parameters corresponding to the current version number as the corresponding distributed model parameters.

9. The federated learning system of the medical large language model according to claim 1, wherein, The central server is specifically configured to, when updating the base model parameters according to each parameter directory to obtain a new version of the central model parameters: Use each parameter directory as the corresponding current directory; use the fine-tuning layer number and the central model parameters stored in the current directory as the corresponding current number and current central model parameters; and identify the current number; If the current number is 0, use all the parameters of the current central model parameters as the current reference parameters; If the current number is greater than 0, use the model layer parameters corresponding to the current number in the current central model parameters as the current reference parameters; and identify whether the total number of the institutional model parameters stored in the current directory matches the total number of the institutional networks. If so, perform model parameter fusion on all the institutional model parameters in the current directory to obtain corresponding current fusion parameters, and replace some or all of the parameters in the current central model parameters corresponding to the current reference parameters based on the current fusion parameters to obtain a new version of the central model parameters; Among them, the model parameter fusion is specifically: 1 ≤ institutional index k ≤ N org , θ k is the k-th institutional model parameter within the current directory θ R is the current fusion parameter.

10. The federated learning system of the medical large language model according to claim 6, characterized in that, The institutional server is specifically configured to, when performing one local training based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework: Identify the current fine-tuning layer number; If the current fine-tuning layer number is 0, form a corresponding first adjustable parameter set from all the model parameters of the second feature projection model, the institutional large language model, the second mask prediction model, and the second generative adversarial network; and form a corresponding second adjustable parameter set from all the model parameters of the institutional large language model and the institutional text classification model; If the current fine-tuning layer number is greater than 1, only form the corresponding first adjustable parameter set from the overall mode parameters of the second feature projection model and the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the institutional large language model; and only form the corresponding second adjustable parameter set from the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the institutional large language model and the overall model parameters of the institutional text classification model; Perform a pre-training based on the text set, the local fine-tuning framework, and the first adjustable parameter set; and after the completion of this pre-training, perform a task training based on the text set, the local task framework, and the second adjustable parameter set; and at the end of this task training, confirm the end of the current local training.

11. The federated learning system of the medical large language model according to claim 10, characterized in that, The institutional server is specifically configured to, when performing a pre-training based on the text set, the local fine-tuning framework, and the first adjustable parameter set: Step 111, count the total number of the first medical texts in the text set to obtain the corresponding total number N b ; and denote each of the first medical texts as the corresponding second text t j , where 1 ≤ index j ≤ N b ; and perform pre-tokenization processing on each of the second texts t j according to the tokenization rules of the institutional large language model to obtain the corresponding third pre-tokenized sequence sc j ; and randomly mask and replace the tokens in each of the third pre-tokenized sequences sc j according to a preset second masking rate and the masking token "[MASK]" to obtain the corresponding fourth pre-tokenized sequence sd j ; Step 112, take the sequence index of each masked token in each of the fourth pre-tokenized sequences sd j as a corresponding second training index s j,m ; and form a corresponding second training index sequence S j from all the corresponding second training indices s j,m of each of the fourth pre-tokenized sequences sd j ; and set a corresponding second label vector y j,m based on each of the second training indices s j,m ; and form a corresponding second label vector sequence Y j from all the corresponding second label vectors y j,m of each of the second training index sequences S j ; Among them, each of the second training index sequences S j has the second training index s j,m , and the total number thereof is denoted as N j , where 1 ≤ index m ≤ N j ; the vector length of the second label vector y j,m is consistent with the total number of word segments in the model vocabulary of the institutional large language model and the total number of word segments W in the model vocabulary of the base large language model; the second label vector y j,m is composed of a plurality of second label probabilities ρ j,m,w , and the second label probability ρ j,m,w corresponds one-to-one to the word segments in the model vocabulary, where 1 ≤ index w ≤ W; only one of the second label probabilities ρ j,m in the second label vector y j,m,w is 1, and the rest are all 0; the second label probability ρ j,m that is specifically 1 in the second label vector y j,m,w has the corresponding word segment in the model vocabulary that matches the corresponding word segment of the second training index s j,m in the third pre-tokenized sequence sc j . Step 113, take each of the second texts t j and the corresponding second training index sequence S j as the corresponding second text and the second training index sequence and input them into the local fine-tuning framework for a forward inference; and take the second probability vector sequence output this time as the corresponding second prediction vector sequence and denote the second text feature vector output by the second preprocessing module during this inference process as the corresponding second feature vector x j ; and denote the second text encoding vector output by the institutional large language model during this inference process as the corresponding first encoding vector e j ; Among them, each of the second prediction vector sequences is composed of the corresponding N j second prediction vectors ; the second prediction vector is composed of W second prediction probabilities . Step 114, from each of the second tag vector sequences Y j and its corresponding second prediction vector sequence to form a corresponding second data group and by N b second feature vectors x j to form a corresponding second vector sequence {x j}; Step 115, take each of the first encoding vectors e j as the corresponding current encoding vector, and add noise to the current encoding vector to obtain the corresponding positive sample vector and denote the remaining N b −1 first encoding vectors e j as the corresponding negative sample vectors 1 ≤ index u ≤ N b −1; and from each of the first encoding vectors e j and its corresponding one positive sample vector and N b −1 negative sample vectors form a corresponding third data group Step 116, denote the parameter part of the institutional large language model in the first adjustable parameter set as parameter θ l , and denote the parameter θ L corresponding to the issued model parameter as parameter θ C ; and bring the obtained N a second and third data groups into the preset second model loss function L2 for calculation to obtain the corresponding second loss value; Wherein, the second model loss function L2 is: L2 = L 21 + αL 22 + β||θ l - θ C || 2 , α and β are two preset weight parameters, τ are two preset temperature coefficients, and the loss function is L 21 is implemented based on the cross-entropy loss function, and the loss function is L 22 is implemented based on the contrastive loss function; ||θ l -θ C || 2 is the latest parameter θ l and the parameter θ C is the L2 regularization of; sim() is the cosine similarity function of vectors; is the cosine similarity between each of the first encoded vectors e j and its corresponding positive sample vector , is the cosine similarity between the first encoded vector e j and each of its corresponding negative sample vectors ; Step 117, identify whether the second loss value satisfies a preset second loss value range; if the second loss value satisfies the second loss value range, go to step 118; if the second loss value does not satisfy the second loss value range, perform one round of modulation on the model parameters in the first adjustable parameter set in the direction of minimizing the second model loss function L2 based on a preset fourth model optimizer, and return to step 113 at the end of this round of modulation; Wherein, the fourth model optimizer includes at least an Adam optimizer and an SGD optimizer; Step 118, input the latest second vector sequence {x j} into the preset second discriminant loss L D2 and the second generation loss L G2 ; and based on the preset fifth model optimizer, perform one round of modulation on the model parameters in the first adjustable parameter set in the direction of minimizing the second discriminant loss L D2 ; and at the end of this round of modulation, based on the preset sixth model optimizer, perform one round of modulation on the model parameters in the first adjustable parameter set in the direction of minimizing the second generation loss L G2 ; and confirm the end of this training at the end of this round of modulation; Among them, the second discrimination loss L D2 is as follows: G is the first adjustable parameter set; θ 21 are the model parameters of the second feature projection model; θ 21 ∩G is the adjustable parameter part corresponding to the second feature projection model in the first adjustable parameter set; F 21 () is the inference function of the second feature projection model; F 21 (x j ; θ 21 ) is used to represent the text mapping vector output by the second feature projection model under the condition that the projection model parameters are θ 21 and the model input is the text feature vector x j ; θ 22 are the model parameters of the second discriminator of the second adversarial network; θ 22 ∩G is the adjustable parameter part corresponding to the second discriminator in the first adjustable parameter set; F 22 () is the inference function of the second discriminator; F 22 [F 21 (x j ; θ 21 ); θ 22 is used to represent the discrimination probability output by the second discriminator under the condition that the model parameters of the discriminator are θ 22 and the input of the discriminator is the text mapping vector F 21 (x j ; θ 21 ); The second generation loss L G2 is as follows: θ 23 are the model parameters of the second generator of the second generative adversarial network; θ 23 ∩G is the adjustable parameter part corresponding to the second generator in the first adjustable parameter set; F 23 () is the inference function of the second generator; F 23 [F 21 (x j ; θ 21 ), θ 23 is used to represent the text scrambling vector output by the second generator under the condition that the generator model parameters are θ 23 and the model input is the text mapping vector F 21 (x j ; θ 21 ); F 22 {F 23 [F 21 (x j ; θ 21 ), θ 23 ; θ 22} is used to represent the discrimination probability output by the second discriminator under the condition that the discriminator model parameters are θ 22 and the discriminator input is the text scrambling vector F 23 [F 21 (x j ; θ 21 ), θ 23 ; The fifth model optimizer includes at least an Adam optimizer and an SGD optimizer; The sixth model optimizer includes at least an Adam optimizer and an SGD optimizer.

12. The federated learning system of the medical large language model according to claim 10, characterized in that, The institutional server is specifically configured to, when performing a task training based on the text set, the local task framework, and the second adjustable parameter set: Step 121, count the total number of the first medical texts in the text set to obtain the corresponding total number N c ; and denote each of the first medical texts as the corresponding third text t h , 1 ≤ index h ≤ N c ; and perform pre-tokenization processing on each of the third texts t h according to the tokenization rules of the institutional large language model to obtain the corresponding fifth pre-tokenization sequence se h ; and denote the first classification label corresponding to each of the first medical texts as the corresponding label vector Y h ; Among them, the label vector Y h includes a plurality of label probabilities ρ h,v , where 1 ≤ index v ≤ N d , and N d is the total number of preset disease types; Step 122, input each of the third texts t h as the corresponding third text into the local task framework for a forward inference; and use the first classification probability vector output this time as the corresponding prediction vector and denote the second text mapping vector output by the second feature projection model during this inference process as the corresponding mapping vector X h ; Among them, the prediction vector includes a plurality of prediction probabilities Step 123, from each of the label vectors Y h and its corresponding prediction vector to form a corresponding fourth data group and set a standard normal distribution space with a feature dimension consistent with the mapping vector X h and randomly sample N e space vectors from the standard normal distribution space and denote them as X g ; and from the mapping vector X h and the space vector P g pairwise combination to obtain N c ×N e fifth data groups (X h , P g ); Wherein, N e is a positive integer, and N e > N c ; 1 ≤ index g ≤ N e ; Step 124, denote the overall model parameters of the second feature projection model as parameter θ proj ; and denote the parameter θ proj , N c of the fourth data groups , N c ×N e of the fifth data groups (X h , P g ) are input into the preset third model loss function L3 for calculation to obtain the corresponding third loss value; Wherein, the third model loss function L3 is: γ and λ are two pre-set weight parameters; 1 ≤ index h * ≤ N c , 1 ≤ index g * ≤ N e ; the loss function L 31 is implemented based on the cross-entropy loss function, and the loss function L 32 is implemented based on the maximum mean discrepancy loss function, is the Frobenius norm regularizer of the latest parameter θ proj ; f kernel () is a pre-set kernel function with two input quantities; Step 125, identify whether the third loss value satisfies a preset third loss value range; if the third loss value does not satisfy the third loss value range, perform one round of modulation on the model parameters in the second adjustable parameter set in the direction of minimizing the third model loss function L3 based on a preset seventh model optimizer, and return to step 122 at the end of this round of modulation; if the third loss value satisfies the third loss value range, stop the training and confirm the end of the current task training.

Images