A federated learning system of a medical large language model

Abstract

This invention relates to a federated learning system for a medical large-scale language model. The system includes a knowledge graph, a central network, and multiple institutional networks. The central network configures global feature mapping rules, performs initial pre-training on the base large-scale language model, generates and distributes a version number, distributed model parameters, and fine-tuning layer number for each new version of the central model parameters, aggregates institutional model parameters using the mean, and generates a new version of the central model parameters based on the aggregated parameters. The institutional networks perform local training upon receiving a set of version numbers, distributed model parameters, and fine-tuning layer numbers, and then distribute the trained institutional model parameters to the central network. This invention enables continuous improvement of the generalization ability of a medical large-scale language model even when medical data is not shared.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a federated learning system for a large medical language model. Background Technology

[0002] Large Language Models (LLMs) can perform various Natural Language Processing (NLP) tasks, such as text generation and text classification. Applying LLMs to the medical field (referred to as medical large language models) can improve the classification accuracy of medical documents (such as electronic medical records). Before using a medical large language model, it needs to be trained; insufficient training data will reduce the model's generalization ability. Currently, a large amount of medical data is mainly distributed among various medical institutions (such as hospitals, medical teaching / research institutions, etc.). Using traditional centralized training mechanisms would require centralized collection of medical data from all institutions, which is difficult to achieve in reality.

[0003] Federated learning is a technical framework for distributed model training. In principle, this framework allows for training a model's generalization ability even when institutions do not share data. In other words, theoretically, training a large-scale medical language model using this framework could continuously improve the model's generalization ability even without sharing medical data. How to train a large-scale medical language model using this framework is the technical problem this invention aims to solve. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a federated learning system for a medical large-scale language model. This system includes: a knowledge graph, a central network, and multiple institutional networks. The central network includes a central server, a corpus, and a central model library. The institutional networks include institutional servers, text sets, and institutional model libraries. Specifically: 1) The central server is used to set a global feature mapping rule for the custom terms of all medical institutions; it is also used to pre-train the base large-scale language model; it is also used to assign / set the corresponding version number, parameter directory, fine-tuning layer number, and distribute model parameters each time a version of the central model parameters is generated, and to store the current central model parameters and fine-tuning layer number in the corresponding directory, and to distribute the current version number + distributed model parameters + fine-tuning layer number to each institutional server; it is also used to store each institutional model parameter received from an institutional server in the corresponding directory, and to update the base model parameters according to each parameter directory. 2) The institution server is used to construct a local fine-tuning framework and a local task framework based on feature mapping rules, knowledge graphs, and the institution model library. It is also used to set model parameters for the institution's large language model each time a version number, distributed model parameters, and fine-tuning layer number are received. Based on the fine-tuning layer number, text set, local fine-tuning framework, and local task framework, the institution's large language model is trained locally, and the latest institution model parameters are sent to the central server at the end of the training. This invention provides a federated learning scheme for training a large medical language model. This invention allows for continuous improvement of the model's generalization ability even when medical data is not shared.

[0005] To achieve the above objectives, embodiments of the present invention provide a federated learning system for a medical large language model, the system comprising: a knowledge graph, a central network, and multiple institutional networks;

[0006] The central network includes a central server, a corpus, and a central model library;

[0007] The central server is used to configure the one-to-one correspondence between custom terms and knowledge graph nodes for all medical institutions, obtain the corresponding feature mapping rules, and distribute them to the institution servers of all the institution networks; and initialize the base model parameters.

[0008] The central server is also used to update the corpus through big data collection;

[0009] The central server is also used to, upon completion of the initialization of the base model parameters, set the model parameters of the base large language model in the central model library once according to the base model parameters; construct a central pre-training framework according to the feature mapping rules, the knowledge graph and the central model library; perform a pre-training based on the corpus and the central pre-training framework; and at the end of this training, use the current model parameters of the base large language model as the initial version of the central model parameters.

[0010] The central server is also used to assign a corresponding version number to the current central model parameters each time a new version of the central model parameters is generated; to assign a corresponding parameter directory to the current version number; to set a corresponding fine-tuning layer number and distributed model parameters based on the current version number; to store the current central model parameters and the fine-tuning layer number in the current parameter directory; and to distribute the current version number, the distributed model parameters, and the fine-tuning layer number to each of the institution servers; wherein the fine-tuning layer number is a positive integer greater than or equal to 0.

[0011] The central server is also used to store the current organizational model parameters into the corresponding parameter directory each time it receives organizational model parameters and version number from an organizational server.

[0012] The central server is also used to update the base model parameters according to each of the parameter directories to obtain a new version of the central model parameters;

[0013] The institution network includes the institution server, text set, and institution model library;

[0014] The organization's server is used to update the text set through data collection from this organization;

[0015] The institution server is also used to construct a local fine-tuning framework and a local task framework based on the received feature mapping rules, the knowledge graph, and the institution model library.

[0016] The organization server is also used to, upon receiving a set of version numbers, distributed model parameters, and fine-tuning layer numbers, set model parameters for the organization's large language model in the organization model library based on the fine-tuning layer number and the distributed model parameters; then, perform local training based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework; and at the end of this training, set the corresponding organization model parameters based on the current model parameters of the organization's large language model and the fine-tuning layer number; and send the organization model parameters and the version number to the central server.

[0017] Preferably, the knowledge graph comprises a first node set and a first edge set; the first node set includes multiple first nodes; each first node corresponds to a standard medical term; each first node corresponds to a set of node attributes, the node attributes including at least a term name, a term code, a term type, and a data type; the term name is a medical term name; the term code is a standard code of a specified medical standard system, the medical standard system including at least the ICD-10 standard system; the term type includes at least disease, drug, diagnosis, test, examination, and gene; the first edge set includes multiple first edges; each first edge is a directed edge used to connect two first nodes; the edge attribute of each first edge is a type of medical term association relationship;

[0018] The corpus includes multiple first-level texts;

[0019] The text set includes multiple first text records; each first text record includes a first medical text and a first classification label; each first classification label consists of multiple first label classification probabilities, and each first label classification probability corresponds to a preset disease type; among each first classification label, only one first label classification probability is 1, and the rest of the first label classification probabilities are 0.

[0020] The central model library contains at least the base large language model, the first feature projection model, the first generative adversarial network, and the first mask prediction model; the base large language model is a type of medical large language model.

[0021] The inventory models of the institutional model library include at least the institutional big language model, the second feature projection model, the second generative adversarial network, the second mask prediction model, and the institutional text classification model; the institutional big language model has the same model structure as the base big language model.

[0022] Preferably, the first feature projection model is implemented based on an MLP model structure; the model input and output vectors of the first feature projection model are the corresponding first text feature vector and first text mapping vector; the first text feature vector is composed of multiple first word segmentation feature vectors; the first text mapping vector is composed of multiple first word segmentation mapping vectors, the first word segmentation mapping vector corresponds one-to-one with the first word segmentation feature vector, the vector feature dimension of the first word segmentation mapping vector is higher than the vector feature dimension of the first word segmentation feature vector, and the vector feature dimension of the first word segmentation mapping vector is consistent with the input vector feature dimension of the base large language model;

[0023] The base language model is implemented based on the Med-BERT model; the model input and output vectors of the base language model are the corresponding first text mapping vector and first text encoding vector; the first text encoding vector is composed of multiple first word segmentation encoding vectors, and the first word segmentation encoding vector corresponds one-to-one with the first word segmentation mapping vector;

[0024] The first generative adversarial network is implemented based on a GAN model; the first generative adversarial network includes a first generator and a first discriminator; the model input and output of the first generator are the corresponding first text mapping vector and the first text scrambling vector; the model input of the first discriminator is the corresponding first text mapping vector and the first text scrambling vector, and the model output is the corresponding first discrimination probability; the first text scrambling vector is composed of multiple first word segmentation scrambling vectors, and the first word segmentation scrambling vector corresponds one-to-one with the first word segmentation mapping vector; the first discrimination probability is a probability value between 0 and 1.

[0025] The first mask prediction model is implemented based on the task head network of the BERT model's mask prediction task. The model input of the first mask prediction model is the corresponding first text encoding vector and the first mask position index sequence. The first mask position index sequence consists of one or more first mask position indices, and each first mask position index corresponds to one of the first word segmentation encoding vectors in the first text encoding vector. The model output of the first mask prediction model is the corresponding first probability vector sequence. The first probability vector sequence consists of one or more first probability vectors, and the first probability vector corresponds one-to-one with the first mask position index. The vector length of the first probability vector is the total number of word segments in the model vocabulary of the base large language model. The first probability vector consists of multiple first word segmentation probabilities, and the first word segmentation probabilities correspond one-to-one with the word segments in the model vocabulary.

[0026] The second feature projection model is implemented based on the MLP model structure; the model input and output vectors of the second feature projection model are the corresponding second text feature vector and the second text mapping vector.

[0027] The structure of the institutional large language model is consistent with that of the base large language model; the input and output vectors of the institutional large language model are the corresponding second text mapping vector and second text encoding vector;

[0028] The second generative adversarial network is implemented based on a GAN model; the second generative adversarial network includes a second generator and a second discriminator; the model input and output of the second generator are the corresponding second text mapping vector and the second text scrambling vector; the model input of the second discriminator is the corresponding second text mapping vector and the second text scrambling vector, and the model output is the corresponding second discriminant probability;

[0029] The second mask prediction model is implemented based on the task head network of the BERT model's mask prediction task, and its function is similar to that of the first mask prediction model. The model input of the second mask prediction model is the corresponding second text encoding vector and the second mask position index sequence. The second mask position index sequence consists of one or more second mask position indices. The model output of the second mask prediction model is the corresponding second probability vector sequence. The second probability vector sequence consists of one or more second probability vectors, and the second probability vectors correspond one-to-one with the second mask position indices. The vector length of the second probability vector is the total number of word segments in the model vocabulary of the institutional large language model. The second probability vector consists of multiple second word segmentation probabilities, and the second word segmentation probabilities correspond one-to-one with the word segments in the model vocabulary. The model vocabulary of the institutional large language model is consistent with the model vocabulary of the base large language model.

[0030] The institutional text classification model is implemented based on a multi-classification model structure; the model input and output of the institutional text classification model are the corresponding second text encoding vector and first classification probability vector; the first classification probability vector is composed of multiple first classification probabilities, each of which corresponds to a preset disease type; the institutional text classification model is used to perform classification prediction based on the second text encoding vector input to the model and output the corresponding first classification probability vector.

[0031] Preferably, the central pre-training framework includes a first preprocessing module, a first feature projection model, a base large language model, a first mask prediction model, and a first generative adversarial network;

[0032] The input of the first preprocessing module is used to receive the first text, and its output is connected to the input of the first feature projection model; the output of the first feature projection model is connected to the inputs of the base large language model and the first generative adversarial network respectively; the output of the base large language model is connected to the first input of the first mask prediction model; the second input of the first mask prediction model is used to receive the first training index sequence, and its output is used to output the corresponding first probability vector sequence; the output of the first generative adversarial network is used to output the corresponding first discrimination probability; the first training index sequence consists of one or more first training indices;

[0033] The first preprocessing module is used to take the received first text as the corresponding current text, preprocess the current text to obtain the corresponding first text feature vector, and send it to the first feature projection model, specifically:

[0034] The current text is segmented according to the segmentation rules of the aforementioned base language model to obtain the corresponding first segmentation sequence; and based on the feature mapping rules, the first node corresponding to each first segmentation in the first segmentation sequence in the knowledge graph is recorded as the corresponding first segmentation node; and it is identified whether each first segmentation node is empty. If it is, the first segmentation corresponding to the current first segmentation node is recorded as a non-term segmentation; otherwise, the first segmentation corresponding to the current first segmentation node is recorded as a term segmentation; and some or all of the node attributes in the knowledge graph are used as the corresponding node features; and each term segmentation... All the node features corresponding to a word form a corresponding first term feature vector; and each non-term word is configured with an all-zero vector as the corresponding first term feature vector; and each first word in the first word segmentation sequence is encoded according to the standard BERT model embedding method to obtain the corresponding first embedding encoding vector; and the first term feature vector and the first embedding encoding vector corresponding to each first word are concatenated according to the vector concatenation method to obtain the corresponding first word feature vector; and all the obtained first word feature vectors form the corresponding first text feature vector and are sent to the first feature projection model.

[0035] Preferably, the local fine-tuning framework includes a second preprocessing module, a second feature projection model, the institutional big language model, a second mask prediction model, and a second generative adversarial network;

[0036] The input of the second preprocessing module is used to receive the second text, and its output is connected to the input of the second feature projection model; the output of the second feature projection model is connected to the inputs of the institutional big language model and the second generative adversarial network, respectively; the output of the institutional big language model is connected to the first input of the second mask prediction model; the second input of the second mask prediction model is used to receive the second training index sequence, and its output is used to output the corresponding second probability vector sequence; the output of the second generative adversarial network is used to output the corresponding second discriminant probability; the second training index sequence consists of one or more second training indices.

[0037] The second preprocessing module is used to take the received second text as the corresponding current text, preprocess the current text to obtain the corresponding second text feature vector, and send it to the second feature projection model. Specifically:

[0038] The current text is segmented according to the segmentation rules of the aforementioned large language model to obtain a corresponding second segmentation sequence; and based on the feature mapping rules, the first node corresponding to each second segmentation in the second segmentation sequence in the knowledge graph is recorded as the corresponding second segmentation node; and it is identified whether each second segmentation node is empty. If it is, the second segmentation corresponding to the current second segmentation node is recorded as a non-term segmentation; otherwise, the second segmentation corresponding to the current second segmentation node is recorded as a term segmentation; and a corresponding second term feature vector is formed by the node features corresponding to each term segmentation; and a second term feature vector is generated for each term segmentation. For non-term segmentation, an all-zero vector is configured as the corresponding second term feature vector; and each second segment of the second segmentation sequence is encoded according to the standard BERT model embedding method to obtain the corresponding second embedding encoding vector; and the second term feature vector and the second embedding encoding vector corresponding to each second segment are concatenated according to the vector concatenation method to obtain the corresponding second segmentation feature vector; and the corresponding second text feature vector is composed of all the obtained second segmentation feature vectors and sent to the second feature projection model; the segmentation rules of the institutional big language model are consistent with the segmentation rules of the base big language model.

[0039] Preferably, the local task framework includes the second preprocessing module, the second feature projection model, the institutional big language model, and the institutional text classification model;

[0040] The input of the second preprocessing module is used to receive the third text, and the output is connected to the input of the second feature projection model; the output of the second feature projection model is connected to the input of the institutional big language model; the output of the institutional big language model is connected to the input of the institutional text classification model; the output of the institutional text classification model is used to output the corresponding first classification probability vector.

[0041] Preferably, the central server is specifically used during a pre-training process based on the corpus and the central pre-training framework:

[0042] Step 71: Calculate the total number of the first corpus texts in the corpus to obtain the corresponding total number N. a And each of the first corpus texts is recorded as the corresponding first text t. i 1 ≤ index i ≤ N aThe current training phase is set as the first phase; and each of the first texts is segmented according to the word segmentation rules of the base large language model. i Pre-segmentation is performed to obtain the corresponding first pre-segmented word sequence sa. i ;

[0043] Step 72: Identify the current training stage; if the current training stage is the first stage, then segment each of the first pre-segmented word sequences sa according to the preset first mask rate and mask word segmentation “[MASK]”. i The word segmentation in the sequence is randomly masked and replaced to obtain the corresponding second pre-segmentation sequence sb. i If the current training phase is the second phase, then based on the masked word segmentation “[MASK]”, each of the first pre-segmented word sequences sa i Each word segment representing a specific disease type is masked and replaced to obtain the corresponding second pre-segmentation sequence sb. i ;

[0044] Step 73, divide each of the second pre-segmented word sequences sb i The sequence index of each masked word segment in the image is used as a corresponding first training index s. i,n ; and each of the second presegmented word sequences sb i All corresponding first training indices s i,n Form a corresponding first training index sequence S i ; and based on each of the first training indices s i,n Set a corresponding first label vector y i,n ; and each of the first training index sequences S i All corresponding first label vectors y i,n Form a corresponding first label vector sequence Y i ;

[0045] Wherein, each of the first training index sequences S i The first training index s i,n The total number is denoted as N. i 1 ≤ index n ≤ N i The first label vector y i,n The length of the vector is equal to the total number of words in the model vocabulary of the base large language model, and the total number of words in the model vocabulary of the base large language model is denoted as W; the first label vector y i,n Composed of multiple first label probabilities ρ i,n,w Composition, the first label probability ρ i,n,w The word segmentation corresponds one-to-one with the word segmentation in the model vocabulary, where 1 ≤ index w ≤ W; the first label vector y i,n There is only one first label with probability ρi,n,w The first label vector y is 1, and all others are 0; i,n Specifically, the probability ρ of the first label is 1. i,n,w The corresponding word segmentation in the model vocabulary and the first training index s i,n In the first presegmented word sequence sa i Corresponding word segmentation matching in;

[0046] Step 74, transfer each of the first text t i and the corresponding first training index sequence S i The first text and the first training index sequence are input into the central pre-training framework for a forward inference; and the first probability vector sequence output from this inference is used as the corresponding first prediction vector sequence. The first text feature vector output by the first preprocessing module during this inference process is denoted as the corresponding first feature vector x. i ;

[0047] Wherein, each of the first prediction vector sequences From the corresponding N i First prediction vector Composition; the first prediction vector Given W first prediction probabilities composition;

[0048] Step 75, from each of the first label vector sequences Y i and its corresponding first prediction vector sequence Form a corresponding first data group And by N a The first feature vector x i Form the corresponding first vector sequence {x i};

[0049] Step 76, obtain N a The first data group Substitute the preset first model loss function L1 into the calculation to obtain the corresponding first loss value;

[0050] Wherein, the loss function L1 of the first model is:

[0051]

[0052] Step 77: Identify whether the first loss value meets the preset first loss value range; if the first loss value meets the first loss value range, identify whether the current training stage is the first stage; if so, reset the current training stage to the second stage and return to step 72; otherwise, proceed to step 78; if the first loss value does not meet the first loss value range, based on the preset first model optimizer, perform a round of modulation on the model parameters of the first feature projection model, the base large language model, and the first mask prediction model in the central pre-training framework in the direction of minimizing the first model loss function L1, and return to step 74 when the modulation ends.

[0053] The first model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0054] Step 78, the latest first vector sequence {x} i Substitute the preset first discriminant loss L D1 and the first generation loss L G1 ; and based on a preset second model optimizer, it moves towards making the first discriminative loss L D1 The model parameters of the first feature projection model in the central pre-trained framework and the first discriminator of the first generative adversarial network are modulated in one round to reach the minimum value; and at the end of this round of modulation, a preset third model optimizer is used to optimize the first generative loss L. G1 The model parameters of the first feature projection model and the first generator of the first generative adversarial network are modulated in one round to reach the minimum value, and the training is confirmed to be over when the modulation is completed.

[0055] Wherein, the first discrimination loss L D1 for:

[0056]

[0057] θ 11 F represents the model parameters of the first feature projection model. 11 () is the inference function of the first feature projection model; F 11 (x i ;θ 11 ) is used to represent the projection model with parameter θ. 11 And the model input is a text feature vector x i Under the given conditions, the text mapping vector output by the first feature projection model;

[0058] θ 12 For the model parameters of the first discriminator of the first generative adversarial network, F 12() is the inference function of the first discriminator; F 12 [F 11 (x i ;θ 11 );θ 12 ] is used to indicate that the model parameters of the discriminator are θ 12 The discriminator input is a text mapping vector F. 11 (x i ;θ 11 Under the condition that the first discriminator outputs the discrimination probability;

[0059] The first generation loss L G1 for:

[0060]

[0061] θ 13 For the model parameters of the first generator of the first generative adversarial network, F 13 () is the inference function of the first generator; F 13 [F 11 (x i ;θ 11 ),θ 13 ] is used to indicate that the model parameters in the generator are θ 13 And the model input is a text mapping vector F 11 (x i ;θ 11 Under the condition of ), the text scrambling vector output by the first generator; F 12 {F 13 [F 11 (x i ;θ 11 ),θ 13 ];θ 12} is used to represent the model parameters of the discriminator being θ. 12 The discriminator input is a text scrambling vector F. 13 [F 11 (x i ;θ 11 ),θ 13 Under the condition of ], the discrimination probability output by the first discriminator;

[0062] The second model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0063] The third model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0064] Preferably, the central server is specifically used when setting the corresponding fine-tuning layer number and distributing model parameters based on the current version number:

[0065] The system identifies whether the version number matches a preset initial version number. If they match, the fine-tuning layer number is set to 0, and the central model parameter corresponding to the current version number is used as the corresponding distributed model parameter. If they do not match, the layer number corresponding to the last Encoder layer of the base large language model is used as the fine-tuning layer number, and the model parameter in the central model parameter corresponding to the current version number that corresponds to the last Encoder layer is used as the corresponding distributed model parameter.

[0066] Preferably, the central server is specifically used when the base model parameters are updated according to each of the parameter directories to obtain a new version of the central model parameters:

[0067] Each parameter directory is designated as the corresponding current directory; the fine-tuning layer number and the central model parameter stored in the current directory are designated as the corresponding current number and current central model parameter; the current number is identified; if the current number is 0, all parameters of the current central model parameter are designated as the current baseline parameter; if the current number is greater than 0, the model layer parameter corresponding to the current number in the current central model parameter is designated as the current baseline parameter; the total number of the organizational model parameters stored in the current directory is identified as matching the total number of the organizational network; if so, all organizational model parameters in the current directory are fused to obtain the corresponding current fused parameter, and some or all parameters corresponding to the current baseline parameter in the current central model parameter are replaced based on the current fused parameter to obtain a new version of the central model parameter;

[0068] Specifically, the model parameter fusion is as follows:

[0069]

[0070] 1 ≤ Institutional Index k ≤ N org ,

[0071] θ k For the kth parameter of the mechanism model in the current directory,

[0072] θ R The current fusion parameters are defined as follows.

[0073] Preferably, the institution server is specifically used during a local training session based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework:

[0074] Identify the current fine-tuning layer number;

[0075] If the current fine-tuning layer number is 0, then the first adjustable parameter set is composed of all model parameters of the second feature projection model, the institutional big language model, the second mask prediction model, and the second generative adversarial network; and the second adjustable parameter set is composed of all model parameters of the institutional big language model and the institutional text classification model.

[0076] If the current fine-tuning layer number is greater than 1, then the first adjustable parameter set is composed only of the overall mode parameters of the second feature projection model and the model layer parameters of the overall model parameters of the institutional large language model that correspond to the current fine-tuning layer number; and the second adjustable parameter set is composed only of the model layer parameters of the overall model parameters of the institutional large language model that correspond to the current fine-tuning layer number and the overall model parameters of the institutional text classification model.

[0077] A pre-training process is performed based on the text set, the local fine-tuning framework, and the first adjustable parameter set; after the pre-training is completed, a task training process is performed based on the text set, the local task framework, and the second adjustable parameter set; and at the end of the task training, the local training is confirmed to be complete.

[0078] Furthermore, the institution server is specifically used during the pre-training process based on the text set, the local fine-tuning framework, and the first adjustable parameter set:

[0079] Step 111: Count the total number of the first medical texts in the text set to obtain the corresponding total number N. b ; and each of the first medical texts is recorded as the corresponding second text t. j 1 ≤ index j ≤ N b ; and according to the word segmentation rules of the aforementioned large language model, each of the second texts t j Pre-segmentation is performed to obtain the corresponding third pre-segmented word sequence sc. j ; and according to the preset second mask rate and mask segmentation "[MASK]", each of the third pre-segmented word sequences sc j The word segments in the sequence are randomly masked and replaced to obtain the corresponding fourth pre-segmentation sequence sd. j ;

[0080] Step 112, divide each of the fourth pre-segmented word sequences sd j The sequence index of each masked word segment in the image is used as a corresponding second training index s. j,m ; and by each of the aforementioned fourth pre-segmentation sequences sd j All corresponding second training indices s j,m Form a corresponding second training index sequence S j; and based on each of the second training indices s j,m Set a corresponding second label vector y j,m ; and by each of the second training index sequences S j All corresponding second label vectors y j,m Form a corresponding second label vector sequence Y j ;

[0081] Wherein, each of the second training index sequences S j The second training index s j,m The total number is denoted as N. j 1 ≤ index m ≤ N j The second label vector y j,m The length of the vector is consistent with the total number of word segments in the model vocabulary of the institutional large language model and the total number of word segments W in the model vocabulary of the base large language model; the second label vector y j,m Composed of multiple second label probabilities ρ j,m,w Composition, the second label probability ρ j,m,w The word segmentation corresponds one-to-one with the word segmentation in the model vocabulary, where 1 ≤ index w ≤ W; the second label vector y j,m There is only one second label probability ρ j,m,w The first label vector is 1, and the rest are 0; the second label vector y j,m The second label probability ρ, specifically 1, is... j,m,w The corresponding word segmentation in the model vocabulary and the second training index s j,m In the third pre-segmentation sequence sc j Corresponding word segmentation matching in;

[0082] Step 113, transfer each of the second text t j and the corresponding second training index sequence S j The second text and the second training index sequence are input into the local fine-tuning framework for a forward inference; and the second probability vector sequence output in this iteration is used as the corresponding second prediction vector sequence. The second text feature vector output by the second preprocessing module during this inference process is denoted as the corresponding second feature vector x. j The second text encoding vector output by the institutional large language model during this inference process is denoted as the corresponding first encoding vector e. j ;

[0083] Among them, each of the second prediction vector sequences From the corresponding N j The second prediction vector Composition; the second prediction vector Given W second prediction probabilities composition;

[0084] Step 114, from each of the second label vector sequences Y j and its corresponding second prediction vector sequence Form a corresponding second data group And by N b The second feature vector x j Form the corresponding second vector sequence {x j};

[0085] Step 115, convert each of the first encoding vectors e j This serves as the corresponding current encoding vector, and noise is added to the current encoding vector to obtain the corresponding positive sample vector. And the remaining N besides the current encoding vector b -1 of the first encoded vector e j Each is denoted as the corresponding negative sample vector. 1≤index u≤N b -1; and by each of the first encoded vectors e j and its corresponding positive sample vector and N b -1 of the negative sample vectors Form a corresponding third data group

[0086] Step 116: Denote the parameter portion of the mechanism-wide language model in the first adjustable parameter set as parameter θ. l , the parameter θ L The corresponding model parameters are denoted as parameter θ. C ; and the obtained N a The second and third data groups The corresponding second loss value is obtained by substituting the preset second model loss function L2 into the calculation.

[0087] Wherein, the loss function L2 of the second model is:

[0088] L2 = L 21 +αL 22 +β||θ l -θ C || 2 ,

[0089]

[0090] α and β are two pre-set weighting parameters, τ is two pre-set temperature coefficients, and the loss function L 21Implemented based on the cross-entropy loss function, loss function L 22 Implemented based on the contrastive loss function; ||θ l -θ C || 2 For the latest parameter θ l With the parameter θ C The L2 regularization; sim() is the cosine similarity function of vectors; For each of the first encoding vectors e j The corresponding positive sample vector cosine similarity, For the first encoding vector e j Each of the corresponding negative sample vectors Cosine similarity;

[0091] Step 117: Identify whether the second loss value meets the preset second loss value range; if the second loss value meets the second loss value range, proceed to step 118; if the second loss value does not meet the second loss value range, perform a round of modulation on the model parameters in the first adjustable parameter set based on the preset fourth model optimizer in the direction of minimizing the second model loss function L2, and return to step 113 when the current round of modulation ends.

[0092] The fourth model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0093] Step 118, the latest second vector sequence {x} j Substitute the preset second discriminant loss L D2 Second generation loss L G2 ; and based on the preset fifth model optimizer, it moves towards making the second discriminative loss L D2 The model parameters in the first adjustable parameter set are modulated once in the direction that reaches the minimum value; and at the end of this modulation round, the second generation loss L is then modulated based on the preset sixth model optimizer. G2 The model parameters in the first adjustable parameter set are modulated once in the direction that reaches the minimum value; and the training ends when the modulation ends.

[0094] Wherein, the second discriminant loss L D2 for:

[0095]

[0096] G is the first set of adjustable parameters;

[0097] θ 21 The model parameters for the second feature projection model; θ21 ∩G represents the adjustable parameter portion corresponding to the second feature projection model in the first adjustable parameter set; F 21 () is the inference function of the second feature projection model; F 21 (x j ;θ 21 ) is used to represent the projection model with parameter θ. 21 And the model input is a text feature vector x j Under the condition that the second feature projection model outputs the text mapping vector;

[0098] θ 22 θ represents the model parameters of the second discriminator in the second adversarial network. 22 ∩G represents the adjustable parameter portion corresponding to the second discriminator in the first adjustable parameter set; F 22 () is the inference function of the second discriminator; F 22 [F 21 (x j ;θ 21 );θ 22 ] is used to indicate that the model parameters of the discriminator are θ 22 The discriminator input is a text mapping vector F. 21 (x j ;θ 21 Under the condition of ), the discrimination probability output by the second discriminator;

[0099] The second generation loss L G2 for:

[0100]

[0101] θ 23 θ represents the model parameters of the second generator in the second generative adversarial network. 23 ∩G represents the adjustable parameter portion corresponding to the second generator in the first adjustable parameter set; F 23 () is the inference function of the second generator; F 23 [F 21 (x j ;θ 21 ),θ 23 ] is used to indicate that the generator model parameter is θ 23 And the model input is a text mapping vector F 21 (x j ;θ 21 Under the condition of ), the text scrambling vector output by the second generator; F 22 {F 23 [F 21 (x j ;θ 21),θ 23 ];θ 22} is used to represent the discriminator model parameters being θ. 22 The discriminator input is a text scrambling vector F. 23 [F 21 (x j ;θ 21 ),θ 23 Under the condition of ], the discrimination probability output by the second discriminator;

[0102] The fifth model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0103] The sixth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0104] Furthermore, the institution server is specifically used during the task training based on the text set, the local task framework, and the second adjustable parameter set:

[0105] Step 121: Count the total number of the first medical texts in the text set to obtain the corresponding total number N. c ; and record each of the first medical texts as the corresponding third text t. h 1 ≤ index h ≤ N c ; and according to the word segmentation rules of the aforementioned institutional large language model, each of the aforementioned third texts t h Pre-segmentation is performed to obtain the corresponding fifth pre-segmented word sequence se. h And denote the first classification label corresponding to each of the first medical texts as the corresponding label vector Y. h ;

[0106] Wherein, the label vector Y h Includes multiple label probabilities ρ h,v 1 ≤ index v ≤ N d N d The total number of preset disease types;

[0107] Step 122, transfer each of the third texts t h The corresponding third text is input into the local task framework for a forward inference; and the first classification probability vector output in this instance is used as the corresponding prediction vector. The second text mapping vector output by the second feature projection model during this inference process is denoted as the corresponding mapping vector X. h ;

[0108] Wherein, the prediction vector Includes multiple predicted probabilities

[0109] Step 123, from each of the stated label vectors Y h and the corresponding prediction vector Form a corresponding fourth data group And set a feature dimension and the mapping vector X h A consistent standard normal distribution space, and N randomly sampled from said standard normal distribution space. e Let X be a spatial vector. g ; and by the mapping vector X h and the space vector P g Pairwise combinations yield N c ×N e The fifth data set (X) h ,P g );

[0110] Where, N e Let N be a positive integer, and N be a positive integer. e >N c ; 1 ≤ index g ≤ N e ;

[0111] Step 124: Denote the overall model parameters of the second feature projection model as parameter θ. proj ; and the parameter θ proj N c The fourth data group N c ×N e The fifth data group (X) h ,P g Substitute the input into the preset third model loss function L3 to calculate the corresponding third loss value;

[0112] The third model loss function L3 is:

[0113]

[0114] γ and λ are two pre-set weight parameters; 1 ≤ index h * ≤N c , 1≤index g * ≤N e Loss function L 31 Implemented based on the cross-entropy loss function, loss function L 32 Implemented based on the maximum mean difference loss function. For the latest parameter θ proj The Frobenius norm regularity; f kernel () is a pre-defined kernel function with two input variables;

[0115] Step 125: Identify whether the third loss value meets the preset range of the third loss value; if the third loss value does not meet the range of the third loss value, then based on the preset seventh model optimizer, perform a round of modulation on the model parameters in the second adjustable parameter set in the direction of minimizing the third model loss function L3, and return to step 122 when the modulation ends; if the third loss value meets the range of the third loss value, then stop training and confirm the end of the training for this task.

[0116] This invention provides a federated learning system for a medical large language model. The system includes: a knowledge graph, a central network, and multiple institutional networks. The central network includes a central server, a corpus, and a central model library. The institutional networks include institutional servers, text sets, and institutional model libraries. Specifically: 1) The central server is used to set a global feature mapping rule for custom terms from all medical institutions; it is also used to pre-train the base large language model; it is also used to assign / set a corresponding version number, parameter directory, fine-tuning layer number, and distribute model parameters each time a version of the central model parameters is generated, and to store the current central model parameters and fine-tuning layer number in the corresponding directory, and to distribute the current version number + distributed model parameters + fine-tuning layer number to each institutional server; it is also used to store each institutional model parameter received from an institutional server in the corresponding directory, and to update the base model parameters according to each parameter directory. 2) The institution server is used to construct a local fine-tuning framework and a local task framework based on feature mapping rules, knowledge graphs, and the institution model library. It is also used to set model parameters for the institution's large language model each time a version number, distributed model parameters, and fine-tuning layer number are received. Furthermore, it performs local training on the institution's large language model based on the fine-tuning layer number, text set, local fine-tuning framework, and local task framework, and sends the latest institution model parameters to the central server at the end of this training. This invention provides a federated learning scheme for training a medical large language model, which can continuously improve the model's generalization ability even without sharing medical data. Attached Figure Description

[0117] Figure 1 A module structure diagram of a federated learning system for a medical large language model provided in an embodiment of the present invention;

[0118] Figure 2 A module structure diagram of the central pre-training framework provided in the embodiments of the present invention;

[0119] Figure 3 A module structure diagram of the local fine-tuning framework provided in the embodiments of the present invention;

[0120] Figure 4This is a module structure diagram of the local task framework provided in an embodiment of the present invention. Detailed Implementation

[0121] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0122] This invention provides a federated learning system for a medical large language model, such as... Figure 1 The module structure diagram of a federated learning system for a medical big language model provided in an embodiment of the present invention is shown, which mainly includes: knowledge graph 1, central network 2 and multiple institutional networks 3.

[0123] (I) Knowledge Graph 1:

[0124] The knowledge graph 1 of this invention comprises a first node set and a first edge set.

[0125] The first node set includes multiple first nodes; each first node corresponds to a standard medical term; each first node corresponds to a set of node attributes, which include at least term name, term code, term type, and data type; the term name is a medical term name; the term code is a standard code of a specified medical standard system, which includes at least the ICD-10 standard system; the term type includes at least disease, drug, diagnosis, test, examination, and gene; the first edge set includes multiple first edges; each first edge is a directed edge used to connect two first nodes; the edge attribute of each first edge is a type of medical term association relationship.

[0126] (II) Central Network 2:

[0127] The central network 2 in this embodiment of the invention includes a central server 21, a corpus 22, and a central model library 23. The central server 21 is connected to the knowledge graph 1, the corpus 22, the central model library 23, and the institutional servers 31 of each institutional network 3.

[0128] 1) Central Server 21:

[0129] In this embodiment of the invention, the central server 21 is used to configure the one-to-one correspondence between the custom terms of all medical institutions and the nodes of the knowledge graph 1 to obtain the corresponding feature mapping rules, which are then distributed to the institution servers 31 of all institution networks 3; and to initialize the base model parameters.

[0130] The central server 21 is also used to update the corpus 22 through big data collection.

[0131] The central server 21 is also used to set the model parameters of the base large language model in the central model library 23 according to the base model parameters when the base model parameter initialization is completed; and to build the central pre-training framework according to the feature mapping rules, knowledge graph 1 and central model library 23; and to perform a pre-training according to the corpus 22 and the central pre-training framework; and to use the current model parameters of the base large language model as the initial version of the central model parameters at the end of this training.

[0132] The central server 21 is also used to assign a corresponding version number to the current central model parameters each time a new version of the central model parameters is generated; to assign a corresponding parameter directory to the current version number; to set the corresponding fine-tuning layer number and the distributed model parameters based on the current version number; to store the current central model parameters and the fine-tuning layer number in the current parameter directory; and to distribute the current version number, the distributed model parameters, and the fine-tuning layer number to each institution server 31. Here, the fine-tuning layer number is a positive integer greater than or equal to 0.

[0133] The central server 21 is also used to store the current organizational model parameters into the corresponding parameter directory each time it receives organizational model parameters and version numbers from an organizational server 31.

[0134] The central server 21 is also used to update the base model parameters according to the various parameter directories to obtain a new version of the central model parameters.

[0135] 2) Corpus 22:

[0136] The corpus 22 of this invention includes multiple first corpus texts.

[0137] 3) Central Model Library 23:

[0138] The central model library 23 of this invention includes at least a base large language model, a first feature projection model, a first generative adversarial network, and a first mask prediction model; the base large language model is a type of medical large language model.

[0139] A. The first feature projection model in this embodiment of the invention is implemented based on an MLP model structure. The input and output vectors of the first feature projection model are the corresponding first text feature vector and first text mapping vector; wherein, the first text feature vector is composed of multiple first word segmentation feature vectors; the first text mapping vector is composed of multiple first word segmentation mapping vectors, and the first word segmentation mapping vector corresponds one-to-one with the first word segmentation feature vector. The vector feature dimension of the first word segmentation mapping vector is higher than that of the first word segmentation feature vector, and the vector feature dimension of the first word segmentation mapping vector is consistent with the input vector feature dimension of the base large language model. The first feature projection model is used to perform high-dimensional vector mapping on the input first text feature vector and output the corresponding first text mapping vector.

[0140] B. The base-based large language model in this embodiment of the invention is implemented based on the Med-BERT model. The input and output vectors of the base-based large language model are the corresponding first text mapping vector and first text encoding vector; the first text encoding vector is composed of multiple first word segmentation encoding vectors, and the first word segmentation encoding vector corresponds one-to-one with the first word segmentation mapping vector. The base-based large language model is used to perform feature encoding processing on the input first text mapping vector and output the corresponding first text encoding vector.

[0141] C. The first generative adversarial network in this embodiment of the invention is implemented based on a GAN model. The first generative adversarial network includes a first generator and a first discriminator. The model input and output of the first generator are the corresponding first text mapping vector and the first text scrambling vector; the model input of the first discriminator is the corresponding first text mapping vector and the first text scrambling vector, and the model output is the corresponding first discriminant probability; the first text scrambling vector is composed of multiple first word segmentation scrambling vectors, and the first word segmentation scrambling vector corresponds one-to-one with the first word segmentation mapping vector; the first discriminant probability is a probability value between 0 and 1. The first generator is used to add random noise to each of the first word segmentation mapping vectors of the first text mapping vector to obtain the corresponding first word segmentation scrambling vector, and the corresponding first text scrambling vector is composed of all the first word segmentation scrambling vectors; the first discriminator is used to predict the true data probability of the input first text scrambling vector based on the input first text mapping vector and use the obtained predicted probability as the corresponding first discriminant probability.

[0142] D. The first mask prediction model is implemented using the task head network of the BERT model for mask prediction. The input to the first mask prediction model is the corresponding first text encoding vector and the first mask position index sequence. The first mask position index sequence consists of one or more first mask position indices, each corresponding to a first word segmentation encoding vector in the first text encoding vector. The output of the first mask prediction model is the corresponding first probability vector sequence. The first probability vector sequence consists of one or more first probability vectors, each corresponding to a first mask position index. The length of the first probability vector is the total number of words in the base large language model's vocabulary. The first probability vector consists of multiple first word segmentation probabilities, each corresponding to a word segmentation in the model vocabulary. The first mask prediction model is used to predict the word segmentation probability of the masked word corresponding to the current first mask position index based on the first word segmentation encoding vectors corresponding to each first mask position index in the first text encoding vector, and then uses all the obtained first probability vectors to form the corresponding first probability vector sequence.

[0143] like Figure 2 As shown in the module structure diagram of the central pre-training framework provided in this embodiment of the invention, the central pre-training framework includes a first preprocessing module, a first feature projection model, a base large language model, a first mask prediction model, and a first generative adversarial network. The input of the first preprocessing module is used to receive first text, and its output is connected to the input of the first feature projection model. The output of the first feature projection model is connected to the inputs of both the base large language model and the first generative adversarial network. The output of the base large language model is connected to the first input of the first mask prediction model. The second input of the first mask prediction model is used to receive a first training index sequence, and its output is used to output the corresponding first probability vector sequence. The output of the first generative adversarial network is used to output the corresponding first discriminant probability. The first training index sequence consists of one or more first training indices.

[0144] The first preprocessing module of the central pre-training framework is used to take the received first text as the corresponding current text and preprocess the current text to obtain the corresponding first text feature vector, which is then sent to the first feature projection model.

[0145] Specifically, the process is as follows: The current text is segmented according to the segmentation rules of the base large language model to obtain the corresponding first segmentation sequence; based on the feature mapping rules, the first node corresponding to each first segmentation in the first segmentation sequence in Knowledge Graph 1 is recorded as the corresponding first segmentation node; it is then determined whether each first segmentation node is empty. If it is, the first segmentation corresponding to the current first segmentation node is recorded as a non-term segmentation; otherwise, the first segmentation corresponding to the current first segmentation node is recorded as a term segmentation; some or all attributes of the node attributes in Knowledge Graph 1 are used as the corresponding node features; and all node features corresponding to each term segmentation are combined to form a corresponding first term feature vector; and a zero vector is configured for each non-term segmentation as the corresponding first term feature vector; each first segmentation in the first segmentation sequence is encoded according to the standard BERT model embedding method to obtain the corresponding first embedding encoding vector; and the first term feature vector and the first embedding encoding vector corresponding to each first segmentation are concatenated according to the vector concatenation method to obtain the corresponding first segmentation feature vector; and all the obtained first segmentation feature vectors are combined to form the corresponding first text feature vector and sent to the first feature projection model.

[0146] The first feature projection model of the central pre-training framework performs high-dimensional vector mapping on the first text feature vector and sends the resulting first text mapping vector to the base large language model and the first generative adversarial network.

[0147] The base language model of the central pre-training framework performs feature encoding on the first text mapping vector and sends the resulting first text encoding vector to the first mask prediction model.

[0148] The first mask prediction model of the central pre-training framework uses the first training index sequence as the corresponding first mask position index sequence, and performs mask word segmentation probability prediction based on the current first mask position index sequence and the first text encoding vector, and outputs the corresponding first probability vector sequence.

[0149] The first generative adversarial network of the central pre-training framework obtains the corresponding first text scrambling vector by adding random noise to the first text mapping vector by the first generator, and the first discriminator predicts the real data probability of the first text scrambling vector and outputs the corresponding first discriminant probability.

[0150] In a specific implementation of this invention, the central server 21 is specifically used to: obtain multiple first collected documents from publicly available medical literature and clinical medical documents in the open domain using a large dataset; segment each first collected document to obtain multiple first text segments; perform text cleaning on each first text segment according to preset text cleaning rules to obtain corresponding first cleaned text; remove stop words from each first cleaned text to obtain corresponding second cleaned text; perform deduplication on all obtained second cleaned texts; and after deduplication, add each remaining second cleaned text as a corresponding first corpus text to the corpus 22. Here, the text cleaning rules in this invention are conventional text cleaning rules, such as removing punctuation marks, numbers / letters, special symbols, and correcting spelling / writing errors.

[0151] In another specific implementation of this invention, the central server 21 is specifically used when performing a pre-training operation based on the corpus 22 and the central pre-training framework:

[0152] Step A1: Count the total number of texts in the first corpus of corpus 22 to obtain the corresponding total number N. a ; and each first corpus text is denoted as the corresponding first text t. i 1 ≤ index i ≤ N a The current training phase is set as the first phase; and each first text t is segmented according to the word segmentation rules of the base large language model. i Pre-segmentation is performed to obtain the corresponding first pre-segmented word sequence sa. i ;

[0153] Step A2: Identify the current training stage; if the current training stage is the first stage, then segment each first pre-segmented word sequence sa according to the preset first mask rate and mask word segmentation "[MASK]". i The word segmentation in the sequence is randomly masked and replaced to obtain the corresponding second pre-segmentation sequence sb. i If the current training phase is the second phase, then based on the masked word segmentation “[MASK]”, each first pre-segmented word sequence sa is segmented... i Each word segment representing a specific disease type is masked and replaced to obtain the corresponding second pre-segmentation sequence sb. i ;

[0154] Here, the first mask rate is a preset ratio parameter, such as 15%, 35%, 88%, etc.;

[0155] Step A3, divide each second pre-segmented word sequence sb i The sequence index of each masked word segment in the image is used as the corresponding first training index s. i,n; and composed of each second presegmentation sequence sb i All corresponding first training indices s i,n Form a corresponding first training index sequence S i ; and based on each first training index s i,n Set a corresponding first label vector y i,n ; and by each of the first training index sequences S i All corresponding first label vectors y i,n Form a corresponding first label vector sequence Y i ;

[0156] Among them, each of the first training index sequences S i The first training indexes s i,n The total number is denoted as N. i 1 ≤ index n ≤ N i ; First label vector y i,n The length of the vector is the total number of word segments in the vocabulary of the base large language model, denoted as W; the first label vector y i,n Composed of multiple first label probabilities ρ i,n,w Composition, first label probability ρ i,n,w The word segmentation corresponds one-to-one with the word segmentation in the model vocabulary, where 1 ≤ index w ≤ W; the first label vector y i,n The probability ρ of having only one first label i,n,w The first label vector is y, with 1 for all others and 0 for the rest. i,n Specifically, the first label probability ρ is 1. i,n,w The corresponding word segmentation in the model vocabulary and the first training index s i,n In the first presegmentation sequence sa i Corresponding word segmentation matching in;

[0157] Step A4, transfer each of the first text t i and the corresponding first training index sequence S i The first text and the first training index sequence are used as inputs to the pre-trained framework for a forward inference; and the first probability vector sequence output from this inference is used as the corresponding first prediction vector sequence. The first text feature vector output by the first preprocessing module during this inference process is denoted as the corresponding first feature vector x. i ;

[0158] Among them, each of the first prediction vector sequences From the corresponding N i First prediction vector Composition; First prediction vector Given W first prediction probabilities composition;

[0159] Step A5, based on each first label vector sequence Y i and its corresponding first prediction vector sequence Form a corresponding first data group And by N a The first eigenvector x i Form the corresponding first vector sequence {x i};

[0160] Step A6, obtain N a The first data group Substitute the preset first model loss function L1 into the calculation to obtain the corresponding first loss value;

[0161] The loss function L1 for the first model is:

[0162]

[0163] Step A7: Identify whether the first loss value meets the preset first loss value range; if the first loss value meets the first loss value range, identify whether the current training stage is the first stage; if so, reset the current training stage to the second stage and return to step A2; otherwise, proceed to step A8; if the first loss value does not meet the first loss value range, based on the preset first model optimizer, perform a round of modulation on the model parameters of the first feature projection model, the base large language model, and the first mask prediction model in the central pre-training framework in the direction of minimizing the first model loss function L1, and return to step A4 when the modulation ends.

[0164] The first loss value range is a pre-set numerical range; the first model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0165] Step A8, the latest first vector sequence {x i Substitute the preset first discriminant loss L D1 and the first generation loss L G1 ; and based on the preset second model optimizer, it moves towards making the first discriminative loss L D1 The model parameters of the first feature projection model and the first discriminator of the first generative adversarial network in the central pre-trained framework are modulated once in the direction that minimizes the first generative loss L. At the end of this round of modulation, the first generative loss L is minimized again based on the preset third model optimizer. G1 The direction that reaches the minimum value modulates the model parameters of the first feature projection model and the first generator of the first generative adversarial network in one round, and confirms the end of this training at the end of this round of modulation.

[0166] Among them, the first discriminant loss L D1 for:

[0167]

[0168] θ 11 F represents the model parameters of the first feature projection model. 11 () is the inference function of the first feature projection model; F 11 (x i ;θ 11 ) is used to represent the projection model with parameter θ. 11 And the model input is a text feature vector x i Under the given conditions, the text mapping vector output by the first feature projection model;

[0169] θ 12 For the model parameters of the first discriminator in the first generative adversarial network, F 12 () is the inference function of the first discriminator; F 12 [F 11 (x i ;θ 11 );θ 12 ] is used to indicate that the model parameters of the discriminator are θ 12 The discriminator input is a text mapping vector F. 11 (x i ;θ 11 Under the condition of ), the discrimination probability output by the first discriminator;

[0170] First generation loss L G1 for:

[0171]

[0172] θ 13 F represents the model parameters of the first generator in the first generative adversarial network. 13 () is the inference function of the first generator; F 13 [F 11 (x i ;θ 11 ),θ 13 ] is used to indicate that the model parameters in the generator are θ 13 And the model input is a text mapping vector F 11 (x i ;θ 11 Under the condition of ), the text scrambling vector output by the first generator; F 12 {F 13 [F 11 (x i ;θ 11 ),θ 13 ];θ 12} is used to represent the model parameters of the discriminator being θ. 12 The discriminator input is a text scrambling vector F. 13 [F 11 (x i ;θ 11 ),θ 13 Under the condition of ], the discrimination probability output by the first discriminator;

[0173] The second model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0174] The third model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0175] In another specific implementation of this invention, the central server 21 is specifically used to: identify whether the version number matches a preset initial version number when setting the corresponding fine-tuning layer number and distributing model parameters based on the current version number; if they match, set the fine-tuning layer number to 0 and use the central model parameters corresponding to the current version number as the corresponding distributing model parameters; if they do not match, use the layer number corresponding to the last Encoder layer of the base large language model as the fine-tuning layer number, and use the model parameters in the central model parameters corresponding to the current version number that correspond to the last Encoder layer as the corresponding distributing model parameters. Here, the initial version number is a preset version number value.

[0176] In another specific implementation of this invention, the central server 21 is specifically used to: when updating the base model parameters according to each parameter directory to obtain a new version of the central model parameters, use each parameter directory as the corresponding current directory; use the fine-tuning layer number and central model parameters stored in the current directory as the corresponding current number and current central model parameters; identify the current number; if the current number is 0, use all parameters of the current central model parameters as the current baseline parameters; if the current number is greater than 0, use the model layer parameters in the current central model parameters corresponding to the current number as the current baseline parameters; identify whether the total number of institutional model parameters stored in the current directory matches the total number of institutional networks 3, and if so, perform model parameter fusion on all institutional model parameters in the current directory to obtain the corresponding current fusion parameters, and replace some or all parameters in the current central model parameters corresponding to the current baseline parameters based on the current fusion parameters to obtain a new version of the central model parameters;

[0177] Specifically, the model parameter fusion is as follows:

[0178] 1 ≤ Institutional Index k ≤ N org θ k θ is the parameter of the k-th mechanism model in the current directory.R These are the current fusion parameters.

[0179] (III) Institutional Network 3:

[0180] The institutional network 3 in this embodiment of the invention includes an institutional server 31, a text set 32, and an institutional model library 33. The institutional server 31 is connected to the knowledge graph 1, the text set 32, and the institutional model library 33, respectively; each institutional network 3 corresponds to a medical institution.

[0181] 1) Institutional Server 31:

[0182] In this embodiment of the invention, the organization server 31 is used to update the text set 32 ​​through the organization's data collection.

[0183] Institutional server 31 is also used to build a local fine-tuning framework and a local task framework based on the received feature mapping rules, knowledge graph 1 and institutional model library 33.

[0184] The organization server 31 is also used to set the model parameters of the organization big language model in the organization model library 33 based on the fine-tuning layer number and the model parameters it receives each set of version number, distributed model parameters and fine-tuning layer number; then it performs a local training based on the fine-tuning layer number, text set 32, local fine-tuning framework and local task framework; and at the end of this training, it sets the corresponding organization model parameters based on the current model parameters and fine-tuning layer number of the organization big language model; and sends the organization model parameters and version number to the central server 21.

[0185] 2) Text set 32:

[0186] The text set 32 ​​in this embodiment of the invention includes multiple first text records; each first text record includes first medical text and a first classification label. The first classification label consists of multiple first label classification probabilities, each first label classification probability corresponding to a preset disease type; only one first label has a classification probability of 1, while the remaining first label classification probabilities are all 0.

[0187] 3) Institutional Model Library 33:

[0188] The inventory models of the institutional model library 33 in this embodiment of the invention include at least an institutional big language model, a second feature projection model, a second generative adversarial network, a second mask prediction model, and an institutional text classification model; the institutional big language model has the same model structure as the base big language model.

[0189] A. The second feature projection model in this embodiment of the invention is implemented based on an MLP model structure, and its function is similar to that of the first feature projection model. The input and output vectors of the second feature projection model are the corresponding second text feature vector and the second text mapping vector; the second feature projection model is used to perform high-dimensional vector mapping on the input second text feature vector and output the corresponding second text mapping vector.

[0190] B. The organizational large language model and the base large language model in this embodiment of the invention have the same model structure. The model input and output vectors of the organizational large language model are the corresponding second text mapping vector and the second text encoding vector; the base large language model is used to perform feature encoding processing on the input second text mapping vector and output the corresponding second text encoding vector.

[0191] C. The second generative adversarial network in this embodiment of the invention is implemented based on a GAN model, and its function is similar to that of the first generative adversarial network. The second generative adversarial network includes a second generator and a second discriminator; the model input and output of the second generator are the corresponding second text mapping vector and the second text scrambling vector; the model input of the second discriminator is the corresponding second text mapping vector and the second text scrambling vector, and the model output is the corresponding second discrimination probability. The second generator is used to add random noise to each second word segmentation mapping vector of the second text mapping vector to obtain the corresponding second word segmentation scrambling vector, and the corresponding second text scrambling vector is composed of all the second word segmentation scrambling vectors; the second discriminator is used to predict the true data probability of the input second text scrambling vector based on the input second text mapping vector and use the obtained predicted probability as the corresponding second discrimination probability.

[0192] D. The second mask prediction model is implemented based on the task head network of the BERT model's mask prediction task, and its function is similar to that of the first mask prediction model. The input to the second mask prediction model is the corresponding second text encoding vector and the second mask position index sequence; the second mask position index sequence consists of one or more second mask position indices; the output of the second mask prediction model is the corresponding second probability vector sequence; the second probability vector sequence consists of one or more second probability vectors, each corresponding one-to-one with a second mask position index; the length of the second probability vector is the total number of words in the institutional large language model's vocabulary, and each second probability vector consists of multiple second word segmentation probabilities, each corresponding one-to-one with a word segmentation in the model vocabulary; the institutional large language model's vocabulary is consistent with the base large language model's vocabulary. The second mask prediction model is used to predict the word segmentation probability of the masked word segmentation corresponding to the current second mask position index based on the second word segmentation encoding vectors corresponding to each second mask position index in the second text encoding vector, and then uses all the obtained second probability vectors to form the corresponding second probability vector sequence.

[0193] E. The institutional text classification model of this invention is based on a multi-classification model structure. The model input and output of the institutional text classification model are a corresponding second text encoding vector and a first classification probability vector; the first classification probability vector consists of multiple first classification probabilities, each corresponding to a preset disease type. The institutional text classification model is used to perform classification prediction based on the second text encoding vector input to the model and output the corresponding first classification probability vector.

[0194] like Figure 3 As shown in the module structure diagram of the local fine-tuning framework provided in this embodiment of the invention, the local fine-tuning framework includes a second preprocessing module, a second feature projection model, an institutional big language model, a second mask prediction model, and a second generative adversarial network. The input of the second preprocessing module is used to receive second text, and its output is connected to the input of the second feature projection model. The output of the second feature projection model is connected to the inputs of both the institutional big language model and the second generative adversarial network. The output of the institutional big language model is connected to the first input of the second mask prediction model. The second input of the second mask prediction model is used to receive a second training index sequence, and its output is used to output the corresponding second probability vector sequence. The output of the second generative adversarial network is used to output the corresponding second discriminant probability. The second training index sequence consists of one or more second training indices.

[0195] The second preprocessing module of the local fine-tuning framework is used to take the received second text as the corresponding current text and preprocess the current text to obtain the corresponding second text feature vector, which is then sent to the second feature projection model.

[0196] Specifically, the process involves: segmenting the current text according to the segmentation rules of the institutional language model to obtain the corresponding second segmentation sequence; based on the feature mapping rules, recording the first node corresponding to each second segmentation in the knowledge graph 1 as the corresponding second segmentation node; identifying whether each second segmentation node is empty, if so, recording the second segmentation corresponding to the current second segmentation node as a non-term segmentation, otherwise recording the second segmentation corresponding to the current second segmentation node as a term segmentation; forming a corresponding second term feature vector from all node features corresponding to each term segmentation; configuring an all-zero vector as the corresponding second term feature vector for each non-term segmentation; encoding each second segmentation in the second segmentation sequence according to the standard BERT model embedding method to obtain the corresponding second embedding encoding vector; concatenating the second term feature vector and the second embedding encoding vector corresponding to each second segmentation according to the vector concatenation method to obtain the corresponding second segmentation feature vector; and sending the corresponding second text feature vector, composed of all the obtained second segmentation feature vectors, to the second feature projection model. Here, the word segmentation rules of the institutional big language model are consistent with those of the base big language model.

[0197] The second feature projection model of the local fine-tuning framework performs high-dimensional vector mapping on the second text feature vector and sends the resulting second text mapping vector to the institutional big language model and the second generative adversarial network.

[0198] The local fine-tuning framework and the large language model encode the features of the second text mapping vector and send the resulting second text encoding vector to the second mask prediction model.

[0199] The second mask prediction model of the local fine-tuning framework uses the second training index sequence as the corresponding second mask position index sequence, and performs mask word segmentation probability prediction based on the current second mask position index sequence and the second text encoding vector, and outputs the corresponding second probability vector sequence.

[0200] The second generative adversarial network of the local fine-tuning framework obtains the corresponding second text scrambling vector by adding random noise to the second text mapping vector by the second generator, and the second discriminator predicts the real data probability of the second text scrambling vector and outputs the corresponding second discriminant probability.

[0201] like Figure 4As shown in the module structure diagram of the local task framework provided in this embodiment of the invention, the local task framework includes a second preprocessing module, a second feature projection model, an institutional large language model, and an institutional text classification model. The input of the second preprocessing module is used to receive third text, and its output is connected to the input of the second feature projection model. The output of the second feature projection model is connected to the input of the institutional large language model. The output of the institutional large language model is connected to the input of the institutional text classification model. The output of the institutional text classification model is used to output the corresponding first classification probability vector.

[0202] The second preprocessing module of the local task framework takes the received third text as the corresponding current text, preprocesses the current text to obtain the corresponding second text feature vector, and sends it to the second feature projection model.

[0203] The second feature projection model of the local task framework performs high-dimensional vector mapping on the second text feature vector and sends the resulting second text mapping vector to the institutional large language model.

[0204] The local task framework's institutional big language model encodes the second text mapping vector and sends the resulting second text encoding vector to the institutional text classification model.

[0205] The local task framework's organizational text classification model performs classification prediction based on the second text encoding vector and outputs the corresponding first classification probability vector.

[0206] In another specific implementation of this invention, the institutional server 31 is specifically used to: periodically collect data from newly added electronic medical record documents within the medical institution to obtain multiple first medical record documents; desensitize the patient privacy information of each first medical record document to obtain a corresponding first desensitized document; segment each first desensitized document to obtain multiple second text segments; clean each second text segment according to preset text cleaning rules to obtain a corresponding third cleaned text; remove stop words from each third cleaned text to obtain a corresponding fourth cleaned text; deduplicate all obtained fourth cleaned texts; and after deduplication, use each remaining fourth cleaned text as a corresponding first medical text; confirm the disease type corresponding to each first medical record document to obtain a corresponding first disease type; set a corresponding first classification label according to each first disease type; form a corresponding first text record by each first medical text and its corresponding first classification label; and add all obtained first text records to the text set 32.

[0207] The first category label consists of multiple first label classification probabilities, each first label classification probability corresponds to a preset disease type; among all first category labels, only the first label corresponding to the corresponding first disease type has a classification probability of 1, and the classification probabilities of the other first labels are all 0.

[0208] In another specific implementation of this invention, the organization server 31 is specifically used to: identify the current fine-tuning layer number when setting model parameters for the organization's large language model in the organization model library 33 based on the fine-tuning layer number and the issued model parameters; if the current fine-tuning layer number is 0, then set the overall model parameters of the organization's large language model based on the issued model parameters; if the current fine-tuning layer number is greater than 1, then set the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the organization's large language model based on the issued model parameters.

[0209] In another specific implementation of this invention, the mechanism server 31 is specifically used to perform a local training based on the fine-tuning layer number, text set 32, local fine-tuning framework, and local task framework:

[0210] Step B1: Identify the current fine-tuning layer number;

[0211] Step B2: If the current fine-tuning layer number is 0, then the first adjustable parameter set is composed of all model parameters of the second feature projection model, the institutional big language model, the second mask prediction model, and the second generative adversarial network; and the second adjustable parameter set is composed of all model parameters of the institutional big language model and the institutional text classification model.

[0212] Step B3: If the current fine-tuning layer number is greater than 1, then the first adjustable parameter set is composed only of the overall mode parameters of the second feature projection model and the model layer parameters of the overall model parameters of the institutional large language model that correspond to the current fine-tuning layer number; and the second adjustable parameter set is composed only of the model layer parameters of the overall model parameters of the institutional large language model that correspond to the current fine-tuning layer number and the overall model parameters of the institutional text classification model.

[0213] Step B4, and perform a pre-training based on text set 32, local fine-tuning framework and first adjustable parameter set; after the pre-training is completed, perform a task training based on text set 32, local task framework and second adjustable parameter set; and at the end of the task training, confirm that the local training is over.

[0214] In another specific implementation of this invention, the mechanism server 31 is specifically used during a pre-training process based on the text set 32, the local fine-tuning framework, and the first adjustable parameter set:

[0215] Step C1: Count the total number of the first medical texts in text set 32 ​​to obtain the corresponding total number N. b And each first medical text is recorded as the corresponding second text t. j 1 ≤ index j ≤ N b And according to the word segmentation rules of the institutional big language model, each second text t is segmented. j Pre-segmentation is performed to obtain the corresponding third pre-segmented word sequence sc. j ; and according to the preset second mask rate and mask segmentation "[MASK]", each third pre-segmentation sequence sc j The word segments in the sequence are randomly masked and replaced to obtain the corresponding fourth pre-segmentation sequence sd. j ;

[0216] Here, the second mask rate is a pre-set ratio parameter. In this embodiment of the invention, the second mask rate is required to be no less than the first mask rate.

[0217] Step C2, divide each fourth pre-segmented sequence sd j The sequence index of each masked word segment in the image is used as a corresponding second training index s. j,m ; and composed of each fourth presegmentation sequence sd j All corresponding second training indices s j,m Form a corresponding second training index sequence S j ; and based on each second training index s j,m Set a corresponding second label vector y j,m ; and by each of the second training index sequences S j All corresponding second label vectors y j,m Form a corresponding second label vector sequence Y j ;

[0218] Among them, each of the second training index sequences S j The second training indexes j,m The total number is denoted as N. j 1 ≤ index m ≤ N j ; Second label vector y j,m The length of the vector is consistent with the total number of word segments in the model vocabulary of the institutional large language model and the total number of word segments W in the model vocabulary of the base large language model; the second label vector y j,m Composed of multiple second label probabilities ρ j,m,w Composition, second label probability ρ j,m,w The word segmentation corresponds one-to-one with the word segmentation in the model vocabulary, where 1 ≤ index w ≤ W; the second label vector y j,m There is only one second label probability ρ j,m,w The first label vector is 1, and the rest are 0; the second label vector y j,m Specifically, the second label probability ρ is 1.j,m,w The corresponding word segmentation in the model vocabulary and the second training index s j,m In the third presegmentation sequence sc j Corresponding word segmentation matching in;

[0219] Step C3, transfer each of the second text t j and the corresponding second training index sequence S j The corresponding second text and second training index sequence are input into the local fine-tuning framework for a forward inference; and the second probability vector sequence output in this instance is used as the corresponding second prediction vector sequence. The second text feature vector output by the second preprocessing module during this inference process is denoted as the corresponding second feature vector x. j The second text encoding vector output by the institutional large language model during this inference process is denoted as the corresponding first encoding vector e. j ;

[0220] Among them, each second prediction vector sequence From the corresponding N j The second prediction vector Composition; Second prediction vector Given W second prediction probabilities composition;

[0221] Step C4, based on the sequence of each second label vector Y j and its corresponding second prediction vector sequence Form a corresponding second data group And by N b The second eigenvector x j Form the corresponding second vector sequence {x j};

[0222] Step C5, convert each of the first encoding vectors e j This serves as the corresponding current encoding vector, and noise is added to the current encoding vector to obtain the corresponding positive sample vector. And the remaining N outside the current encoding vector b -1 first encoded vector e j Each is denoted as the corresponding negative sample vector. 1≤index u≤N b -1; and by each of the first encoding vectors e j and its corresponding positive sample vector and N b -1 negative sample vectors Form a corresponding third data group

[0223] Step C6: Denote the parameter part of the large language model of the first adjustable parameter set as parameter θ. l , will the parameter θ L The corresponding model parameters are denoted as parameter θ. C ; and the obtained N a The second and third data groups The corresponding second loss value is obtained by substituting the preset second model loss function L2 into the calculation.

[0224] The loss function L2 for the second model is:

[0225] L2 = L 21 +αL 22 +β||θ l -θ C || 2 ,

[0226]

[0227] α and β are two pre-set weighting parameters, τ is two pre-set temperature coefficients, and the loss function L 21 Implemented based on the cross-entropy loss function, loss function L 22 Implemented based on the contrastive loss function; ||θ l -θ C || 2 For the latest parameter θ l With parameter θ C The L2 regularization; sim() is the cosine similarity function of vectors; For each first encoding vector e j Its corresponding positive sample vector cosine similarity, The first encoding vector e j Its corresponding negative sample vectors Cosine similarity;

[0228] Step C7: Identify whether the second loss value meets the preset range of the second loss value; if the second loss value meets the range of the second loss value, proceed to step C8; if the second loss value does not meet the range of the second loss value, perform a round of modulation on the model parameters in the first adjustable parameter set based on the preset fourth model optimizer in the direction of minimizing the second model loss function L2, and return to step C3 when the modulation ends.

[0229] The second loss value range is a pre-set numerical range; the fourth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0230] Step C8, the latest second vector sequence {x} jSubstitute the preset second discriminant loss L D2 Second generation loss L G2 ; and based on the preset fifth model optimizer, it moves towards making the second discriminative loss L D2 The model parameters in the first adjustable parameter set are modulated once in the direction that reaches the minimum value; and at the end of this round of modulation, the second generation loss L is then modulated based on the preset sixth model optimizer. G2 The direction that reaches the minimum value is used to modulate the model parameters in the first adjustable parameter set in one round; and the training is confirmed to be over when the modulation ends.

[0231] Among them, the second discriminant loss L D2 for:

[0232]

[0233] G is the first set of adjustable parameters;

[0234] θ 21 The model parameters for the second feature projection model; θ 21 ∩G represents the adjustable parameter portion corresponding to the second feature projection model in the first adjustable parameter set; F 21 () is the inference function of the second feature projection model; F 21 (x j ;θ 21 ) is used to represent the projection model with parameter θ. 21 And the model input is a text feature vector x j Under the given conditions, the text mapping vector output by the second feature projection model;

[0235] θ 22 θ represents the model parameters of the second discriminator in the second adversarial network. 22 ∩G represents the adjustable parameter portion corresponding to the second discriminator in the first adjustable parameter set; F 22 () is the inference function of the second discriminator; F 22 [F 21 (x j ;θ 21 );θ 22 ] is used to indicate that the model parameters of the discriminator are θ 22 The discriminator input is a text mapping vector F. 21 (x j ;θ 21 Under the condition of ), the discrimination probability output by the second discriminator;

[0236] Second generation loss L G2 for:

[0237]

[0238] θ 23 For the model parameters of the second generator in the second generative adversarial network; θ 23 ∩G represents the adjustable parameter portion corresponding to the second generator in the first adjustable parameter set; F 23 () is the inference function of the second generator; F 23 [F 21 (x j ;θ 21 ),θ 23 ] is used to indicate that the generator model parameter is θ 23 And the model input is a text mapping vector F 21 (x j ;θ 21 Under the condition of ), the text scrambling vector output by the second generator; F 22 {F 23 [F 21 (x j ;θ 21 ),θ 23 ];θ 22} is used to represent the discriminator model parameters being θ. 22 The discriminator input is a text scrambling vector F. 23 [F 21 (x j ;θ 21 ),θ 23 Under the condition of ], the discrimination probability output by the second discriminator;

[0239] The fifth model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0240] The sixth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0241] In another specific implementation of this invention, the institution server 31 is specifically used during a task training based on the text set 32, the local task framework, and the second adjustable parameter set:

[0242] Step D1: Count the total number of the first medical texts in text set 32 ​​to obtain the corresponding total number N. c And each first medical text is recorded as the corresponding third text t. h 1 ≤ index h ≤ N c And according to the word segmentation rules of the institutional big language model, each third text t is segmented. h Pre-segmentation is performed to obtain the corresponding fifth pre-segmented word sequence se. h And denote the first classification label corresponding to each first medical text as the corresponding label vector Y. h ;

[0243] Wherein, the label vector Yh Includes multiple label probabilities ρ h,v 1 ≤ index v ≤ N d N d The total number of preset disease types;

[0244] Step D2, transfer each third text t h The corresponding third text input is used as the local task framework for a forward inference; and the first classification probability vector output this time is used as the corresponding prediction vector. The second text mapping vector output by the second feature projection model during this inference process is denoted as the corresponding mapping vector X. h ;

[0245] Among them, the prediction vector Includes multiple predicted probabilities

[0246] Step D3, based on each label vector Y h and its corresponding prediction vector Form a corresponding fourth data group And set a feature dimension and a mapping vector X. h A consistent standard normal distribution space, and N randomly sampled from the standard normal distribution space. e Let X be a spatial vector. g ; and mapped by vector X h and space vector P g Pairwise combinations yield N c ×N e The fifth data set (X) h ,P g );

[0247] Where, N e Let N be a positive integer, and N be a positive integer. e >N c ; 1 ≤ index g ≤ N e ;

[0248] Step D4: Denote the overall model parameters of the second feature projection model as parameter θ. proj ; and the parameter θ proj N c The fourth data set N c ×N e The fifth data set (X) h ,P g Substitute the input into the preset third model loss function L3 to calculate the corresponding third loss value;

[0249] The loss function L3 for the third model is:

[0250]

[0251] γ and λ are two pre-set weight parameters; 1 ≤ index h * ≤N c , 1≤index g * ≤N e Loss function L 31 Implemented based on the cross-entropy loss function, loss function L 32 Implemented based on the maximum mean difference loss function. For the latest parameter θ proj The Frobenius norm regularity; f kernel () is a pre-defined kernel function with two input variables;

[0252] Step D5 involves identifying whether the third loss value meets the preset range. If the third loss value does not meet the range, the model parameters in the second adjustable parameter set are modulated once based on the preset seventh model optimizer in the direction of minimizing the third model loss function L3, and the process returns to step D2 at the end of this modulation round. If the third loss value meets the range, training is stopped and the end of this task is confirmed. Here, the range of the third loss value is a preset numerical range.

[0253] In another specific implementation of this invention, the mechanism server 31 is specifically used to: identify the current fine-tuning layer number when setting the corresponding mechanism model parameters based on the current model parameters and fine-tuning layer number of the mechanism large language model; if the current fine-tuning layer number is 0, then the overall model parameters of the mechanism large language model are used as the corresponding mechanism model parameters; if the current fine-tuning layer number is greater than 1, then only the model layer parameters corresponding to the current fine-tuning layer number in the overall model parameters of the mechanism large language model are used as the corresponding mechanism model parameters.

[0254] It should be noted that the division of the various modules in the above system is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls; they can be implemented entirely in hardware; or they can be partially implemented through processing element calls to software and partially in hardware. For example, the central server can be a separate processing element, or a chip integrated into a device, equipment, or server. Alternatively, it can be stored as program code in the memory of the device, equipment, or server and called by a processing element to implement the corresponding processing function. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. During implementation, the processing steps of each module in the aforementioned system can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0255] For example, these modules in the above system can be one or more integrated circuits configured to implement the aforementioned processing steps, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module in the above system is implemented through processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0256] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0257] This invention provides a federated learning system for a medical large language model. The system includes: a knowledge graph, a central network, and multiple institutional networks. The central network includes a central server, a corpus, and a central model library. The institutional networks include institutional servers, text sets, and institutional model libraries. Specifically: 1) The central server is used to set a global feature mapping rule for custom terms from all medical institutions; it is also used to pre-train the base large language model; it is also used to assign / set a corresponding version number, parameter directory, fine-tuning layer number, and distribute model parameters each time a version of the central model parameters is generated, and to store the current central model parameters and fine-tuning layer number in the corresponding directory, and to distribute the current version number + distributed model parameters + fine-tuning layer number to each institutional server; it is also used to store each institutional model parameter received from an institutional server in the corresponding directory, and to update the base model parameters according to each parameter directory. 2) The institution server is used to construct a local fine-tuning framework and a local task framework based on feature mapping rules, knowledge graphs, and the institution model library. It is also used to set model parameters for the institution's large language model each time a version number, distributed model parameters, and fine-tuning layer number are received. Furthermore, it performs local training on the institution's large language model based on the fine-tuning layer number, text set, local fine-tuning framework, and local task framework, and sends the latest institution model parameters to the central server at the end of this training. This invention provides a federated learning scheme for training a medical large language model, which can continuously improve the model's generalization ability even without sharing medical data.

[0258] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0259] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0260] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A federated learning system for a medical large language model, characterized in that, The system includes: a knowledge graph, a central network, and multiple institutional networks; The central network includes a central server, a corpus, and a central model library; The central server is used to configure the one-to-one correspondence between custom terms and knowledge graph nodes for all medical institutions, obtain the corresponding feature mapping rules, and distribute them to the institution servers of all the institution networks; and initialize the base model parameters. The central server is also used to update the corpus through big data collection; The central server is also used to, upon completion of the initialization of the base model parameters, set the model parameters of the base large language model in the central model library once according to the base model parameters; construct a central pre-training framework according to the feature mapping rules, the knowledge graph and the central model library; perform a pre-training based on the corpus and the central pre-training framework; and at the end of this training, use the current model parameters of the base large language model as the initial version of the central model parameters. The central server is also used to assign a corresponding version number to the current central model parameters each time a new version of the central model parameters is generated; to assign a corresponding parameter directory to the current version number; to set a corresponding fine-tuning layer number and distributed model parameters based on the current version number; to store the current central model parameters and the fine-tuning layer number in the current parameter directory; and to distribute the current version number, the distributed model parameters, and the fine-tuning layer number to each of the institution servers; wherein the fine-tuning layer number is a positive integer greater than or equal to 0. The central server is also used to store the current organizational model parameters into the corresponding parameter directory each time it receives organizational model parameters and version number from an organizational server. The central server is also used to update the base model parameters according to each of the parameter directories to obtain a new version of the central model parameters; The institution network includes the institution server, text set, and institution model library; The organization's server is used to update the text set through data collection from this organization; The institution server is also used to construct a local fine-tuning framework and a local task framework based on the received feature mapping rules, the knowledge graph, and the institution model library. The organization server is also used to, upon receiving a set of version numbers, distributed model parameters, and fine-tuning layer numbers, set model parameters for the organization's large language model in the organization model library based on the fine-tuning layer number and the distributed model parameters; then, perform local training based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework; and at the end of this training, set the corresponding organization model parameters based on the current model parameters of the organization's large language model and the fine-tuning layer number; and send the organization model parameters and the version number to the central server.

2. The federated learning system for a medical large language model according to claim 1, characterized in that, The knowledge graph comprises a first node set and a first edge set. The first node set includes multiple first nodes. Each first node corresponds to a standard medical term. Each first node corresponds to a set of node attributes, which at least include a term name, a term code, a term type, and a data type. The term name is a medical term name. The term code is a standard code of a specified medical standard system, which at least includes the ICD-10 standard system. The term type at least includes disease, drug, diagnosis, test, examination, and gene. The first edge set includes multiple first edges. Each first edge is a directed edge used to connect two first nodes. The edge attribute of each first edge is a type of medical term association relationship. The corpus includes multiple first-level texts; The text set includes multiple first text records; each first text record includes a first medical text and a first classification label; each first classification label consists of multiple first label classification probabilities, and each first label classification probability corresponds to a preset disease type; among each first classification label, only one first label classification probability is 1, and the rest of the first label classification probabilities are 0. The inventory models of the central model library include at least the base large language model, the first feature projection model, the first generative adversarial network, and the first mask prediction model; The aforementioned base language model is a type of medical language model; The inventory models of the institutional model library include at least the institutional big language model, the second feature projection model, the second generative adversarial network, the second mask prediction model, and the institutional text classification model; the institutional big language model has the same model structure as the base big language model.

3. The federated learning system for a medical large language model according to claim 2, characterized in that, The first feature projection model is implemented based on the MLP model structure; the model input and output vectors of the first feature projection model are the corresponding first text feature vector and first text mapping vector; the first text feature vector is composed of multiple first word segmentation feature vectors; the first text mapping vector is composed of multiple first word segmentation mapping vectors, the first word segmentation mapping vector corresponds one-to-one with the first word segmentation feature vector, the vector feature dimension of the first word segmentation mapping vector is higher than the vector feature dimension of the first word segmentation feature vector, and the vector feature dimension of the first word segmentation mapping vector is consistent with the input vector feature dimension of the base large language model; The base language model is implemented based on the Med-BERT model; the model input and output vectors of the base language model are the corresponding first text mapping vector and first text encoding vector; the first text encoding vector is composed of multiple first word segmentation encoding vectors, and the first word segmentation encoding vector corresponds one-to-one with the first word segmentation mapping vector; The first generative adversarial network is implemented based on a GAN model; the first generative adversarial network includes a first generator and a first discriminator; the model input and output of the first generator are the corresponding first text mapping vector and the first text scrambling vector; the model input of the first discriminator is the corresponding first text mapping vector and the first text scrambling vector, and the model output is the corresponding first discrimination probability; the first text scrambling vector is composed of multiple first word segmentation scrambling vectors, and the first word segmentation scrambling vector corresponds one-to-one with the first word segmentation mapping vector; the first discrimination probability is a probability value between 0 and 1. The first mask prediction model is implemented based on the task head network of the BERT model's mask prediction task. The model input of the first mask prediction model is the corresponding first text encoding vector and the first mask position index sequence. The first mask position index sequence consists of one or more first mask position indices, and each first mask position index corresponds to one of the first word segmentation encoding vectors in the first text encoding vector. The model output of the first mask prediction model is the corresponding first probability vector sequence. The first probability vector sequence consists of one or more first probability vectors, and the first probability vector corresponds one-to-one with the first mask position index. The vector length of the first probability vector is the total number of word segments in the model vocabulary of the base large language model. The first probability vector consists of multiple first word segmentation probabilities, and the first word segmentation probabilities correspond one-to-one with the word segments in the model vocabulary. The second feature projection model is implemented based on the MLP model structure; the model input and output vectors of the second feature projection model are the corresponding second text feature vector and the second text mapping vector. The structure of the institutional large language model is consistent with that of the base large language model; the input and output vectors of the institutional large language model are the corresponding second text mapping vector and second text encoding vector; The second generative adversarial network is implemented based on a GAN model; the second generative adversarial network includes a second generator and a second discriminator; the model input and output of the second generator are the corresponding second text mapping vector and the second text scrambling vector; the model input of the second discriminator is the corresponding second text mapping vector and the second text scrambling vector, and the model output is the corresponding second discriminant probability; The second mask prediction model is implemented based on the task head network of the BERT model's mask prediction task, and its function is similar to that of the first mask prediction model. The model input of the second mask prediction model is the corresponding second text encoding vector and the second mask position index sequence. The second mask position index sequence consists of one or more second mask position indices. The model output of the second mask prediction model is the corresponding second probability vector sequence. The second probability vector sequence consists of one or more second probability vectors, and the second probability vectors correspond one-to-one with the second mask position indices. The vector length of the second probability vector is the total number of word segments in the model vocabulary of the institutional large language model. The second probability vector consists of multiple second word segmentation probabilities, and the second word segmentation probabilities correspond one-to-one with the word segments in the model vocabulary. The model vocabulary of the institutional large language model is consistent with the model vocabulary of the base large language model. The institutional text classification model is implemented based on a multi-classification model structure; the model input and output of the institutional text classification model are the corresponding second text encoding vector and first classification probability vector; the first classification probability vector is composed of multiple first classification probabilities, each of which corresponds to a preset disease type; the institutional text classification model is used to perform classification prediction based on the second text encoding vector input to the model and output the corresponding first classification probability vector.

4. The federated learning system for a medical large language model according to claim 3, characterized in that, The central pre-training framework includes a first preprocessing module, a first feature projection model, a base large language model, a first mask prediction model, and a first generative adversarial network. The input of the first preprocessing module is used to receive the first text, and its output is connected to the input of the first feature projection model; the output of the first feature projection model is connected to the inputs of the base large language model and the first generative adversarial network respectively; the output of the base large language model is connected to the first input of the first mask prediction model; the second input of the first mask prediction model is used to receive the first training index sequence, and its output is used to output the corresponding first probability vector sequence; the output of the first generative adversarial network is used to output the corresponding first discrimination probability. The first training index sequence consists of one or more first training indices; The first preprocessing module is used to take the received first text as the corresponding current text, preprocess the current text to obtain the corresponding first text feature vector, and send it to the first feature projection model, specifically: The current text is segmented according to the segmentation rules of the base language model to obtain the corresponding first segmentation sequence; and based on the feature mapping rules, the first node corresponding to each first segmentation in the first segmentation sequence in the knowledge graph is recorded as the corresponding first segmentation node; and it is identified whether each first segmentation node is empty. If it is, the first segmentation corresponding to the current first segmentation node is recorded as a non-term segmentation; otherwise, the first segmentation corresponding to the current first segmentation node is recorded as a term segmentation; and some or all of the node attributes in the knowledge graph are used as the corresponding node features. A first term feature vector is formed by combining all the node features corresponding to each term segmentation; a zero vector is configured for each non-term segmentation as the corresponding first term feature vector; each first segmentation in the first segmentation sequence is encoded according to the standard BERT model embedding method to obtain the corresponding first embedding encoding vector; the first term feature vector and the first embedding encoding vector corresponding to each first segmentation are concatenated according to the vector concatenation method to obtain the corresponding first segmentation feature vector; and the first text feature vector is formed by combining all the obtained first segmentation feature vectors and sent to the first feature projection model.

5. The federated learning system for a medical large language model according to claim 4, characterized in that, The local fine-tuning framework includes a second preprocessing module, a second feature projection model, an institutional big language model, a second mask prediction model, and a second generative adversarial network; The input of the second preprocessing module is used to receive the second text, and its output is connected to the input of the second feature projection model; the output of the second feature projection model is connected to the inputs of the institutional big language model and the second generative adversarial network, respectively; the output of the institutional big language model is connected to the first input of the second mask prediction model; the second input of the second mask prediction model is used to receive the second training index sequence, and its output is used to output the corresponding second probability vector sequence; the output of the second generative adversarial network is used to output the corresponding second discriminant probability; the second training index sequence consists of one or more second training indices. The second preprocessing module is used to take the received second text as the corresponding current text, preprocess the current text to obtain the corresponding second text feature vector, and send it to the second feature projection model. Specifically: The current text is segmented according to the segmentation rules of the aforementioned large language model to obtain a corresponding second segmentation sequence; and based on the feature mapping rules, the first node corresponding to each second segmentation in the second segmentation sequence in the knowledge graph is recorded as the corresponding second segmentation node; and it is identified whether each second segmentation node is empty. If it is, the second segmentation corresponding to the current second segmentation node is recorded as a non-term segmentation; otherwise, the second segmentation corresponding to the current second segmentation node is recorded as a term segmentation; and a corresponding second term feature vector is formed by all the node features corresponding to each term segmentation; and a second term feature vector is generated for each term segmentation. The non-term segmentation is configured with an all-zero vector as the corresponding second term feature vector; and each second segmentation in the second segmentation sequence is encoded according to the standard BERT model embedding method to obtain the corresponding second embedding encoding vector; and the second term feature vector and the second embedding encoding vector corresponding to each second segmentation are concatenated according to the vector concatenation method to obtain the corresponding second segmentation feature vector; and the corresponding second text feature vector is composed of all the obtained second segmentation feature vectors and sent to the second feature projection model; the segmentation rules of the institutional big language model are consistent with the segmentation rules of the base big language model.

6. The federated learning system for a medical large language model according to claim 5, characterized in that, The local task framework includes the second preprocessing module, the second feature projection model, the institutional big language model, and the institutional text classification model; The input of the second preprocessing module is used to receive the third text, and the output is connected to the input of the second feature projection model; the output of the second feature projection model is connected to the input of the institutional big language model; the output of the institutional big language model is connected to the input of the institutional text classification model; the output of the institutional text classification model is used to output the corresponding first classification probability vector.

7. The federated learning system for a medical large language model according to claim 4, characterized in that, The central server is specifically used during a pre-training process based on the corpus and the central pre-training framework: Step 71: Calculate the total number of the first corpus texts in the corpus to obtain the corresponding total number N. a And each of the first corpus texts is recorded as the corresponding first text t. i 1 ≤ index i ≤ N a The current training phase is set as the first phase; and each of the first texts is segmented according to the word segmentation rules of the base large language model. i Pre-segmentation is performed to obtain the corresponding first pre-segmented word sequence sa. i ; Step 72: Identify the current training stage; if the current training stage is the first stage, then segment each of the first pre-segmented word sequences sa according to the preset first mask rate and mask word segmentation "[MASK]". i The word segmentation in the sequence is randomly masked and replaced to obtain the corresponding second pre-segmentation sequence sb. i If the current training phase is the second phase, then based on the masked word segmentation "[MASK]", each of the first pre-segmented word sequences sa i Each word segment representing a specific disease type is masked and replaced to obtain the corresponding second pre-segmentation sequence sb. i ; Step 73, divide each of the second pre-segmented word sequences sb i The sequence index of each masked word segment in the image is used as a corresponding first training index s. i,n ; and each of the second presegmented word sequences sb i All corresponding first training indices s i,n Form a corresponding first training index sequence S i ; and based on each of the first training indices s i,n Set a corresponding first label vector y i,n ; and each of the first training index sequences S i All corresponding first label vectors y i,n Form a corresponding first label vector sequence Y i ; Wherein, each of the first training index sequences S i The first training index s i,n The total number is denoted as N. i 1 ≤ index n ≤ N i The first label vector y i,n The length of the vector is equal to the total number of words in the model vocabulary of the base large language model, and the total number of words in the model vocabulary of the base large language model is denoted as W; the first label vector y i,n Composed of multiple first label probabilities ρ i,n,w Composition, the first label probability ρ i,n,w The word segmentation corresponds one-to-one with the word segmentation in the model vocabulary, where 1 ≤ index w ≤ W; the first tag vector y i,n There is only one first label with probability ρ i,n,w The first label vector y is 1, and all others are 0; i,n The first label probability ρ, specifically 1, is... i,n,w The corresponding word segmentation in the model vocabulary and the first training index s i,n In the first presegmented word sequence sa i Corresponding word segmentation matching in; Step 74, transfer each of the first text t i and the corresponding first training index sequence S i The first text and the first training index sequence are input into the central pre-training framework for a forward inference; and the first probability vector sequence output from this inference is used as the corresponding first prediction vector sequence. The first text feature vector output by the first preprocessing module during this inference process is denoted as the corresponding first feature vector x. i ; Wherein, each of the first prediction vector sequences From the corresponding N i First prediction vector Composition; the first prediction vector Given W first prediction probabilities composition; Step 75, from each of the first label vector sequences Y i and its corresponding first prediction vector sequence Form a corresponding first data group And by N a The first feature vector x i Form the corresponding first vector sequence {x i }; Step 76, obtain N a The first data group Substitute the preset first model loss function L1 into the calculation to obtain the corresponding first loss value; Wherein, the loss function L1 of the first model is: Step 77: Identify whether the first loss value meets the preset first loss value range; if the first loss value meets the first loss value range, identify whether the current training stage is the first stage; if so, reset the current training stage to the second stage and return to step 72; otherwise, proceed to step 78; if the first loss value does not meet the first loss value range, based on the preset first model optimizer, perform a round of modulation on the model parameters of the first feature projection model, the base large language model, and the first mask prediction model in the central pre-training framework in the direction of minimizing the first model loss function L1, and return to step 74 when the modulation ends. The first model optimizer includes at least the Adam optimizer and the SGD optimizer; Step 78, the latest first vector sequence {x} i Substitute the preset first discriminant loss L D1 and the first generation loss L G1 ; and based on a preset second model optimizer, it moves towards making the first discriminative loss L D1 The model parameters of the first feature projection model in the central pre-trained framework and the first discriminator of the first generative adversarial network are modulated in one round to reach the minimum value; and at the end of this round of modulation, a preset third model optimizer is used to optimize the first generative loss L. G1 The model parameters of the first feature projection model and the first generator of the first generative adversarial network are modulated in one round to reach the minimum value, and the training is confirmed to be over when the modulation is completed. Wherein, the first discrimination loss L D1 for: θ 11 F represents the model parameters of the first feature projection model. 11 () is the inference function of the first feature projection model; F 11 (x i ;θ 11 ) is used to represent the projection model with parameter θ. 11 And the model input is a text feature vector x i Under the given conditions, the text mapping vector output by the first feature projection model; θ 12 For the model parameters of the first discriminator of the first generative adversarial network, F 12 () is the inference function of the first discriminator; F 12 [F 11 (x i ;θ 11 );θ 12 ] is used to indicate that the model parameters of the discriminator are θ 12 The discriminator input is a text mapping vector F. 11 (x i ;θ 11 Under the condition that the first discriminator outputs the discrimination probability; The first generation loss L G1 for: θ 13 For the model parameters of the first generator of the first generative adversarial network, F 13 () is the inference function of the first generator; F 13 [F 11 (x i ;θ 11 ),θ 13 ] is used to indicate that the model parameters in the generator are θ 13 And the model input is a text mapping vector F 11 (x i ;θ 11 Under the condition of ), the text scrambling vector output by the first generator; F 12 {F 13 [F 11 (x i ;θ 11 ),θ 13 ];θ 12 } is used to represent the model parameters of the discriminator being θ. 12 The discriminator input is a text scrambling vector F. 13 [F 11 (x i ;θ 11 ),θ 13 Under the condition of ], the discrimination probability output by the first discriminator; The second model optimizer includes at least the Adam optimizer and the SGD optimizer; The third model optimizer includes at least the Adam optimizer and the SGD optimizer.

8. The federated learning system for a medical large language model according to claim 1, characterized in that, The central server is specifically used when setting the corresponding fine-tuning layer number and distributing model parameters based on the current version number: The system identifies whether the version number matches a preset initial version number. If a match is found, the fine-tuning layer number is set to 0, and the central model parameter corresponding to the current version number is used as the corresponding distributed model parameter; if a mismatch is found, the layer number corresponding to the last Encoder layer of the base large language model is used as the fine-tuning layer number, and the model parameter in the central model parameter corresponding to the current version number that corresponds to the last Encoder layer is used as the corresponding distributed model parameter.

9. The federated learning system for a medical large language model according to claim 1, characterized in that, The central server is specifically used when the base model parameters are updated according to each of the parameter directories to obtain a new version of the central model parameters: Each of the parameter directories is taken as the corresponding current directory; and the fine-tuning layer number and the central model parameter stored in the current directory are taken as the corresponding current number and current central model parameter; and the current number is identified. If the current number is 0, then all parameters of the current central model parameter are used as the current baseline parameter; If the current number is greater than 0, then the model layer parameter corresponding to the current number in the current central model parameter is taken as the current baseline parameter; and it is identified whether the total number of the organization model parameters stored in the current directory matches the total number of the organization network. If so, model parameter fusion is performed on all the organization model parameters in the current directory to obtain the corresponding current fusion parameter, and based on the current fusion parameter, some or all of the parameters in the current central model parameter corresponding to the current baseline parameter are replaced to obtain a new version of the central model parameter; Specifically, the model parameter fusion is as follows: 1 ≤ Institutional Index k ≤ N org , θ k For the kth parameter of the mechanism model in the current directory, θ R The current fusion parameters are defined as follows.

10. The federated learning system for a medical large language model according to claim 6, characterized in that, The institution server is specifically used during a local training session based on the fine-tuning layer number, the text set, the local fine-tuning framework, and the local task framework: Identify the current fine-tuning layer number; If the current fine-tuning layer number is 0, then the first adjustable parameter set is composed of all model parameters of the second feature projection model, the institutional big language model, the second mask prediction model, and the second generative adversarial network; and the second adjustable parameter set is composed of all model parameters of the institutional big language model and the institutional text classification model. If the current fine-tuning layer number is greater than 1, then the first adjustable parameter set is composed only of the overall mode parameters of the second feature projection model and the model layer parameters of the overall model parameters of the institutional large language model that correspond to the current fine-tuning layer number; and the second adjustable parameter set is composed only of the model layer parameters of the overall model parameters of the institutional large language model that correspond to the current fine-tuning layer number and the overall model parameters of the institutional text classification model. A pre-training process is performed based on the text set, the local fine-tuning framework, and the first adjustable parameter set; after the pre-training is completed, a task training process is performed based on the text set, the local task framework, and the second adjustable parameter set; and at the end of the task training, the local training is confirmed to be complete.

11. The federated learning system for a medical large language model according to claim 10, characterized in that, The institution server is specifically used during a pre-training process based on the text set, the local fine-tuning framework, and the first adjustable parameter set: Step 111: Count the total number of the first medical texts in the text set to obtain the corresponding total number N. b ; and each of the first medical texts is recorded as the corresponding second text t. j 1 ≤ index j ≤ N b ; and according to the word segmentation rules of the aforementioned large language model, each of the second texts t j Pre-segmentation is performed to obtain the corresponding third pre-segmented word sequence sc. j ; and according to the preset second mask rate and mask segmentation "[MASK]", each of the third pre-segmented word sequences sc j The word segments in the sequence are randomly masked and replaced to obtain the corresponding fourth pre-segmentation sequence sd. j ; Step 112, divide each of the fourth pre-segmented word sequences sd j The sequence index of each masked word segment in the image is used as a corresponding second training index s. j,m ; and by each of the aforementioned fourth pre-segmentation sequences sd j All corresponding second training indices s j,m Form a corresponding second training index sequence S j ; and based on each of the second training indices s j,m Set a corresponding second label vector y j,m ; and by each of the second training index sequences S j All corresponding second label vectors y j,m Form a corresponding second label vector sequence Y j ; Wherein, each of the second training index sequences S j The second training index s j,m The total number is denoted as N. j 1 ≤ index m ≤ N j The second label vector y j,m The length of the vector is consistent with the total number of word segments in the model vocabulary of the institutional large language model and the total number of word segments W in the model vocabulary of the base large language model; the second label vector y j,m Composed of multiple second label probabilities ρ j,m,w Composition, the second label probability ρ j,m,w The word segmentation corresponds one-to-one with the word segmentation in the model vocabulary, where 1 ≤ index w ≤ W; the second label vector y j,m There is only one second label probability ρ j,m,w The second label vector y is 1, and all others are 0; j,m The second label probability ρ, specifically 1, is... j,m,w The corresponding word segmentation in the model vocabulary and the second training index s j,m In the third pre-segmentation sequence sc j Corresponding word segmentation matching in; Step 113, transfer each of the second text t j and the corresponding second training index sequence S j The second text and the second training index sequence are input into the local fine-tuning framework for a forward inference; and the second probability vector sequence output in this iteration is used as the corresponding second prediction vector sequence. The second text feature vector output by the second preprocessing module during this inference process is denoted as the corresponding second feature vector x. j The second text encoding vector output by the institutional large language model during this inference process is denoted as the corresponding first encoding vector e. j ; Among them, each of the second prediction vector sequences From the corresponding N j The second prediction vector Composition; the second prediction vector Given W second prediction probabilities composition; Step 114, from each of the second label vector sequences Y j and its corresponding second prediction vector sequence Form a corresponding second data group And by N b The second feature vector x j Form the corresponding second vector sequence {x j }; Step 115, convert each of the first encoding vectors e j This serves as the corresponding current encoding vector, and noise is added to the current encoding vector to obtain the corresponding positive sample vector. And the remaining N besides the current encoding vector b -1 of the first encoded vector e j Each is denoted as the corresponding negative sample vector. 1≤index u≤N b -1; and by each of the first encoded vectors e j and its corresponding positive sample vector and N b -1 of the negative sample vectors Form a corresponding third data group Step 116: Denote the parameter portion of the mechanism-wide language model in the first adjustable parameter set as parameter θ. l , the parameter θ L The corresponding model parameters are denoted as parameter θ. C ; and the obtained N a The second and third data groups The corresponding second loss value is obtained by substituting the preset second model loss function L2 into the calculation. Wherein, the loss function L2 of the second model is: L2=L 21 +αL 22 +β||θ l -θ C || 2 , α and β are two pre-set weighting parameters, τ is two pre-set temperature coefficients, and the loss function L 21 Implemented based on the cross-entropy loss function, loss function L 22 Implemented based on the contrastive loss function; ||θ l -θ C || 2 For the latest parameter θ l With the parameter θ C The L2 regularization; sim() is the cosine similarity function of vectors; For each of the first encoding vectors e j The corresponding positive sample vector cosine similarity, For the first encoding vector e j Each of the corresponding negative sample vectors Cosine similarity; Step 117: Identify whether the second loss value meets the preset second loss value range; if the second loss value meets the second loss value range, proceed to step 118; if the second loss value does not meet the second loss value range, perform a round of modulation on the model parameters in the first adjustable parameter set based on the preset fourth model optimizer in the direction of minimizing the second model loss function L2, and return to step 113 when the current round of modulation ends. The fourth model optimizer includes at least the Adam optimizer and the SGD optimizer; Step 118, the latest second vector sequence {x} j Substitute the preset second discriminant loss L D2 Second generation loss L G2 ; and based on the preset fifth model optimizer, it moves towards making the second discriminative loss L D2 The model parameters in the first adjustable parameter set are modulated once in the direction that reaches the minimum value; and at the end of this modulation round, the second generation loss L is then modulated based on the preset sixth model optimizer. G2 The model parameters in the first adjustable parameter set are modulated once in the direction that reaches the minimum value; and the training ends when the modulation ends. Wherein, the second discriminant loss L D2 for: G is the first set of adjustable parameters; θ 21 The model parameters for the second feature projection model; θ 21 ∩G represents the adjustable parameter portion corresponding to the second feature projection model in the first adjustable parameter set; F 21 () is the inference function of the second feature projection model; F 21 (x j ;θ 21 ) is used to represent the projection model with parameter θ. 21 And the model input is a text feature vector x j Under the condition that the second feature projection model outputs the text mapping vector; θ 22 The model parameters for the second discriminator in the second generative adversarial network; θ 22 ∩G represents the adjustable parameter portion corresponding to the second discriminator in the first adjustable parameter set; F 22 () is the inference function of the second discriminator; F 22 [F 21 (x j ;θ 21 );θ 22 ] is used to indicate that the model parameters of the discriminator are θ 22 The discriminator input is a text mapping vector F. 21 (x j ;θ 21 Under the condition that ), the discrimination probability output by the second discriminator; The second generation loss L G2 for: θ 23 θ represents the model parameters of the second generator in the second generative adversarial network. 23 ∩G represents the adjustable parameter portion corresponding to the second generator in the first adjustable parameter set; F 23 () is the inference function of the second generator; F 23 [F 21 (x j ;θ 21 ),θ 23 ] is used to indicate that the generator model parameter is θ 23 And the model input is a text mapping vector F 21 (x j ;θ 21 Under the condition of ), the text scrambling vector output by the second generator; F 22 {F 23 [F 21 (x j ;θ 21 ),θ 23 ];θ 22 } is used to represent the discriminator model parameters being θ. 22 The discriminator input is a text scrambling vector F. 23 [F 21 (x j ;θ 21 ),θ 23 Under the condition of ], the discrimination probability output by the second discriminator; The fifth model optimizer includes at least the Adam optimizer and the SGD optimizer; The sixth model optimizer includes at least the Adam optimizer and the SGD optimizer.

12. The federated learning system for a medical large language model according to claim 10, characterized in that, The institution server is specifically used during a task training operation based on the text set, the local task framework, and the second adjustable parameter set: Step 121: Count the total number of the first medical texts in the text set to obtain the corresponding total number N. c ; and record each of the first medical texts as the corresponding third text t. h 1 ≤ index h ≤ N c ; and according to the word segmentation rules of the aforementioned institutional large language model, each of the aforementioned third texts t h Pre-segmentation is performed to obtain the corresponding fifth pre-segmented word sequence se. h And denote the first classification label corresponding to each of the first medical texts as the corresponding label vector Y. h ; Wherein, the label vector Y h Includes multiple label probabilities ρ h,v 1 ≤ index v ≤ N d N d The total number of preset disease types; Step 122, transfer each of the third texts t h The corresponding third text is input into the local task framework for a forward inference; and the first classification probability vector output in this instance is used as the corresponding prediction vector. The second text mapping vector output by the second feature projection model during this inference process is denoted as the corresponding mapping vector X. h ; Wherein, the prediction vector Includes multiple predicted probabilities Step 123, from each of the stated label vectors Y h and the corresponding prediction vector Form a corresponding fourth data group And set a feature dimension and the mapping vector X h A consistent standard normal distribution space, and N randomly sampled from said standard normal distribution space. e Let X be a spatial vector. g ; and by the mapping vector X h and the space vector P g Pairwise combinations yield N c ×N e The fifth data set (X) h ,P g ); Where, N e Let N be a positive integer, and N be a positive integer. e >N c ; 1 ≤ index g ≤ N e ; Step 124: Denote the overall model parameters of the second feature projection model as parameter θ. proj ; and the parameter θ proj N c The fourth data group N c ×N e The fifth data group (X) h ,P g Substitute the input into the preset third model loss function L3 to calculate the corresponding third loss value; The third model loss function L3 is: γ and λ are two pre-set weight parameters; 1 ≤ index h * ≤N c , 1≤index g * ≤N e Loss function L 31 Implemented based on the cross-entropy loss function, loss function L 32 Implemented based on the maximum mean difference loss function. For the latest parameter θ proj The Frobenius norm regularity; f kernel () is a pre-defined kernel function with two input variables; Step 125: Identify whether the third loss value meets the preset range of the third loss value; if the third loss value does not meet the range of the third loss value, then based on the preset seventh model optimizer, perform a round of modulation on the model parameters in the second adjustable parameter set in the direction of minimizing the third model loss function L3, and return to step 122 when the modulation ends; if the third loss value meets the range of the third loss value, then stop training and confirm the end of the training for this task.

Images