Medical department recommendation method and device based on federal large model, equipment and medium

Abstract

The invention discloses a medical department recommendation method and device based on a federal large model, equipment and a medium, and relates to the technical field of large models, and the method comprises the steps that a server side sends a divided hidden layer and a remaining hidden layer to a calculation node of a medical institution, so that the calculation node inputs a historical electronic medical record text to a model input layer for processing, processing the first processing result output by the model input layer to obtain a second processing result; determining a target divergence based on the first processing result and the second processing result; inputting the second processing result into the remaining hidden layer for processing to obtain a third processing result, so that the computational node determines a loss function; training the remaining hidden layers based on a target gradient determined according to the loss function; generating a target medical department recommendation large model according to the divided hidden layers and the trained remaining hidden layers; and generating a medical department recommendation reply result by using the target medical department recommendation large model and the medical department query request. And a medical department recommendation result is accurately obtained.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to a method, apparatus, equipment and medium for recommending medical departments based on a federated large model. Background Technology

[0002] To reduce the computational and communication burden on the client-side, Split Federated Learning (SFL) has been proposed as a novel distributed training paradigm that balances privacy protection and resource efficiency. It can be effectively used for fine-tuning large federated models, hence the term "Federated Split Large Model Fine-tuning." This method structurally splits the large model into client and server parts: the client retains the input layer and the intermediate layers closer to the input (hereinafter referred to as the "client model," where the client model of the nth client is denoted as...). The server-side manages the computationally intensive intermediate layers (hereinafter referred to as the "server-side model"). During training, the client and server collaborate on forward and backward propagation by passing activation values ​​and their corresponding gradients. Since the server-side models are centrally hosted on the server and have no parameter differences, aggregation is unnecessary. However, client-side models can only be updated based on their local data, leading to differences between models. Therefore, periodic client-side model aggregation is required to achieve global consistency. The aggregation process occurs on the server side, and the aggregated client-side model is represented as follows. Therefore, assuming there are N clients, the relationship between the aggregated client model and the individual client models is as follows:

[0003] ;

[0004] Under this training mechanism, the parameters uploaded from the client to the server for aggregation only include the intermediate layers closer to the input side, rather than the complete model parameters. This significantly reduces communication overhead and model synchronization latency while ensuring training effectiveness.

[0005] However, in existing technologies, client-side models typically need to be periodically uploaded to the server for aggregation to integrate knowledge learned from local data by different clients. However, in large-scale model fine-tuning scenarios, the sheer number of model parameters, even involving only some intermediate layer parameters, generates a large amount of data transmission. This periodic model aggregation process significantly increases communication latency, placing a high communication load on client devices and hindering efficient federated fine-tuning. Furthermore, in heterogeneous client environments, the computing power and communication bandwidth of different devices vary significantly. Model aggregation often requires waiting for all clients to upload before unified aggregation can occur, leading to lag issues. The upload process of slow devices becomes a bottleneck for the overall system training speed. Compared to traditional small-scale models, the large number of parameters in large models further amplifies this problem, significantly reducing the overall model fine-tuning efficiency. Moreover, existing technologies do not fully consider the personalized needs of clients during model aggregation. Different clients have different local data distributions, task objectives, and application scenarios, and a unified aggregated client model cannot adequately accommodate the personalized characteristics of each client. In scenarios where adaptive fine-tuning and optimization based on local data are required, the lack of personalized model design and adjustment mechanisms will limit the performance of local models. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for recommending medical departments based on a federated large model. This method reduces the amount of parameter transmission between the client and server, decouples communication overhead from model size, thereby effectively reducing communication latency and bandwidth consumption, and improving the overall efficiency and scalability of federated fine-tuning. The specific solution is as follows:

[0007] Firstly, this application discloses a medical department recommendation method based on a federal large-scale model, applied to the server side, including:

[0008] The partitioned hidden layer and the remaining hidden layer are sent to the computing nodes of each medical institution. These computing nodes input historical electronic medical record text into the model input layer for processing, and then process the first processing result output from the model input layer using the partitioned hidden layer to obtain a corresponding second processing result. The partitioned hidden layer is the hidden layer adjacent to the model input layer among the two hidden layers obtained after partitioning the hidden layers in the initial medical department recommendation model. The remaining hidden layer is the hidden layer remaining after removing the partitioned hidden layer from the hidden layers in the initial medical department recommendation model. The initial medical department recommendation model is a model built based on a federated large model.

[0009] Obtain the first processing result and the second processing result sent by the computing node, and determine the target divergence based on the first processing result and the second processing result;

[0010] The second processing result is input into the remaining hidden layer for processing to obtain a third processing result. The third processing result and the target divergence are sent to the computing node so that the computing node can determine the loss function based on the third processing result and the target divergence, and determine the target gradient based on the loss function.

[0011] Obtain the target gradient sent by the computing node, and train the remaining hidden layers based on the target gradient;

[0012] Based on the partitioned hidden layers and the remaining hidden layers after training, a large model for recommending target medical departments is generated.

[0013] Obtain a medical department query request, and use the target medical department recommendation model to respond to the medical department query request, so as to generate corresponding medical department recommendation response results.

[0014] Optionally, determining the target divergence based on the first processing result and the second processing result includes:

[0015] The first processing result is input into the partitioned hidden layer for processing to obtain the target processed result;

[0016] The target divergence is determined based on the processed target result and the second processing result using a target divergence determination formula; the target divergence determination formula is:

[0017] ;

[0018] in, Let the target divergence be denoted as . Operators for calculating KL divergence; The result after processing the target; This is the result of the second processing; It represents a probability distribution.

[0019] Optionally, before sending the third processing result and the target divergence to the computing node so that the computing node determines the loss function based on the third processing result and the target divergence, the method further includes:

[0020] The computing node determines the regularization parameter based on the total number of batches of data processed by the computing node using a preset regularization parameter determination formula; the preset regularization parameter determination formula is as follows:

[0021] ;

[0022] in, The regularization parameter is... is the preset initial value for the regularization parameter; e is the natural constant; The attenuation coefficient; This represents the total number of batches of data processed by the computing node.

[0023] Optionally, sending the third processing result and the target divergence to the computing node, so that the computing node determines the loss function based on the third processing result and the target divergence, includes:

[0024] The third processing result and the target divergence are sent to the computing node so that the computing node can determine the loss function based on the regularization parameter, the third processing result, and the target divergence.

[0025] Optionally, sending the third processing result and the target divergence to the computing node, so that the computing node determines the loss function based on the regularization parameter, the third processing result, and the target divergence, includes:

[0026] The third processing result and the target divergence are sent to the computing node so that the computing node can input the third processing result into the model output layer for processing, determine the initial loss function based on the corresponding processing result, determine the target product of the target divergence and the regularization parameter, and determine the loss function by summing the target product and the initial loss function.

[0027] Optionally, training the remaining hidden layer based on the target gradient includes:

[0028] The remaining hidden layers are trained by backpropagation based on the target gradient to obtain the trained remaining hidden layers.

[0029] Optionally, the step of generating a target medical department recommendation model based on the partitioned hidden layers and the remaining hidden layers after training includes:

[0030] The backpropagation gradient obtained by backpropagating the remaining hidden layers is sent to the computing node so that the computing node can backpropagate the partitioned hidden layers based on the backpropagation gradient to obtain the trained partitioned hidden layers. Based on the trained partitioned hidden layers and the trained remaining hidden layers, the current medical department recommendation model is determined, and the process jumps back to the computing node to input the historical electronic medical record text into the model input layer for processing, until the current medical department recommendation model is obtained.

[0031] Secondly, this application discloses a medical department recommendation device based on a federal big data model, applied on the server side, including:

[0032] The hidden layer sending module is used to send the partitioned hidden layer and the remaining hidden layer to the computing nodes of each medical institution, so that the computing nodes can input the historical electronic medical record text into the model input layer for processing, and process the first processing result output by the model input layer through the partitioned hidden layer to obtain the corresponding second processing result; the partitioned hidden layer is the hidden layer adjacent to the model input layer among the two hidden layers obtained after partitioning the hidden layer in the initial medical department recommendation model; the remaining hidden layer is the hidden layer remaining after removing the partitioned hidden layer from the hidden layer in the initial medical department recommendation model; the initial medical department recommendation model is a model built based on a federated large model;

[0033] The target divergence determination module is used to acquire the first processing result and the second processing result sent by the computing node, and determine the target divergence based on the first processing result and the second processing result;

[0034] The target gradient determination module is used to input the second processing result into the remaining hidden layer for processing to obtain a third processing result, and send the third processing result and the target divergence to the computing node so that the computing node can determine the loss function based on the third processing result and the target divergence, and determine the target gradient based on the loss function.

[0035] The training module is used to obtain the target gradient sent by the computing node and train the remaining hidden layer based on the target gradient;

[0036] The large model generation module is used to generate a target medical department recommendation large model based on the partitioned hidden layer and the remaining hidden layer after training.

[0037] The response module is used to obtain medical department query requests and respond to the medical department query requests using the target medical department recommendation model to generate corresponding medical department recommendation response results.

[0038] Thirdly, this application discloses an electronic device, including:

[0039] Memory, used to store computer programs;

[0040] The processor is used to execute computer programs to implement the healthcare department recommendation method based on the aforementioned federated big model.

[0041] Fourthly, this application discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for recommending medical departments based on a federated big model.

[0042] In this application, the server sends the partitioned hidden layer and the remaining hidden layer to the computing nodes of each medical institution. The computing nodes then input historical electronic medical record text into the model input layer for processing, and process the first processing result output by the model input layer through the partitioned hidden layer to obtain a corresponding second processing result. The partitioned hidden layer is the hidden layer adjacent to the model input layer among the two hidden layers obtained after partitioning the hidden layers in the initial medical department recommendation model. The remaining hidden layer is the hidden layer remaining after removing the partitioned hidden layer from the hidden layers in the initial medical department recommendation model. The initial medical department recommendation model is a model built based on a federated model. The first processing result and the second processing result sent by the computing nodes are obtained. The target divergence is determined based on the first and second processing results; the second processing result is input into the remaining hidden layer for processing to obtain a third processing result; the third processing result and the target divergence are sent to the computing node so that the computing node determines a loss function based on the third processing result and the target divergence, and determines a target gradient based on the loss function; the target gradient sent by the computing node is obtained, and the remaining hidden layer is trained based on the target gradient; a target medical department recommendation model is generated based on the partitioned hidden layer and the trained remaining hidden layer; a medical department query request is obtained, and the target medical department recommendation model is used to respond to the medical department query request to generate a corresponding medical department recommendation response result. It can be seen that this application determines the target divergence through the processing structure of historical electronic medical record text by the hidden layer and input layer, determines the loss function based on the target divergence, and then trains the model based on the loss function. This application proposes a novel client-side model knowledge aggregation method based on target divergence. This method eliminates the need for client-side model parameter transmission. Instead, it aggregates client-side model knowledge by sending the divergence information from the server to the corresponding client and adding it to the client's original loss function. This significantly reduces the amount of parameter transmission between the client and server, decoupling communication overhead from model size, thereby effectively reducing communication latency and bandwidth consumption, and improving the overall efficiency and scalability of federated fine-tuning. Furthermore, by not relying on unified parameter aggregation, it enhances the generalization ability and local adaptability of client-side models in different scenarios. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This application discloses a flowchart of a medical department recommendation method based on a federal big data model.

[0045] Figure 2 This is a schematic diagram of a federal large model fine-tuning process disclosed in this application;

[0046] Figure 3 This is a schematic diagram of a medical department recommendation device based on a federal big data model disclosed in this application;

[0047] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] The bottleneck in existing technologies lies in client-side model aggregation. Existing client-side model aggregation methods result in high communication load, latency issues, and the aggregated client-side models fail to consider the personalized needs of individual clients. To address these technical problems, this application discloses a method, apparatus, device, and medium for recommending medical departments based on a federated large model. This method reduces the amount of parameter transmission between the client and server, decouples communication overhead from model size, thereby effectively reducing communication latency and bandwidth consumption, and improving the overall efficiency and scalability of federated fine-tuning.

[0050] See Figure 1 As shown, this embodiment of the invention discloses a medical department recommendation method based on a federated large model, applied to the server side, including:

[0051] Step S11: Send the partitioned hidden layer and the remaining hidden layer to the computing nodes of each medical institution, so that the computing nodes can input the historical electronic medical record text into the model input layer for processing, and process the first processing result output by the model input layer through the partitioned hidden layer to obtain the corresponding second processing result; the partitioned hidden layer is the hidden layer adjacent to the model input layer among the two hidden layers obtained after partitioning the hidden layer in the initial medical department recommendation model; the remaining hidden layer is the hidden layer remaining after removing the partitioned hidden layer from the hidden layer in the initial medical department recommendation model; the initial medical department recommendation model is a model built based on the federated model.

[0052] In this embodiment, the server divides the pre-trained large model into several parts, including the input layer and the hidden layer after division. The output layer sends the data to the client (in this embodiment, the client is a computing node of a medical institution) for processing. The server retains all intermediate layer parameters; the remaining intermediate layer parameters are the remaining hidden layer parameters. In this application, the partitioned hidden layer is the hidden layer adjacent to the model input layer among the two hidden layers obtained after partitioning the hidden layers in the initial medical department recommendation model; the remaining hidden layer is the hidden layer remaining after removing the partitioned hidden layer from the hidden layers in the initial medical department recommendation model.

[0053] Then, the computing node inputs the historical electronic medical record text into the model input layer for processing, and processes the first processing result output by the model input layer through the partitioned hidden layer to obtain the corresponding second processing result. In a specific embodiment, the client uses local data input into the input layer and the client model, and through forward propagation, first calculates the activation value output by the input layer. Then calculate the activation value output by the client model. .

[0054] Step S12: Obtain the first processing result and the second processing result sent by the computing node, and determine the target divergence based on the first processing result and the second processing result.

[0055] In this embodiment, the computing node sends the first processing result and the second processing result to the server. Specifically, the client of this application sends the calculated activation value. and Send them together to the server. Then, based on the first processing result ( ) and the second processing result ( To determine the target divergence, the first processing result is input into the partitioned hidden layer for further processing to obtain the target-processed result. The target divergence is determined based on the processed target result and the second processing result using a target divergence determination formula; the target divergence determination formula is:

[0056] ;

[0057] in, Let the target divergence be denoted as . Operators for calculating KL divergence; The result after processing the target; This is the result of the second processing; It represents a probability distribution.

[0058] Step S13: Input the second processing result into the remaining hidden layer for processing to obtain the third processing result. Send the third processing result and the target divergence to the computing node so that the computing node can determine the loss function based on the third processing result and the target divergence, and determine the target gradient based on the loss function.

[0059] In this embodiment, the server will receive the activation value. The input is fed into the server-side model to calculate the activation values. The server then combines the activation values ​​calculated by the server-side model with the calculated KL divergence. The data is sent to the corresponding client along with the data. In this application, the computing node determines the regularization parameter based on the total number of batches of data processed by the computing node using a preset regularization parameter determination formula; the preset regularization parameter determination formula is as follows:

[0060] ;

[0061] in, The regularization parameter is... ≥0 is the preset initial value of the regularization parameter; e is the natural constant; >0 represents the attenuation coefficient; This represents the total number of batches of data processed by the computing node. In the initial stage of fine-tuning, the client model needs to fully utilize the server's global knowledge to quickly adapt to the current federated fine-tuning task; however, as training progresses and the client continues to process more local data (i.e., ... Increase), regularization weight By gradually reducing the weights exponentially, the dependence on the global model is decreased, allowing the model to gradually conform to the local data distribution and achieve personalized optimization. Furthermore, the degree of client-side personalization can be adjusted through initial weights. With attenuation coefficient Co-adjustment: For greater personalization, a larger adjustment can be selected. With smaller This allows the model to align with global knowledge in the early stages, but shifts towards local adaptation more quickly in the later stages; if generality and global consistency are emphasized more, a smaller value can be chosen. With larger This allows the client to maintain stronger global constraints throughout the fine-tuning process.

[0062] Therefore, in this application, the third processing result and the target divergence are sent to the computing node so that the computing node inputs the third processing result into the model output layer for processing, in order to determine the initial loss function based on the corresponding processing result, and to determine the target product of the target divergence and the regularization parameter, and to determine the loss function by summing the target product with the initial loss function.

[0063] Specifically, the client receives the activation value and KL divergence (forward KL divergence, relative entropy) returned by the server. Then, the loss function is calculated using the output layer. The total loss function is Finally, the gradient can be determined based on the total loss function.

[0064] Step S14: Obtain the target gradient sent by the computing node, and train the remaining hidden layer based on the target gradient.

[0065] In this embodiment, the client will output the total loss function of the layer. The calculated gradient is sent to the server. Upon receiving the gradient from the client, the server performs backpropagation and updates its model. In other words, the server trains the remaining hidden layers using backpropagation based on the target gradient to obtain the trained remaining hidden layers.

[0066] Step S15: Generate a large model for recommending target medical departments based on the partitioned hidden layers and the remaining hidden layers after training.

[0067] In this embodiment, the server sends the gradient obtained from backpropagation of the server-side model to the client. After receiving the gradient from the server, the client backpropagates and updates its client model. Finally, it jumps back to the computation node to input the historical electronic medical record text into the model input layer for processing, until the model converges (model convergence refers to the convergence of the complete model, which is composed of the input layer, the aggregated client model (the trained and updated hidden layer after partitioning), the server-side model centrally hosted on the server (the trained and updated remaining hidden layer), and the output layer). The backpropagation gradient obtained from backpropagation training of the remaining hidden layer is sent to the computation node, so that the computation node can backpropagate and train the hidden layer after partitioning based on the backpropagation gradient to obtain the trained hidden layer after partitioning. Based on the trained hidden layer after partitioning and the trained remaining hidden layer, the current medical department's recommended large model is determined, and it jumps back to the computation node to input the historical electronic medical record text into the model input layer for processing, until the current medical department's recommended large model is obtained.

[0068] Step S16: Obtain the medical department query request, and use the target medical department recommendation model to respond to the medical department query request to generate the corresponding medical department recommendation response result.

[0069] In this embodiment, after fine-tuning the model, that is, after obtaining the target medical department recommendation model, a medical department query request can be received, and the target medical department recommendation model can be used to respond to the medical department query request to generate corresponding medical department recommendation response results. The overall fine-tuning process of the large model in this application is as follows: Figure 2 As shown, this application designs a novel client-side model knowledge aggregation method based on KL divergence, eliminating the need for client-side model parameter transmission. The client needs to additionally transmit the activation values ​​output from the input layer to the server. The server calculates the activation values ​​using the global client model and then uses this result to calculate KL divergence with the activation values ​​output and uploaded by the client model. This KL divergence represents the degree of knowledge similarity between the global client model and a specific client model. The server sends this KL divergence information to the corresponding client, adding it to the client's original loss function to achieve client model knowledge aggregation. A new loss function is designed, which is the original loss function multiplied by the KL divergence and a preset controllable variable. A controllable variable is designed to adjust the impact of KL divergence on the loss function. This controllable variable can be adjusted according to the client's personalized needs for the model. A larger controllable variable means a closer similarity between the client model and the global client model, indicating weaker personalized capabilities in the client model but better global knowledge acquisition and stronger ability to handle globally applicable data tasks. The smaller the controllable variable (greater than or equal to 0), the more the client-side model tends to process local client data, and the stronger the client-side model's personalization capabilities.

[0070] As can be seen, this application determines the target divergence through the processing structure of historical electronic medical record texts in the hidden layer and input layer, and determines the loss function based on the target divergence, then trains the model based on the loss function. In this way, this application designs a novel client-side model knowledge aggregation method based on target divergence. It eliminates the need for client-side model parameter transmission; instead, it aggregates client-side model knowledge by sending the divergence information from the server to the corresponding client and adding it to the client's original loss function. This significantly reduces the amount of parameter transmission between the client and server, decoupling communication overhead from model size, thereby effectively reducing communication latency and bandwidth consumption, and improving the overall efficiency and scalability of federated fine-tuning. By not relying on unified parameter aggregation, it improves the generalization ability and local adaptability of the client-side model in different scenarios.

[0071] As described in the previous embodiment, this application discloses a method for recommending medical departments based on a federated large model. This method achieves medical department recommendation using the large model through fine-tuning. Next, specific embodiments will be used to describe in detail the medical department recommendation method based on the federated large model of this application.

[0072] I. System and Participants:

[0073] 1. Server-side: Deploy complete middleware parameters, including global client-side model parameters and server-side model parameters, providing alignment calculation and server-side forward / backward calculation capabilities.

[0074] 2. Client: Computing nodes from multiple medical institutions (such as hospitals A / B / C) hold their own private electronic medical record text data (including chief complaint, present illness, past medical history, examination results, etc.), and locally retain the input layer, client model and output layer (task header).

[0075] 3. Heterogeneous configuration: Client A (high-end GPU), Client B (mid-range GPU), Client C (CPU / low-end GPU).

[0076] II. Data and Tasks:

[0077] 1. Task type: Multi-label department recommendation (text classification).

[0078] 2. Pre-trained large model: Llama3.2-1B.

[0079] 2. Local data: Each client uses historical electronic medical records from within the hospital as training data; privacy data does not leave the domain.

[0080] 3. Task Loss Function: The cross-entropy loss function is used for classification tasks.

[0081] III. Model Decomposition:

[0082] 1. Input layer (client-private): text segmentation and embedding.

[0083] 2. Client-side model: Several intermediate layers located close to the input side.

[0084] 3. Global client model (held by the server): The structure is aligned with the client model.

[0085] IV. Preset values ​​for regularization parameters:

[0086] Hospital A: , ;

[0087] Hospital B: , ;

[0088] Hospital C: , .

[0089] V. Training Process:

[0090] Step 1 (Model Distribution): The server distributes the input layer and client model. The output layer distributes the data to each client, while the server retains the data. and .

[0091] Step 2 (Local Forward): The client generates local text. ,through get .

[0092] Step 3 (Uplink Communication): Client Upload and .

[0093] Step 4 (Server-side alignment reference): The server uses... As input get .

[0094] Step 5 (Alignment Calculation): Server-side calculation =KL(P )||P( )) and return (Optionally, return alignment-related intermediate values ​​as well).

[0095] Step 6 (Server-side forward): The server will... enter The server is activated and the result is returned to the client.

[0096] Step 7 (Local Total Loss and Reverse): The client uses the returned server-side activation function to calculate the task loss. , and according to The total loss is then propagated backward. Depend on Updated online.

[0097] Steps 8-12 (Gradient Interaction and Update): The client outputs the total loss function of the layer. The calculated gradients are sent to the server. Upon receiving the gradients from the client, the server backpropagates and updates its model. The server then sends the gradients obtained from backpropagation of its model to the client. Upon receiving the gradients from the server, the client backpropagates and updates its model. Both the client and server update their local parameters; periodic aggregation of client model parameters is not performed.

[0098] Step 13 (Convergence Determination): Execute the loop until the validation set metrics stabilize or the maximum number of iterations is reached. Repeat steps 2-12 until the model converges.

[0099] In addition, it should be noted that the method of this application is not limited to medical department recommendation, does not depend on task-specific structure, and can also be applied to various text tasks such as triage question answering and entity recognition in large health models.

[0100] As can be seen, in the case of cross-institutional, non-independent and identically distributed medical text data, this embodiment can achieve alignment through alignment items without transmitting client model parameters. This enables knowledge sharing, reduces communication pressure, and avoids aggregation waits caused by lagging systems. The adaptive computation leverages global knowledge for rapid alignment in the early stages of training, gradually enhancing the fit to the local distribution in later stages to achieve personalized improvements; devices with strong personalization needs can achieve this through a smaller... With larger This method enhances the personalized adaptation of the model. By preserving a personalized model structure on the client side and utilizing knowledge aggregation to update personalized parameters, each client can maintain its adaptability to local data features while sharing global knowledge. Instead of relying on unified parameter aggregation, this method achieves cross-client knowledge sharing through lightweight knowledge distillation, i.e., KL divergence. This balances personalized model performance with global collaborative effects, improving the generalization ability and local adaptability of client models in different scenarios. Clients can independently complete local training and knowledge uploads based on their own computing and communication capabilities, without waiting for the upload progress of other clients. The server can update the aggregation results in real time after receiving knowledge from some clients, thus avoiding the problem of slow devices becoming system bottlenecks. This mechanism improves the system's fault tolerance and flexibility, significantly accelerates model convergence, and improves training efficiency in heterogeneous network environments.

[0101] See Figure 3 As shown, this embodiment of the invention discloses a medical department recommendation device based on a federated large model, applied to the server side, including:

[0102] The hidden layer sending module 11 is used to send the partitioned hidden layer and the remaining hidden layer to the computing nodes of each medical institution, so that the computing nodes can input the historical electronic medical record text into the model input layer for processing, and process the first processing result output by the model input layer through the partitioned hidden layer to obtain the corresponding second processing result; the partitioned hidden layer is the hidden layer adjacent to the model input layer among the two hidden layers obtained after partitioning the hidden layer in the initial medical department recommendation model; the remaining hidden layer is the hidden layer remaining after removing the partitioned hidden layer from the hidden layer in the initial medical department recommendation model; the initial medical department recommendation model is a model built based on the federated model;

[0103] The target divergence determination module 12 is used to obtain the first processing result and the second processing result sent by the computing node, and determine the target divergence based on the first processing result and the second processing result;

[0104] The target gradient determination module 13 is used to input the second processing result into the remaining hidden layer for processing to obtain a third processing result, and send the third processing result and the target divergence to the computing node so that the computing node can determine the loss function based on the third processing result and the target divergence, and determine the target gradient based on the loss function.

[0105] Training module 14 is used to obtain the target gradient sent by the computing node and train the remaining hidden layer based on the target gradient;

[0106] Large model generation module 15 is used to generate a target medical department recommendation large model based on the partitioned hidden layer and the remaining hidden layer after training.

[0107] The response module 16 is used to obtain medical department query requests and respond to the medical department query requests using the target medical department recommendation big model to generate corresponding medical department recommendation response results.

[0108] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0109] As can be seen, this application determines the target divergence through the processing structure of historical electronic medical record texts in the hidden layer and input layer, and determines the loss function based on the target divergence, then trains the model based on the loss function. In this way, this application designs a novel client-side model knowledge aggregation method based on target divergence. It eliminates the need for client-side model parameter transmission; instead, it aggregates client-side model knowledge by sending the divergence information from the server to the corresponding client and adding it to the client's original loss function. This significantly reduces the amount of parameter transmission between the client and server, decoupling communication overhead from model size, thereby effectively reducing communication latency and bandwidth consumption, and improving the overall efficiency and scalability of federated fine-tuning. By not relying on unified parameter aggregation, it improves the generalization ability and local adaptability of the client-side model in different scenarios.

[0110] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0111] Figure 4This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the federalized large-scale model-based medical department recommendation method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.

[0112] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0113] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0114] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the federalized big data model-based medical department recommendation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0115] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for recommending medical departments based on a federated big data model. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0117] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0119] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for recommending medical departments based on a federated large model, characterized in that, Applied to the server side, including: The partitioned hidden layer and the remaining hidden layer are sent to the computing nodes of each medical institution. These computing nodes input historical electronic medical record text into the model input layer for processing, and then process the first processing result output from the model input layer using the partitioned hidden layer to obtain a corresponding second processing result. The partitioned hidden layer is the hidden layer adjacent to the model input layer among the two hidden layers obtained after partitioning the hidden layers in the initial medical department recommendation model. The remaining hidden layer is the hidden layer remaining after removing the partitioned hidden layer from the hidden layers in the initial medical department recommendation model. The initial medical department recommendation model is a model built based on a federated large model. Obtain the first processing result and the second processing result sent by the computing node, and determine the target divergence based on the first processing result and the second processing result; The second processing result is input into the remaining hidden layer for processing to obtain a third processing result. The third processing result and the target divergence are sent to the computing node so that the computing node can determine the loss function based on the third processing result and the target divergence, and determine the target gradient based on the loss function. Obtain the target gradient sent by the computing node, and train the remaining hidden layers based on the target gradient; Based on the partitioned hidden layers and the remaining hidden layers after training, a large model for recommending target medical departments is generated. Obtain a medical department query request, and use the target medical department recommendation model to respond to the medical department query request, so as to generate corresponding medical department recommendation response results.

2. The medical department recommendation method based on a federated large model according to claim 1, characterized in that, The step of determining the target divergence based on the first processing result and the second processing result includes: The first processing result is input into the partitioned hidden layer for processing to obtain the target processed result; The target divergence is determined based on the processed target result and the second processing result using a target divergence determination formula; the target divergence determination formula is: ； in, Let the target divergence be denoted as . Operators for calculating KL divergence; The result after processing the target; This is the result of the second processing; It represents a probability distribution.

3. The medical department recommendation method based on a federated large model according to claim 1, characterized in that, Before sending the third processing result and the target divergence to the computing node so that the computing node determines the loss function based on the third processing result and the target divergence, the method further includes: The computing node determines the regularization parameter based on the total number of batches of data processed by the computing node using a preset regularization parameter determination formula; the preset regularization parameter determination formula is as follows: ； in, The regularization parameter is... is the preset initial value for the regularization parameter; e is the natural constant; The attenuation coefficient; This represents the total number of batches of data processed by the computing node.

4. The medical department recommendation method based on a federated large model according to claim 3, characterized in that, The step of sending the third processing result and the target divergence to the computing node, so that the computing node determines the loss function based on the third processing result and the target divergence, includes: The third processing result and the target divergence are sent to the computing node so that the computing node can determine the loss function based on the regularization parameter, the third processing result, and the target divergence.

5. The medical department recommendation method based on a federated large model according to claim 4, characterized in that, The step of sending the third processing result and the target divergence to the computing node, so that the computing node determines the loss function based on the regularization parameter, the third processing result, and the target divergence, includes: The third processing result and the target divergence are sent to the computing node so that the computing node can input the third processing result into the model output layer for processing, determine the initial loss function based on the corresponding processing result, determine the target product of the target divergence and the regularization parameter, and determine the loss function by summing the target product and the initial loss function.

6. The medical department recommendation method based on a federated large model according to claim 1, characterized in that, Training the remaining hidden layers based on the target gradient includes: The remaining hidden layers are trained by backpropagation based on the target gradient to obtain the trained remaining hidden layers.

7. The method for recommending medical departments based on a federated large model according to any one of claims 1 to 6, characterized in that, The process of generating a target medical department recommendation model based on the partitioned hidden layers and the remaining hidden layers after training includes: The backpropagation gradient obtained by backpropagating the remaining hidden layers is sent to the computing node so that the computing node can backpropagate the partitioned hidden layers based on the backpropagation gradient to obtain the trained partitioned hidden layers. Based on the trained partitioned hidden layers and the trained remaining hidden layers, the current medical department recommendation model is determined, and the process jumps back to the computing node to input the historical electronic medical record text into the model input layer for processing, until the current medical department recommendation model is obtained.

8. A medical department recommendation device based on a federated large model, characterized in that, Applied to the server side, including: The hidden layer sending module is used to send the partitioned hidden layer and the remaining hidden layer to the computing nodes of each medical institution, so that the computing nodes can input the historical electronic medical record text into the model input layer for processing, and process the first processing result output by the model input layer through the partitioned hidden layer to obtain the corresponding second processing result; the partitioned hidden layer is the hidden layer adjacent to the model input layer among the two hidden layers obtained after partitioning the hidden layer in the initial medical department recommendation model; the remaining hidden layer is the hidden layer remaining after removing the partitioned hidden layer from the hidden layer in the initial medical department recommendation model; the initial medical department recommendation model is a model built based on a federated large model; The target divergence determination module is used to acquire the first processing result and the second processing result sent by the computing node, and determine the target divergence based on the first processing result and the second processing result; The target gradient determination module is used to input the second processing result into the remaining hidden layer for processing to obtain a third processing result, and send the third processing result and the target divergence to the computing node so that the computing node can determine the loss function based on the third processing result and the target divergence, and determine the target gradient based on the loss function. The training module is used to obtain the target gradient sent by the computing node and train the remaining hidden layer based on the target gradient; The large model generation module is used to generate a target medical department recommendation large model based on the partitioned hidden layer and the remaining hidden layer after training. The response module is used to obtain medical department query requests and respond to the medical department query requests using the target medical department recommendation model to generate corresponding medical department recommendation response results.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the healthcare department recommendation method based on a federal big model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, implements the healthcare department recommendation method based on a federated big model as described in any one of claims 1 to 7.

Images