Model debiasing method, device, equipment and medium based on medical large model

Abstract

Embodiments of the present disclosure disclose a model debiasing method, device, equipment and medium based on a medical large model. A specific implementation of the method comprises: performing multi-level debiasing correction on a multi-modal medical data set, and then performing data balanced grouping to obtain a multi-modal balanced medical training data set; performing resource detection on a server cluster; inputting the multi-modal balanced medical training data set into a medical large model to obtain a multi-modal medical diagnosis information set; inputting the multi-modal medical diagnosis information set and a multi-modal medical sample diagnosis information set into a multi-modal adversarial fair debiasing loss function to obtain a model loss value; performing model debiasing processing on a multi-modal balanced medical test data set, and then performing model training on a drug recommendation model, and controlling a drug delivery robot to transport. The implementation can improve the resource load balancing of the server cluster, the quality of the medical data after debiasing, the training speed of the model, and the precise control of the robot to deliver drugs.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer technology, and in particular to a model debiasing method, device, equipment and medium based on a medical large model. BACKGROUND

[0002] With the development of medical artificial intelligence, medical multi-modal large models have been widely used in medical diagnosis, treatment plan recommendation and other medical links. However, medical training data used for medical multi-modal large models have noise interference, privacy risks and biases caused by language and regional differences, which seriously affect the development of medical multi-modal large models. For model debiasing based on a medical large model, the commonly used way is to use a large language model to perform large language model debiasing processing on a single data source to obtain a debiased single data source, and store the debiased single data source.

[0003] However, in practice, it is found that when the above method is used to debias the model based on the medical large model, the following technical problems often exist: due to the single data source itself having a data single data bias problem, and the large amount of multi-modal medical data, the training of the medical large model may not be performed on a resource-limited server or the time length of the training of the medical large model is prolonged, resulting in low accuracy of data debiasing and low quality of multi-modal medical data, a large amount of redundant and erroneous data, and thus reducing the model performance of the drug recommendation model, generating low-accuracy recommended drug information, causing the drug delivery robot to frequently transport drugs, prolonging the time for the drugs to reach the location of the patient who does not have the ability to move, increasing the severity of the patient, and reducing the user experience.

[0004] The above information disclosed in this BACKGROUND section is only for the purpose of enhancing the understanding of the background of the present disclosure, and therefore, it can contain information that is not known in the art. SUMMARY

[0005] The summary of the present disclosure is intended to introduce the concepts in a simplified form, which will be described in detail in the following detailed description. The summary of the present disclosure is not intended to identify key or essential features of the claimed technology, nor is it intended to be used to limit the scope of the claimed technology.

[0006] Some embodiments of the present disclosure propose a model debiasing method, device, equipment and medium based on a medical large model to solve one or more of the technical problems mentioned in the above BACKGROUND section.

[0007] In a first aspect, some embodiments of the present disclosure provide a model debiasing method based on a medical large model, comprising: performing multi-level debiasing correction on an obtained multi-modal medical data set to obtain a multi-modal debiased medical data set, wherein the multi-modal medical data set is obtained from a multi-modal medical corpus; performing data balance grouping processing on the multi-modal debiased medical data set to obtain a multi-modal balanced medical training data set and a multi-modal balanced medical test data set; performing resource detection on a server cluster, and selecting a server that meets a resource selection condition from the server cluster as a target server; controlling the target server to perform the following model debiasing steps based on the multi-modal balanced medical training data set: inputting the multi-modal balanced medical training data set into a medical large model to obtain a multi-modal medical diagnosis information set; inputting the multi-modal medical diagnosis information set and a multi-modal medical sample diagnosis information set into a multi-modal adversarial fair debiasing loss function corresponding to the medical large model to obtain a model loss value; in response to determining that the model loss value meets a preset loss condition, performing model debiasing processing on the multi-modal balanced medical test data set to obtain a debiased multi-modal medical data set; performing model training on a drug recommendation model according to the debiased multi-modal medical data set to obtain recommended drug information, and controlling a drug delivery robot to transport recommended drugs corresponding to the recommended drug information to a target patient's location.

[0008] In a second aspect, some embodiments of the present disclosure provide a model debiasing device based on a medical large model, comprising: a multi-level debiasing correction unit configured to perform multi-level debiasing correction on an obtained multi-modal medical data set to obtain a multi-modal debiased medical data set, wherein the multi-modal medical data set is obtained from a multi-modal medical corpus; a data balance grouping unit configured to perform data balance grouping processing on the multi-modal debiased medical data set to obtain a multi-modal balanced medical training data set and a multi-modal balanced medical test data set; a resource detection unit configured to perform resource detection on a server cluster, and select a server that meets a resource selection condition from the server cluster as a target server; a first control unit configured to control the target server to perform the following model debiasing steps based on the multi-modal balanced medical training data set: inputting the multi-modal balanced medical training data set into a medical large model to obtain a multi-modal medical diagnosis information set; inputting the multi-modal medical diagnosis information set and a multi-modal medical sample diagnosis information set into a multi-modal adversarial fair debiasing loss function corresponding to the medical large model to obtain a model loss value; in response to determining that the model loss value meets a preset loss condition, performing model debiasing processing on the multi-modal balanced medical test data set to obtain a debiased multi-modal medical data set; and a second control unit configured to perform model training on a drug recommendation model according to the debiased multi-modal medical data set to obtain recommended drug information, and control a drug delivery robot to transport recommended drugs corresponding to the recommended drug information to a target patient's location.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having stored thereon one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method as described in any implementation of the first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0011] The above various embodiments of the present disclosure have the following beneficial effects: the model debiasing method based on a medical large model of some embodiments of the present disclosure can improve the resource load balancing of a server cluster, the quality of medical data after debiasing, the training speed of the model, and the precise control of the robot delivering medicine. Specifically, the reasons for the low accuracy of data debiasing and the low quality of multi-modal medical data, the low model performance of the medicine recommendation model, the prolonged delivery time of medicine, the severity of the patient, and the low user experience are as follows: due to the data bias problem of single data source itself, and the large amount of multi-modal medical data, which may not be executed on a resource-limited server or prolong the training time of the medical large model, resulting in low accuracy of data debiasing and low quality of multi-modal medical data, a large amount of redundant and erroneous data, which in turn reduces the model performance of the medicine recommendation model, generates low-accuracy recommended medicine information, causes the medicine delivery robot to frequently transport medicine, prolongs the time of delivering medicine to the location of the patient who does not have the ability to move, increases the severity of the patient, and reduces the user experience. Based on this, the model debiasing method based on a medical large model of some embodiments of the present disclosure can first perform multi-level debiasing correction on the obtained multi-modal medical data set to obtain a multi-modal debiased medical data set, wherein the multi-modal medical data set is obtained from a multi-modal medical corpus. Here, obtaining the multi-modal medical data set from the multi-modal medical corpus can improve the diversity of medical data, can reduce the bias caused by a single data source to a certain extent, and the multi-level debiasing correction can include sensitivity debiasing, model explanation debiasing, and causal relationship debiasing. Sensitivity debiasing can prevent the direct input of minority group features into the medical large model from causing bias, model explanation debiasing can debias by generating local explanations of multi-modal medical data, and can avoid model bias caused by improper feature selection, and causal relationship debiasing can determine whether the causal relationship between the input and output diagnostic information included in the multi-modal medical data set is reasonable, can remove features that deviate from the causal relationship, improve the causal rationality of the input data, and reduce model bias caused by false causality. After the multi-modal debiased medical data set is processed by data balancing grouping, a multi-modal balanced medical training data set and a multi-modal balanced medical test data set are obtained. Here, the balance of the input data of the subsequent medical large model can be balanced, and the bias caused by the majority during the model training process can be avoided. Then, the server cluster is resource detected, and servers that meet the resource selection condition are selected from the server cluster as target servers. Here, resource detection can real-time master the running status of the server cluster, facilitate the sending of the multi-modal balanced medical training data set and the multi-modal balanced medical test data set to the server with sufficient resources, accelerate the training speed of the subsequent medical large model, and improve the resource load balancing of the server cluster.Then, the target server is controlled to perform the following model debiasing steps based on the multi-modal balanced medical training dataset: first, input the multi-modal balanced medical training dataset into the medical large model to obtain a multi-modal medical diagnosis information set. Here, through the learning ability of the medical large model, the performance and accuracy of debiasing can be improved, the group difference and bias of the multi-modal balanced medical training dataset are reduced, and the accuracy of the multi-modal medical diagnosis information is improved. Second, input the multi-modal medical diagnosis information set and the multi-modal medical sample diagnosis information set into the multi-modal adversarial fairness debias loss function corresponding to the medical large model to obtain a model loss value. Here, by incorporating the adversarial loss, fairness correction term and group reconstruction regularization constraint in the loss function, the processing performance and training efficiency of the model for multi-modal medical data can be further improved. Third, in response to determining that the model loss value meets the preset loss condition, the multi-modal balanced medical test dataset is subjected to model debiasing processing to obtain a multi-modal medical dataset after debiasing. Here, the accuracy of data debiasing and the quality of the multi-modal medical data after debiasing can be improved. Finally, according to the multi-modal medical dataset after debiasing, the drug recommendation model is trained to obtain recommended drug information, and a drug delivery robot is controlled to transport the recommended drug corresponding to the recommended drug information to the target patient's location. Here, the accuracy of the recommended drug information can be improved, the transportation time and frequency of the drug delivery robot can be reduced, the time for patients without the ability to act to obtain drugs can be shortened, and user experience can be improved. Thus, the model debiasing method based on the medical large model can effectively reduce the bias caused by a single data source by obtaining a multi-modal medical dataset from a multi-modal medical corpus. By debiasing in the data preprocessing process and the model training process, the bias and potential bias factors in the data can be accurately identified and corrected, key information can be avoided from being mistakenly deleted, the balance of different group data is ensured, the model can learn the characteristics of different groups more fairly during the training process, the model bias caused by data bias is effectively reduced, the training efficiency of the medical large model is improved, the debiasing accuracy is improved, the training speed of the training medical large model and the drug recommendation model is improved, the cluster load is improved, and the robot transportation time and user experience are reduced. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail some embodiments thereof with reference to the attached drawings in which:

[0013] Figure 1 is a flowchart of some embodiments of the model debiasing method based on a medical large model according to the present disclosure;

[0014] Figure 2 is a schematic diagram of a causal path generated in some embodiments of the model debiasing method based on medical large model according to the present disclosure;

[0015] Figure 3 is a schematic diagram of a comparison between the present disclosure and existing debiasing techniques in terms of accuracy and fairness in some embodiments of the model debiasing method based on medical large model according to the present disclosure;

[0016] Figure 4 is a structural schematic diagram of some embodiments of the model debiasing apparatus based on medical large model according to the present disclosure;

[0017] Figure 5 is a structural schematic diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are merely for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0020] It should be noted that the terms “first”, “second”, and the like mentioned in the present disclosure are merely used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the terms “one”, “multiple” mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless otherwise explicitly stated in the context, it should be understood as “one or more”.

[0022] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are merely for illustrative purposes and are not intended to limit the scope of the messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0024] Figure 1Flow 100 showing some embodiments of the medical large model based model debiasing method according to the present disclosure. The medical large model based model debiasing method comprises the following steps:

[0025] Step 101, performing multi-level debiasing correction on the obtained multi-modal medical data set to obtain a multi-modal debiased medical data set.

[0026] In some embodiments, the subject performing the above-mentioned medical large model based model debiasing method (for example, an electronic device) can perform multi-level debiasing correction on the obtained multi-modal medical data set to obtain a multi-modal debiased medical data set, wherein the multi-modal medical data set is obtained from a multi-modal medical corpus. Wherein the multi-modal medical corpus can be a database that stores data after data preprocessing by establishing a secure encrypted data interface, using a data acquisition engine to automatically collect data of different modalities from multiple channels such as HIS system (Hospital Information System), PACS system (Picture Archiving and Communication System), electronic medical record database and public medical database according to a preset time. The preset time can be a pre-set collection time. For example, the preset time can be early morning every day. The multi-modal medical data in the multi-modal medical data set can be medical data of different data formats representing patient body state information. The different data formats can include but are not limited to at least one of the following: text data, medical images, user body state time series data obtained by sensors. The sensors can include but are not limited to at least one of the following: electronic watch, smart phone, blood pressure monitoring instrument. The multi-modal debiased medical data in the multi-modal debiased medical data set can be medical data obtained by removing data with bias or unfairness in the multi-modal medical data set.

[0027] In some optional implementations of some embodiments, the multi-modal medical corpus can be obtained by the following steps:

[0028] First, collect multi-modal regional medical data sets of different languages in different regions. The multi-modal regional medical data in the multi-modal regional medical data set can be medical data including different languages and different data formats for different regional medical habits. For example, the multi-modal regional medical data can be the treatment method of using ginger and brown sugar water to assist the treatment of cold in northern China.

[0029] Secondly, a medical credibility set of the multi-modal regional medical data set is determined. The medical credibility in the medical credibility set can represent the credibility of the multi-modal regional medical data. The greater the medical credibility value is, the greater the quality and credibility of the multi-modal regional medical data are. In practice, the execution subject can first perform medical entity relation extraction on the medical text data set included in the multi-modal regional medical data set to obtain a medical entity set. The medical entity in the medical entity set can be a triple entity related to human body detection data. For example, the medical entity can be {patient identity information: xxx, entity name: white blood cell, count: 1000, unit: 10^9 / L, context description: white blood cell count is significantly increased}. The medical entity relation extraction can be named entity recognition and relation extraction by using a series of BiLSTM (Bidirectional Long Short-Term Memory)-CRF (Conditional Random Fields) model and a pre-training model. The pre-training model can be BERT (Bidirectional Encoder Representations from Transformers). Then, the medical entity set is used to perform multi-dimensional index retrieval on a preset medical data feature library to obtain a medical entity multi-dimensional index group set. The medical entity multi-dimensional index in the medical entity multi-dimensional index group set can be index information used to measure the credibility evaluation of the medical entity. The medical entity multi-dimensional index can include but is not limited to at least one of the following: medical entity normal value range, abnormal value range, medical entity associated index consistency, data source reliability. The preset medical data feature library can be a pre-set knowledge base related to medical data. Finally, for each medical entity in the medical entity set, the following determination steps are performed: firstly, the index weight value group of the medical entity multi-dimensional index group of the medical entity is determined. Secondly, the index weight value group and the medical entity multi-dimensional index group are weighted and summed to obtain a medical credibility.

[0030] Thirdly, according to the medical credibility set, the multi-modal regional medical data set is processed by a de-duplication based on a sparse attention mechanism to obtain a de-duplicated medical data set.

[0031] As an example, the execution subject can first screen at least one multi-modal regional medical data corresponding to the medical credibility satisfying a preset credibility condition from the multi-modal regional medical data set. Wherein, the preset credibility condition can be a condition that the medical credibility is greater than or equal to a preset credibility threshold. The preset credibility threshold can be a minimum value preset for evaluating the credibility of the medical entity. For example, the preset credibility threshold can be 0.6. Then, using the coarse-grained label compression technology in the sparse attention mechanism, the at least one multi-modal regional medical data is quickly screened to obtain a screened multi-modal regional medical data set. Wherein, the quick screening can be a preliminary screening with the user identification information of the patient as a coarse-grained label. The user identification information can represent the unique identification information of the user identity information. Then, in response to detecting that there is a medical entity with the same user identification information in the screened multi-modal regional medical data set, using the fine-grained label selection technology in the sparse attention mechanism, the screened multi-modal regional medical data set is accurately compared. Wherein, the accurate comparison can be a comparison by indicators such as visit time, diagnosis result, treatment plan, etc. It should be noted that this step can ensure that the repeated data is accurately identified and removed, greatly improving the deduplication efficiency while maintaining the perception of the global context and local precision. The sparse attention mechanism can be a mechanism that retains key information and reduces redundant calculations through attention mechanism. The sparse attention mechanism can include but is not limited to at least one of the following: NSA (Nested Sparse Attention, native sparse attention mechanism), MOBA (Mixture of Block Assignment), MLA (Multi-Head Latent Attention).

[0032] In the fourth step, the multi-modal feature alignment is performed on the deduplicated medical data set and the multi-modal regional medical data set after removing the medical text data set, to obtain a multi-modal associated medical diagnosis information set. The multi-modal associated medical diagnosis information in the multi-modal associated medical diagnosis information set can be information from different modalities, which can reduce the inter-modal heterogeneity difference and the semantic integrity of each modality. In practice, the execution subject can first perform multi-modal data extraction on the deduplicated medical data set and the multi-modal regional medical data set after removing the medical text data set, to obtain a multi-modal medical diagnosis information set. The multi-modal medical diagnosis information in the multi-modal medical diagnosis information set can be medical feature information obtained by performing feature extraction on medical data of different modalities. The multi-modal data extraction can include: text recognition of image data by an optical character recognition algorithm, accurate segmentation of image data by an image semantic segmentation algorithm to identify lesion region information, and multi-modal data extraction by processing the deduplicated medical data set by part-of-speech tagging and named entity recognition through natural language processing technology. Then, cross-modal data alignment is performed on the multi-modal medical diagnosis information set to obtain a multi-modal associated medical diagnosis information set. For example, the multi-modal medical diagnosis information set can include lung CT (Computed Tomography, computed tomography) image and corresponding medical record text description "patient has ground glass opacity in the lung, considering early lung cancer". The cross-modal data alignment can be achieved by the following steps: first, feature extraction is performed on each region in the CT image (for example, shape, density, etc. of the ground glass opacity region), to obtain a region feature information set. Second, the keywords "lung", "ground glass opacity", "early lung cancer" in the medical record text are simultaneously subjected to semantic understanding and feature extraction, to obtain a medical record feature information set. Third, the multi-modal alignment is performed on the region feature information set and the medical record feature information set by a cross-modal attention mechanism, to obtain a multi-modal associated medical diagnosis information set. It should be noted that this step can accurately associate data of different modalities, ensure data annotation consistency, and further enhance the fusion understanding ability of the large language model for multi-modal data.

[0033] In the fifth step, the multi-modal associated medical diagnosis information set is subjected to generative data augmentation to obtain a multi-modal augmented medical data set. The multi-modal augmented medical data in the multi-modal augmented medical data set can be medical data obtained by augmenting the multi-modal associated medical diagnosis information set. The generative data augmentation can be data augmentation of the multi-modal associated medical diagnosis information set by using a generative adversarial network model in combination with a large language model. For example, taking the rare disease patient data of cystic fibrosis as an example, first, the large language model analyzes the feature distribution information of the existing cystic fibrosis patient data. The feature distribution information can include, but is not limited to, at least one of the following: patient genetic characteristics, clinical manifestations (for example, the frequency of occurrence of symptoms such as dyspnea and repeated lung infections), treatment plans (for example, specific drug information and dosage used). Then, a new data sample is generated according to the feature distribution information by using a generative adversarial network, and a multi-modal associated augmented medical information set is obtained. The new data sample can include a new combination of genetic sequences, a combination of different degrees of symptoms, and a corresponding treatment plan adjustment. It should be noted that this step can balance the sample size of different groups by data augmentation, and significantly improve the representativeness of the samples.

[0034] In the sixth step, the multi-modal augmented medical data set is subjected to privacy classification isolated storage to obtain a multi-modal medical corpus. The privacy classification isolated storage can be performed by data segmentation, encrypted transmission and isolated storage. It should be noted that this step can avoid data leakage and cross contamination, and ensure the fairness and reliability of model training.

[0035] Optionally, the privacy classification isolated storage of the multi-modal augmented medical data set to obtain the multi-modal medical corpus can include the following steps:

[0036] In the first step, the multi-modal augmented medical data set and a preset keyword set are subjected to keyword matching to obtain a keyword matching result set. The keyword matching result in the keyword matching result set can represent a result of whether the matching is successful. The preset keyword in the preset keyword set can be a pre-set word segmentation. For example, the preset keyword can include, but is not limited to, at least one of the following: diagnosis, prescription, chief complaint.

[0037] In the second step, at least one multi-modal augmented medical data corresponding to the keyword matching result representing a successful match is selected from the multi-modal augmented medical data set.

[0038] In the third step, the at least one multi-modal augmented medical data is stored in a database of the visit record category included in the multi-modal medical corpus after privacy processing. The privacy processing can be sensitive data encryption and anonymization processing of the multi-modal augmented medical data using differential privacy technology, or can be encryption privacy processing of sensitive numerical data using a homomorphic encryption algorithm.

[0039] In the fourth step, distance similarity matching is performed on the target multi-modal augmented medical data set and the preset dynamic medical record template set to obtain an entity template matching probability set. The target multi-modal augmented medical data set is obtained by removing the at least one multi-modal augmented medical data from the multi-modal augmented medical data set. The entity template matching probability in the entity template matching probability set can represent the matching degree of the target multi-modal augmented medical data and the preset dynamic medical record template. The distance similarity matching can be performed by weighted summation of the cosine similarity based on the Euclidean distance and the semantic similarity extracted by the BERT model. The preset dynamic medical record template in the preset dynamic medical record template set can be a template preset in advance about the medical record format. The preset dynamic medical record template set can be updated by a clustering algorithm every month to adapt to the newly added multi-modal augmented medical information set.

[0040] In the fifth step, at least one target multi-modal augmented medical data corresponding to an entity template matching probability greater than or equal to a preset matching probability threshold in the target multi-modal augmented medical data set is stored in an electronic medical record database included in the multi-modal medical corpus after privacy processing. The preset matching probability threshold can be a preset minimum value of the matching probability.

[0041] Optionally, after the at least one target multi-modal augmented medical data corresponding to an entity template matching probability greater than or equal to a preset matching probability threshold in the target multi-modal augmented medical data set is stored in an electronic medical record database included in the multi-modal medical corpus after privacy processing, the method further includes:

[0042] In a first step, the multi-modal augmented medical data set to be classified and the preset medical knowledge graph are standardized and verified for mapping to determine the medical knowledge graph storage category set of the multi-modal augmented medical data set to be classified. The multi-modal augmented medical data set to be classified is at least one target multi-modal augmented medical data set less than the preset matching probability threshold. The preset medical knowledge graph can be an existing knowledge graph related to medical data. In practice, the execution subject can first determine whether the medical entity set corresponding to the multi-modal augmented medical data set to be classified exists in the preset medical knowledge graph. Then, in response to the existence, the medical entity set corresponding to the multi-modal augmented medical data set to be classified is mapped for entity coding according to the preset medical knowledge graph to map non-standardized terms to standardized terms to obtain a standardized mapping entity set. Finally, the disease category of each standardized mapping entity in the standardized mapping entity set in the preset medical knowledge graph is determined as the medical knowledge graph storage category set.

[0043] In a second step, the multi-modal augmented medical data set to be classified is stored in the database of the medical knowledge graph category of the multi-modal medical corpus after being processed for privacy according to the medical knowledge graph storage category set.

[0044] In a third step, the data source set of each multi-modal augmented medical data in the multi-modal augmented medical data set to be classified is determined.

[0045] In a fourth step, the multi-modal augmented medical data set to be classified is stored in the database of the data source category after being processed for privacy according to the data source set. The database of the data source category can include but is not limited to at least one of the following: medical images, physical examination reports.

[0046] In some optional implementations of some embodiments, the multi-level debiasing correction of the obtained multi-modal medical data set to obtain a multi-modal debiased medical data set can include the following steps:

[0047] The first step involves extracting feature information from the aforementioned multimodal medical dataset to obtain a medical diagnostic feature information set. Each medical diagnostic feature in this set can influence the output diagnostic information of the large medical model and belongs to the medical attributes of the multimodal medical dataset. For example, the medical diagnostic feature information may include, but is not limited to, at least one of the following: gender, place of residence, and regional medical level. In practice, the execution entity can first perform anomaly detection on the multimodal medical dataset using a variational autoencoder to obtain an anomaly dataset. Secondly, the anomaly dataset is removed from the multimodal medical dataset to obtain an anomaly-removed multimodal medical dataset. Thirdly, a large language model is used to extract semantically ambiguous multimodal medical data from the anomaly-removed multimodal medical dataset to obtain a fuzzy medical dataset. Finally, a fuzzy mapping process is performed on the fuzzy medical dataset using a pre-defined medical fuzzy rule base to obtain a fuzzified medical dataset. This pre-defined medical fuzzy rule base can be a rule base that quantifies fuzzy semantics. For example, the aforementioned pre-defined medical fuzzy rule base may include a rule base that quantifies "slight cough" as "mild cough," "occasional cough" as "intermittent cough," and "severe cough" as "major cough." Then, the aforementioned fuzzified medical dataset is denoised to obtain a denoised multimodal medical dataset. Finally, principal component analysis is used to extract feature information from the denoised multimodal medical dataset to obtain a set of medical diagnostic feature information. It should be noted that the denoising process based on fuzzy mapping uses the uncertainty of the multimodal medical dataset for fuzzy processing. This can remove obviously erroneous data while reducing the risk of accidentally deleting potentially useful data, thus improving the quality of the denoised multimodal medical dataset and consequently improving the data quality of the medical diagnostic feature information.

[0048] The second step is to perform the following causal inference debiasing steps for each medical diagnostic feature in the above medical diagnostic feature information set:

[0049] Sub-step 1 involves performing a global perturbation process on the aforementioned medical diagnostic feature information to obtain feature sensitivity values. These feature sensitivity values ​​characterize the degree of impact on the diagnostic results when the attribute values ​​corresponding to the medical diagnostic feature information change.

[0050] As an example, the execution subject can first determine a mixed diagnosis feature information set for the medical diagnosis feature information through a preset medical knowledge graph. The mixed diagnosis feature information in the mixed diagnosis feature information set can be feature information that simultaneously affects the medical diagnosis feature information and the medical diagnosis information. Then, the medical diagnosis feature information is processed through a feature information disturbance algorithm to obtain disturbed medical diagnosis feature information. The feature information disturbance algorithm can be a random shuffle algorithm or an counterfactual simulation algorithm. Then, the medical diagnosis feature information, the mixed diagnosis feature information set, and the disturbed medical diagnosis feature information are input into a feature sensitivity analysis model to obtain a pre-disturbance diagnosis accuracy rate and a post-disturbance diagnosis accuracy rate. The feature sensitivity analysis model can be a machine learning model for determining the accuracy rate of the output diagnosis result after the attribute value of the medical diagnosis feature information is disturbed. For example, the feature sensitivity analysis model can be a logistic regression model. Finally, the difference between the pre-disturbance diagnosis accuracy rate and the post-disturbance diagnosis accuracy rate is determined as the feature sensitivity value.

[0051] In substep 2, the propensity causal confounding probability of the medical diagnosis feature information under different attribute values is determined according to the medical diagnosis feature information set. The propensity causal confounding probability can represent the diagnosis accuracy rate before and after the medical diagnosis feature information is disturbed under the condition that the mixed influence feature information is the same.

[0052] As an example, the execution subject can use PSM (propensity score matching) to determine the propensity causal confounding probability of the medical diagnosis feature information under different attribute values according to the medical diagnosis feature information set.

[0053] In step 3, the multi-modal medical data set is regressed and debiased according to the obtained propensity causal confounding probability set and the obtained feature sensitivity value set to obtain a sensitive debiased multi-modal medical data set. The sensitive debiased multi-modal medical data can be multi-modal medical data obtained by removing the medical diagnosis feature information.

[0054] As an example, the execution subject can use DML (double machine learning) to regress and debias the multi-modal medical data set according to the obtained propensity causal confounding probability set and the obtained feature sensitivity value set to obtain sensitive debiased multi-modal medical data.

[0055] In the fourth step, the sensitive debiased multi-modal medical data set is subjected to feature local explanation debiasing processing to obtain an explanation debiased multi-modal medical data set. In the explanation debiased multi-modal medical data set, the explanation debiased multi-modal medical data can be obtained by removing at least one medical diagnosis feature information from the sensitive debiased multi-modal medical data, where the removed medical diagnosis feature information has an influence difference greater than or equal to a preset influence difference threshold. In practice, the execution subject can first map the multiple medical diagnosis feature information included in the sensitive debiased multi-modal medical data to a preset medical knowledge graph for knowledge representation by using a Med-PaLM (Medical Pathways Language Model) model to obtain a medical diagnosis feature vector set. Then, the SHAP (Shapley Additive Explanations) model explanation technology is used to determine the feature prediction contribution probability value of each medical diagnosis feature information in the multiple medical diagnosis feature information to the diagnosis information output by the medical large model to obtain a feature prediction contribution probability value set. The feature prediction contribution probability value can be a Shapley value. Subsequently, the feature prediction contribution probability value set is subjected to group grouping and aggregation to obtain a feature prediction contribution probability value group set. Then, the group coefficient of variation set of each feature prediction contribution probability value group in the feature prediction contribution probability value group set is determined. The group coefficient of variation can be the ratio of the standard deviation and the mean of each feature prediction contribution probability value group. Finally, at least one medical diagnosis feature information corresponding to a group coefficient of variation greater than or equal to a preset group coefficient of variation threshold is removed from the sensitive debiased multi-modal medical data to obtain an explanation debiased multi-modal medical data. The preset group coefficient of variation threshold can be a preset minimum value for measuring the difference between groups.

[0056] In the fifth step, the explanation debiased multi-modal medical data set is subjected to causal inference debiasing processing to obtain a causal debiased multi-modal medical data set as a multi-modal debiased medical data set. In practice, the execution subject can first identify abnormal feature causal relationships in the explanation debiased multi-modal medical data set to obtain an abnormal feature causal relationship set. The abnormal feature causal relationship in the abnormal feature causal relationship set can be a causal relationship that has an abnormality in the causal relationship in the explanation debiased multi-modal medical data. The abnormal feature causal relationship identification can be abnormal relationship identification by using a causal graph model (Causal Graphical Model, CGM) or a PC (Peter-Clark) algorithm. Then, a set of medical diagnosis feature information corresponding to the abnormal feature causal relationship set is removed from the explanation debiased multi-modal medical data set to obtain a multi-modal debiased medical data set.

[0057] In the process of adopting technical solutions to solve the technical problems mentioned in the background, there are often the following technical problems: due to the existence of a large number of biased and unfair data in the multi-modal medical data set, when the multi-modal medical data set is used to train the drug recommendation model, the performance of the model is low, the drug recommendation accuracy is low, and the drug delivery robot frequently transports drugs, prolongs the medication time and aggravates the severity of patients without the ability to move. In view of the above technical problems, the conventional solution is generally: by using a correlation-based causal inference algorithm, the multi-modal medical data set is subjected to causal inference and bias removal to obtain sensitive bias-removed multi-modal medical data. However, the above conventional solution still has the following problems: due to the use of a single correlation relationship by the causal inference algorithm to remove bias in the causal relationship in the multi-modal medical data set, the inference of the causal relationship is one-sided, resulting in low accuracy of data bias removal, low quality of sensitive bias-removed multi-modal medical data, and further low performance of the drug recommendation model, low drug recommendation accuracy, frequent transportation of the drug delivery robot, prolonged medication time and aggravated severity of patients without the ability to move. And considering the shortcomings of the above conventional solution, combined with the advantages / technical status of the data processing bias removal algorithm possessed by the research institute partners in the field, we decide to adopt the following solution:

[0058] Optionally, the above regression bias removal of the multi-modal medical data set according to the obtained set of propensity causal confounding probabilities and the obtained set of feature sensitivity values, to obtain a set of sensitive bias-removed multi-modal medical data, and controlling the drug delivery robot to deliver drugs according to the set of sensitive bias-removed multi-modal medical data, can include the following steps:

[0059] First, for each medical diagnosis feature information in the set of medical diagnosis feature information, the following causal inference bias removal steps are performed:

[0060] Substep 1, obtain medical diagnosis intermediate feature information that has a causal relationship with the medical diagnosis feature information. The medical diagnosis intermediate feature information can be feature information that has a causal relationship with the medical diagnosis feature information obtained from a pre-set medical knowledge graph. The medical diagnosis feature information can be the cause, and the medical diagnosis intermediate feature information can be the result.

[0061] Sub-step 2, constructing a causal path set for the above medical diagnosis feature information, the above medical diagnosis intermediary feature information, the above confounding diagnosis feature information set and the medical diagnosis output information. The medical diagnosis output information can be information of a diagnosis result determined in the multi-modal medical data. The causal path in the causal path set can be a causal relationship existing in the form of a graph display. The causal path set can include: confounding diagnosis feature information-medical diagnosis feature information-medical diagnosis output information, medical diagnosis feature information-medical diagnosis intermediary feature information-medical diagnosis output information. As shown in Figure 2 Figure 2 The causal path set composed of medical identification impact feature information (for example, race sensitive attribute), the above medical identification intermediary feature information (for example, genetic attribute), the above confounding impact feature information (for example, regional attribute) and medical treatment information (for example, imaging medical feature) is displayed.

[0062] Sub-step 3, respectively determining diagnosis feature residual values and diagnosis output feature residual values of the above medical diagnosis feature information and the above medical diagnosis output information with respect to the confounding diagnosis feature information. The diagnosis feature residual value can represent the conditional expectation of the conditional dependence relationship of the confounding diagnosis information to the above medical diagnosis feature information. The diagnosis output feature residual value can represent the conditional expectation of the conditional dependence relationship of the confounding diagnosis information to the above medical diagnosis output information. The determination can be performed by using a random forest regression model.

[0063] Sub-step 4, determining local causal probabilities of the above medical diagnosis feature information and the above medical diagnosis output information according to the above diagnosis feature residual values, the above diagnosis output feature residual values, the set of propensity causal confounding probabilities and the set of feature sensitivity values. The local causal probability can represent a linear relationship between the medical diagnosis feature information and the above medical diagnosis output information. As an example, the execution subject can determine the local causal probability of the medical diagnosis feature information and the above medical diagnosis output information according to the above diagnosis feature residual values and the above diagnosis output feature residual values by using a linear regression model based on least squares fitting. The greater the local causal probability value, the stronger the causal relationship between the medical diagnosis feature information and the medical diagnosis output information.

[0064] Sub-step 5, in response to determining that the local causal probability is greater than or equal to a preset local causal probability threshold, performing counterfactual verification on the above medical diagnosis feature information to obtain causal relationship verification information. The causal relationship verification information can be verification information obtained by constructing a hypothetical scenario opposite to the fact by counterfactual reasoning technology to explore whether a causal relationship exists between the medical feature information and the medical diagnosis output information. The preset local causal probability threshold can be a minimum value preset for determining whether a causal relationship exists.​

[0065] In the second step, in response to determining that the obtained multiple pieces of causal relationship verification information all represent verification success, feature removal is performed on the multi-modal medical data set according to the obtained multiple local causal probabilities, to obtain a multi-modal medical data set after feature removal as a sensitive bias-removed multi-modal medical data set. The multi-modal medical data set after feature removal can be medical data after removing at least one medical diagnosis feature information with a local causal probability less than a preset local causal probability threshold from the multi-modal medical data set.

[0066] In the third step, the target server is controlled to perform model training on the drug recommendation model according to the sensitive bias-removed multi-modal medical data, to obtain recommendation drug information, and a drug delivery robot is controlled to transport a recommended drug set corresponding to the recommendation drug information to a location of the target patient. The execution subject can input the sensitive bias-removed multi-modal medical data into the drug recommendation model to perform model training, to obtain a trained drug recommendation model. Then, the body state information of the target patient is input into the trained drug recommendation model, to obtain drug recommendation information.

[0067] The above technical solutions and related content serve as one invention point of the embodiments of the present disclosure, and solve the technical problems mentioned in the background art. That is, due to the single correlation relationship between the causal relationship in the multi-modal medical data set through the causal inference algorithm, the inference of the causal relationship has certain one-sidedness, resulting in low accuracy of data debiasing, low quality of sensitive debiased multi-modal medical data, further leading to low performance of the drug recommendation model, low drug recommendation accuracy, frequent transportation of the drug delivery robot, prolonged medication time, and increased severity of patients without the ability to move. The accuracy of data debiasing is low, and the quality of sensitive debiased multi-modal medical data is low, which wastes a large amount of storage resources. If the above factors are solved, the accuracy of data debiasing can be improved, the quality and fairness of sensitive debiased multi-modal medical data can be improved, and the performance of the drug recommendation model is further low, the drug recommendation accuracy is low, the drug delivery robot is frequently transported, the medication time is prolonged, and the severity of patients without the ability to move is increased. In order to achieve this effect, the present disclosure first, the propensity causal confounding probability set and the feature sensitivity value can be obtained by PSM and DML, which can estimate the direct causal effect and remove the false association of sensitive features. Second, the causal path set including the medical diagnosis intermediate feature information and the residual analysis can separate the direct effect and the indirect effect of the feature. Then, through the multi-dimensional index residual value, the propensity causal confounding probability set and the feature sensitivity value, the bias path across modalities can be identified to ensure the rationality of the causal path. After that, through the counterfactual verification, the influence of the confounding diagnostic feature information can be further reduced, the debiasing accuracy of the multi-modal medical data set can be improved, and the quality and fairness of the sensitive debiased multi-modal medical data can be improved. Finally, the above target server is controlled, the drug recommendation model is trained according to the sensitive debiased multi-modal medical data, the recommended drug information is obtained, and the drug delivery robot is controlled to transport the recommended drug set corresponding to the recommended drug information to the location of the target patient, which can improve the accuracy of the recommended drug information, reduce the transportation frequency and transportation time of the drug delivery robot, and improve the user experience.

[0068] In step 102, the multi-modal debiased medical data set is subjected to data balanced grouping processing to obtain a multi-modal balanced medical training data set and a multi-modal balanced medical test data set.

[0069] In some embodiments, the execution subject can perform data balancing grouping processing on the multi-modal debiased medical dataset to obtain a multi-modal balanced medical training dataset and a multi-modal balanced medical test dataset. The multi-modal balanced medical training data in the multi-modal balanced medical training dataset can be data for training the subsequent medical large model and including balanced distribution of each medical diagnosis influencing feature information affecting the output of the diagnosis information. The medical diagnosis influencing feature information can include, but is not limited to, at least one of the following: gender feature information, race feature information, region feature information, and age feature information. The multi-modal balanced medical test data in the multi-modal balanced medical test dataset can be a dataset for medical diagnosis and debiasing of the medical large model. The execution subject can first use oversampling, undersampling, or SMOTE (Synthetic Minority Over-sampling Technique) algorithm to perform data balancing processing on the multi-modal debiased medical dataset to obtain a multi-modal balanced medical dataset. Then, the multi-modal balanced medical dataset is grouped to obtain a multi-modal balanced medical training dataset and a multi-modal balanced medical test dataset. The grouping can be grouping of the multi-modal balanced medical training dataset and the multi-modal balanced medical test dataset accounting for 0.8 and 0.2 proportions of the multi-modal debiased medical dataset, respectively.

[0070] In step 103, resource detection is performed on the server cluster, and servers meeting the resource selection condition are selected from the server cluster as target servers.

[0071] In some embodiments, the execution subject can perform resource detection on the server cluster, and select servers meeting the resource selection condition from the server cluster as target servers. The server cluster can be a cluster composed of multiple servers for deploying and training the medical large model. The resource selection condition can be a condition that the idle resources of the server are greater than the required resources for running the multi-modal balanced medical training dataset, the multi-modal balanced medical test dataset, and the medical large model. The target server can be the server with the most idle resources among the at least one server meeting the resource selection condition.

[0072] In step 104, the target server is controlled to perform the following model debiasing steps based on the multi-modal balanced medical training dataset:

[0073] In step 1041, the multi-modal balanced medical training dataset is input into the medical large model to obtain a multi-modal medical diagnosis information set.

[0074] In some embodiments, the above execution subject can input the multi-modal balanced medical training dataset into a medical large model to obtain a multi-modal medical diagnosis information set, wherein the medical large model comprises: a multi-modal extraction network, a multi-modal alignment collaborative network, a meta learning monitoring network, a bias detector, a fair adversarial network, and a multi-task learning network. The multi-modal medical diagnosis information in the multi-modal medical diagnosis information set can be information obtained by using the medical large model to diagnose diseases for the input multi-modal balanced medical training dataset. The medical large model can be a deep neural network model that diagnoses diseases for the input multi-modal balanced medical training dataset and outputs diagnosis information. The multi-modal extraction network can be a neural network model that integrates a Transformer model and a convolutional neural network model (e.g., a residual network model) to extract feature information of different modal medical data of the input multi-modal balanced medical training dataset in parallel, and outputs a multi-modal feature vector set. The multi-modal alignment collaborative network can be a deep neural network model that automatically adjusts the weight ratio between different modal feature vectors included in the multi-modal feature vector set output by the multi-modal extraction network, and can be matched and aligned in terms of features and semantics using a co-training mechanism (Co-training). The multi-modal alignment collaborative network can be a SCAN (Stacked Cross Attention Network) that uses a co-training mechanism. The meta learning monitoring network can be a meta learner that updates the hyperparameters of the medical large model or initializes the initial parameters of the medical large model based on meta learning (Meta Learning) combined with hyperparameter adaptive adjustment of the medical large model, to quickly search when the data distribution changes. The bias detector can be a neural network model that includes a multi-layer Transformer model in series, which detects feature distribution differences of the feature vectors output by the multi-modal alignment collaborative network to detect fairness. The fair adversarial network can be a generative adversarial network that generates a false feature vector set based on the feature vector set output by the bias detector to perform adversarial training, so that the medical large model pays more attention to the fairness of different feature groups. The multi-task learning network can be a training strategy that uses a multi-task learning framework (Multi-Task Learning) to simultaneously optimize the adaptability of the model to different groups through auxiliary tasks (such as group feature difference detection) while training the main task (medical diagnosis), to avoid bias caused by ignoring group differences. The bias detector can include a feature extraction network, a sensitive attribute classifier, and a fairness constraint network. The feature extraction network can be a model that uses a multi-layer Transformer or a convolutional neural network to extract features from the output feature vectors of the multi-modal alignment collaborative network.The sensitive attribute classifier can be a classifier for predicting sensitive attributes (e.g., gender attributes, race attributes) through a fully connected network. The fairness constraint network can be a network for calculating inter-group feature distribution differences. The inter-group feature distribution difference can be the maximum mean difference between different groups, or the difference in measuring the distance between the probability distributions of different groups. The bias detector can be a detector trained by including an adversarial loss and a distribution alignment loss. The adversarial loss can be a cross-entropy loss of the sensitive attribute classifier calculated through a gradient reversal layer (GRL). The distribution alignment loss can be a loss that calculates the maximum mean difference of the feature distribution of different groups. The gradient reversal layer can reverse the gradient of the sensitive attribute classifier when backpropagating, forcing the feature extractor to generate features that are independent of the sensitive attributes. The bias detector can be a detector that balances the adversarial loss and the distribution alignment loss using dynamic weight adjustment (e.g., uncertainty weighting). The bias detector can ensure that the model eliminates sensitive attribute-related bias features while maintaining main task performance through the dual mechanisms of adversarial training and distribution alignment. The output of the multi-modal extraction network can be the input of the multi-modal alignment coordination network, and the output of the multi-modal alignment coordination network is the input of the meta-learning monitoring network, the bias detector, and the multi-task learning network, which are executed in parallel. The output of the bias detector is the input of the fairness adversarial network.

[0075] It should be noted that the multi-modal extraction network can dynamically adjust the weights of different modal data to reduce bias caused by single modal data; the multi-modal alignment coordination network can automatically optimize the alignment relationship between different models to improve multi-modal fusion effect and reduce bias caused by improper modal fusion; the meta-learning monitoring network can automatically detect and correct bias caused by improper selection of training parameters to improve the adaptability and fairness of the medical large model; the bias detector can be a mechanism for joint training of the fairness constraint and adversarial network, which can improve the more balanced distribution of different group data in the medical large model and reduce model bias; the fairness adversarial network can detect the difference in prediction distribution between different groups by incorporating a fairness correction term in the loss function, so that the medical large model pays more attention to the fairness of different group data and less to prediction bias; the multi-task learning network can diagnose diseases while detecting the differences in different disease features, which can avoid bias caused by ignoring group differences.

[0076] In step 1042, the multi-modal medical diagnosis information set and the multi-modal medical sample diagnosis information set are input into a multi-modal adversarial fairness debiasing loss function corresponding to the medical large model to obtain a model loss value.

[0077] In some embodiments, the execution subject can input the multi-modal medical diagnosis information set and the multi-modal medical sample diagnosis information set into a multi-modal adversarial fairness debiasing loss function corresponding to the medical large model to obtain a model loss value, wherein the multi-modal medical sample diagnosis information set can be a sample medical diagnosis information corresponding to the multi-modal balanced medical training data set. The model loss value can be a numerical value for measuring whether the medical large model is trained. The multi-modal adversarial fairness debiasing loss function can be a loss function including a classification cross-entropy loss term of output medical diagnosis information, an adversarial loss term, a multi-modal distribution alignment loss term, and a debiasing fairness correction loss term. The adversarial loss term can be a loss function corresponding to the fair adversarial network. The multi-modal distribution alignment loss term can be a loss function corresponding to the multi-modal alignment collaborative network. The debiasing fairness correction loss term can be a loss function corresponding to the bias detector.

[0078] In the process of adopting the technical solutions to solve the technical problems mentioned in the background, the following technical problems often occur: how to accurately determine the accuracy of the medical large model output diagnosis information, and the degree of debiasing of the medical large model to the input multi-modal balanced medical training data set. Due to the weak debiasing ability of the medical large model, there are a large amount of redundant data in the debiased multi-modal medical data set generated, resulting in low performance of the drug recommendation model, low drug recommendation accuracy, and frequent transportation of the drug delivery robot, prolonging the medication time. In view of the above technical problems, the conventional solution is generally: through the cross-entropy loss function value of the predicted output diagnosis information and the real output diagnosis information obtained by the medical large model, the medical large model is iteratively trained to obtain a trained medical large model. However, the above conventional solution still has the following problems: only through the cross-entropy loss function value of the overall predicted output and the real output, it is difficult to accurately output the prediction of the medical large model containing multiple sub-network modules, resulting in low accuracy of debiasing based on the medical large model, a large amount of redundant data in the debiased multi-modal medical data set generated, low performance of the drug recommendation model, low drug recommendation accuracy, frequent transportation of the drug delivery robot, and prolonging the medication time. Considering the shortcomings of the above conventional solution, and combining the advantages / technical status of the large model training technology owned by the scientific research institute partners in the field, we decide to adopt the following solution:

[0079] In some optional implementations of some embodiments, the multi-modal adversarial fair debiasing loss function described above can include a group reconstruction regularization constraint loss function, a medical diagnosis classification cross-entropy loss function, an adversarial loss function, and a statistical parity-based fairness correction loss function. The group reconstruction regularization constraint loss function described above can be a loss function that promotes the medical large model to sufficiently learn the feature representation of the minority group through a regularization method. The medical diagnosis classification cross-entropy loss function described above can be a loss function that quantifies the difference between the predicted probability distribution of the medical large model and the true distribution. The adversarial loss function can be a loss function for training the fair adversarial network to predict the sensitivity attribute debiasing capability. The statistical parity-based fairness correction loss function can be a loss function that represents the degree of difference between the groups corresponding to the feature information set in the predicted multi-modal balanced medical training data set.

[0080] Optionally, the step of inputting the multi-modal medical diagnosis information set and the multi-modal medical sample diagnosis information set into the multi-modal adversarial fair debiasing loss function corresponding to the medical large model to obtain a model loss value, and controlling the target server to control the medicine delivery robot to transport the medicine according to the model loss value can include the following steps:

[0081] First, the multi-modal balanced medical training data set is divided into sensitive attribute groups to obtain a target medical group data set. The target medical group data in the target medical group data set can be a group composed of at least one multi-modal balanced medical training data whose sample quantity is less than a preset sample quantity threshold. The preset sample quantity threshold can be a preset minimum value of the sample quantity. In practice, the execution subject determines the medical sensitive attribute set in the multi-modal balanced medical training data set by a statistical hypothesis testing method. The medical sensitive attribute in the medical sensitive attribute set can be attribute information whose influence on the accuracy of the medical diagnosis output is greater than or equal to a preset influence degree threshold. The preset influence degree threshold can be a preset minimum value of the influence degree. The statistical hypothesis testing method can include but is not limited to at least one of the following: t-test method, analysis of variance method. Then, the training sample quantity of the multi-modal balanced medical training data included in the multi-modal balanced medical training data corresponding to each medical sensitive attribute in the medical sensitive attribute set is determined to obtain a training sample data group set. Finally, at least one medical sensitive attribute corresponding to at least one training sample data whose training sample quantity is less than or equal to the preset sample quantity threshold is selected from the training sample data group set as the target medical group data set.

[0082] Secondly, feature extraction and reconstruction are performed on the target medical population dataset to obtain a target population feature vector set and a reconstructed target population feature vector set. The target population feature vector in the target population feature vector set can represent semantic information and visual information of the target medical population data. The feature extraction can be performed by using a multi-modal extraction network and a multi-modal alignment collaborative network. The reconstructed target population feature vector set can be obtained by reconstructing the target population feature vector set. The feature reconstruction can be performed by using a reconstruction network. The reconstruction network can be an autoencoder.

[0083] Thirdly, a population reconstruction regularization constraint loss function is used to determine a population reconstruction regularization constraint function value based on the target population feature vector set and the reconstructed target population feature vector set. The population reconstruction regularization constraint function value can represent the reconstruction ability of the target population feature vector set, i.e., the degree of accurately restoring the features of the target medical population dataset. The population reconstruction regularization constraint loss function can be represented as:

[0084] .

[0085] wherein, represents the population reconstruction regularization constraint loss function. represents an index set of the target medical population dataset. represents the i-th target population feature vector in the target population feature vector set. represents the i-th reconstructed target population feature vector in the reconstructed target population feature vector set.

[0086] Fourthly, a medical diagnosis classification cross-entropy loss function is used to determine a medical diagnosis classification cross-entropy loss function value based on the multi-modal medical diagnosis information set and the multi-modal medical sample diagnosis information set. The medical diagnosis classification cross-entropy loss function value can represent the difference between the predicted probability distribution of the medical large model and the true distribution. The medical diagnosis classification cross-entropy loss function can be represented as:

[0087] .

[0088] wherein, represents the medical diagnosis classification cross-entropy loss function. represents the number of multi-modal medical diagnosis information included in the multi-modal medical diagnosis information set. represents the number of classification categories of the medical diagnosis information. represents the i-th medical diagnosis information of the classification category c in the multi-modal medical diagnosis information set. denotes the sample diagnosis information of the i-th classification category c in the multi-modal medical sample diagnosis information set.

[0089] In the fifth step, an adversarial loss function value of the fairness adversarial network included in the medical large model is determined by using an adversarial loss function. The adversarial loss function value can represent the ability of the trained fairness adversarial network to predict the sensitive attribute without bias. The adversarial loss function can be represented as:

[0090] .

[0091] wherein, denotes the adversarial loss function. denotes the prediction probability of the fairness adversarial network. denotes the sensitive attribute in the i-th target medical population data.

[0092] In the sixth step, a fairness correction loss function value of the bias detector included in the medical large model is determined by using a fairness correction loss function based on statistical parity. The fairness correction loss function value can represent the degree of predicting the difference between groups in the target medical population data set. The fairness correction loss function can be represented as:

[0093] .

[0094] wherein, denotes the fairness correction loss function. denotes any target medical population data in the target medical population data set. denotes the target medical population data set. denotes the target medical population data In the bias detector, the average probability of being predicted as a positive class . denotes the average probability of the target medical population data set being predicted as a positive class in the bias detector.

[0095] In the seventh step, the population reconstruction regularization constraint function value, the medical diagnosis classification cross-entropy loss function value, the adversarial loss function value, and the fairness correction loss function value are used to perform debiasing on the multi-modal balanced medical test dataset to obtain a debiased multi-modal medical dataset. In practice, the execution subject can first determine a loss weight value set of the population reconstruction regularization constraint function value, the medical diagnosis classification cross-entropy loss function value, the adversarial loss function value, and the fairness correction loss function value. The loss weight value set can represent the importance of each loss function value. The determination can be dynamically adjusted by the meta-learning monitoring network in the medical large model. Then, the population reconstruction regularization constraint function value, the medical diagnosis classification cross-entropy loss function value, the adversarial loss function value, the fairness correction loss function value, and the loss weight value set are weighted and summed to obtain a model loss value. After that, in response to determining that the model loss value does not satisfy a preset loss condition, the multi-modal balanced medical test dataset is reacquired to perform model iterative training on the medical large model until the model loss value of the medical large model satisfies the preset loss condition, and a trained medical large model is obtained. Finally, the multi-modal balanced medical test dataset is input into the trained medical large model to perform model debiasing to obtain a debiased multi-modal medical dataset.

[0096] In the eighth step, the target server is controlled to perform model training on a drug recommendation model based on the debiased multi-modal medical dataset to obtain recommended drug information, and a drug delivery robot is controlled to transport a recommended drug set corresponding to the recommended drug information to a location of the target patient. The implementation of this step can refer to the implementation of step 105.

[0097] The above technical solutions and related contents are an invention point of an embodiment of the present disclosure, which solves the technical problem mentioned in the background art that it is difficult to accurately predict the medical large model containing multiple sub-network modules only through the cross loss function value of the predicted output and the real output of the whole, resulting in low accuracy of debiased based on the medical large model, low performance of the medical large model training, a large amount of redundant data in the debiased multi-modal medical dataset, low performance of the drug recommendation model, low drug recommendation accuracy, and frequent transportation of the drug delivery robot, which prolongs the medication time. If the above factors are solved, the performance and accuracy of the drug recommendation model can be improved, the number of drug delivery of the drug delivery robot can be reduced, and the medication time can be shortened. In order to achieve this effect, first, the multi-modal balanced medical training dataset is divided into sensitive attribute groups to obtain a minority medical group dataset and perform feature extraction and reconstruction, which can improve the understanding of the characteristics of the minority group, improve the learning of the medical large model on the characteristics of the minority group, and through the group reconstruction regularization constraint loss function, the medical large model is forced to learn the feature representation of the minority group, so that the model can fully consider the characteristics of the minority group when predicting and reduce the bias to the minority group. Second, the medical diagnosis classification cross-entropy loss function value is determined by the sample output and the output of the medical large model, which can accurately grasp the whole medical large model to improve the diagnosis accuracy of the medical large model. Then, the adversarial loss function value is determined, which can learn the feature representation irrelevant to the sensitive attribute through the fair adversarial network to reduce the bias from the feature level. After that, the fairness correction loss function value is determined, which can monitor the difference in prediction distribution between different groups in real time, so that the medical large model pays more attention to the fairness performance of different group data in the training process and reduces the prediction bias. Finally, in response to determining that the above model loss value does not meet the preset loss condition, the medical large model is iteratively trained, which can improve the debiased accuracy of the medical large model, improve the model training speed, and reduce the model iteration time. Finally, according to the model loss value, the recommended drug set corresponding to the recommended drug information is transported to the target patient by controlling the drug delivery robot, which can improve the accuracy of the recommended drug information, reduce the transportation frequency of the drug delivery robot, shorten the transportation time, and improve the user experience.

[0098] In step 1043, in response to determining that the model loss value meets the preset loss condition, the multi-modal balanced medical test dataset is subjected to model debiased processing to obtain a debiased multi-modal medical dataset.

[0099] In some embodiments, the execution entity may, in response to determining that the model loss value meets a preset loss condition, perform model debiasing on the multimodal balanced medical test dataset to obtain a debiased multimodal medical dataset. The preset loss condition may be that the model loss value is less than or equal to a preset loss threshold. The preset loss threshold may be a pre-set minimum value. Figure 3 The diagram illustrates a comparison of model performance (accuracy) and fairness between the present disclosure and baseline debiasing schemes based on large language models. The horizontal axis represents the comparison between different baseline debiasing schemes and the debiasing technique of the present disclosure. Unbiased baseline schemes may be schemes that directly debias the model without processing the multimodal medical dataset. Simple preprocessing schemes may be schemes that debias the model only after data resampling or reweighting. Simple reprocessing schemes may be schemes that debias the model by adding a simple fairness penalty term to the loss function. The vertical axis represents the score of the evaluation metric; a higher score is better. The diagnostic accuracy mentioned above measures the accuracy of the model output. The fairness score measures the fairness of the model to different groups; a higher score indicates less inter-group variability. Unbiased baseline schemes achieve good accuracy but poor fairness. Simple preprocessing schemes improve fairness but usually at the expense of overall accuracy. Simple reprocessing schemes achieve a certain balance between accuracy and fairness, but the effect is limited. This public dissimilar dissimilarity study balances fairness and accuracy while simultaneously improving both.

[0100] Step 105: Based on the debiased multimodal medical dataset, train the drug recommendation model to obtain recommended drug information, and control the drug delivery robot to transport the recommended drugs corresponding to the recommended drug information to the location of the target patient.

[0101] In some embodiments, the execution subject can perform model training on the drug recommendation model according to the above debiased multimodal medical dataset, obtain recommended drug information, and control a drug delivery robot to deliver the recommended drug corresponding to the recommended drug information to the target patient's location. The recommended drug information can be information about a drug that can be suitable for the target patient's symptoms, which is output by the drug recommendation model. The drug delivery robot can be a robot for automated drug delivery. The target patient can be a patient who does not have the ability to act. The drug recommendation model can be a neural network model that performs drug recommendation prediction on the input debiased multimodal medical dataset to output recommended drug information. The drug recommendation model can be a TCMLLM-PR (Traditional Chinese Medicine Language Model-Prescription Recommendation) model.

[0102] Optionally, the execution subject can further perform the following steps after 105:

[0103] First, in response to determining that the model loss value does not satisfy the preset loss condition, re-collecting multimodal medical datasets of different sources as resampled multimodal medical datasets.

[0104] Second, generating resampled balanced medical datasets according to the resampled multimodal medical datasets. The specific implementation can refer to the implementation steps of steps 102 to 104.

[0105] Third, performing model decision path detection processing on the medical large model according to the resampled balanced medical datasets to obtain model decision path detection information. The model decision path detection information can be information about the output decision process of the medical large model on a specific sample. For example, the model decision path detection information can be information that the medical large model pays excessive attention to the patient's ethnic characteristics when predicting the drug treatment response in the output medical diagnosis information, and pays insufficient attention to more critical medical features such as the severity of the disease. As an example, the execution subject can use an attention mechanism visualization tool to perform model decision path detection processing on the medical large model according to the resampled balanced medical datasets to obtain model decision path detection information. The attention mechanism visualization tool can be a tool for analyzing model decision paths to effectively identify introduced bias. The attention mechanism visualization tool can include, but is not limited to, at least one of the following: LIME (Local Interpretable Model-Explain), SHAP (Shapley Additive Explanations).

[0106] In the fourth step, the medical large model is subjected to model debiasing training according to the model decision path detection information, to obtain a first medical debiased large model. As an example, the execution subject can first determine the model weight matrix of the medical large model through the model decision path detection information. Then, the model weight matrix is subjected to L1 norm regularization to suppress the feature weights of the model weight matrix, and the weight matrix of the medical large model corresponding to the removed model weight matrix is subjected to weight enhancement to obtain an adjusted model weight matrix as the weight matrix of the medical large model, so as to perform model debiasing training on the medical large model to obtain the first medical debiased large model. This step can make the medical large model pay more attention to medical related features that really affect medical diagnosis, and reduce the bias problem caused by improper feature selection.

[0107] In the fifth step, the first medical debiased large model is subjected to model fine-tuning according to the obtained medical data feedback information, to obtain a medical fine-tuned large model. The medical data feedback information can be labeled feedback data indicating that the medical large model has insufficient diagnosis accuracy for a small number of resampled multi-modal medical data included in the obtained resampled multi-modal medical data set. For example, the medical data feedback information can be "diagnosis inaccuracy of the medical large model caused by inaccurate recognition of gene features unique to rare diseases". This step can better learn the features of the minority group, improve the fairness of the medical large model to specific language medical scenarios, and reduce the over-reliance of the medical large model on the majority group data. As an example, the execution subject can first extract keywords from the medical data feedback information to obtain a feedback keyword set. Through adapter fine-tuning technology, only the parameters of the neural network layer responsible for extracting the feedback keyword set in the first medical debiased large model are fine-tuned, to obtain the medical fine-tuned large model.

[0108] In the sixth step, the medical fine-tuning large model is trained to remove bias according to the obtained dialogue feedback annotation information, to obtain a second medical debiased large model as a medical large model, and the resampled balanced medical dataset is determined as a multi-modal balanced medical training dataset to perform the model debiasing step again. The dialogue feedback annotation information can be feedback information obtained by a clinical expert annotating the dialogue between the user and the model. For example, the dialogue feedback annotation information can be {question: "I have been feeling tired recently, am I sick?", answer: "Fatigue can be caused by many reasons, further check thyroid function and other indicators are recommended"}. This step can optimize the fairness of the medical large model in diagnosing complex cases to better adapt to the diversity and complexity in real medical scenarios. As an example, the execution subject can first input the dialogue information and the dialogue feedback annotation information into the loss function corresponding to the medical fine-tuning large model to obtain a dialogue feedback loss value. The dialogue information can be the dialogue information between the medical large model and the user without standard feedback by the clinical expert. Then, using a gradient descent optimization algorithm, the medical fine-tuning large model is trained to remove bias according to the dialogue feedback loss value to obtain a second medical debiased large model as a medical large model, and the resampled balanced medical dataset is determined as a multi-modal balanced medical training dataset to perform the model debiasing step again.

[0109] Further reference Figure 4 To implement the methods shown in the above figures, the present disclosure provides some embodiments of a medical large model-based model debiasing device, which corresponds to the method embodiments shown in Figure 1 The medical large model-based model debiasing device can be applied in various electronic devices.

[0110] As Figure 4As shown, the model debiasing apparatus 400 based on a medical large model comprises a multi-level debiasing correction unit 401, a data balanced grouping unit 402, a resource detection unit 403, a first control unit 404 and a second control unit 405. The multi-level debiasing correction unit 401 performs multi-level debiasing correction on the obtained multi-modal medical data set to obtain a multi-modal debiased medical data set, wherein the multi-modal medical data set is obtained from a multi-modal medical corpus. The data balanced grouping unit 402 performs data balanced grouping processing on the multi-modal debiased medical data set to obtain a multi-modal balanced medical training data set and a multi-modal balanced medical test data set. The resource detection unit 403 detects resources of a server cluster, and selects a server satisfying a resource selection condition from the server cluster as a target server. The first control unit 404 controls the target server to perform the following model debiasing steps based on the multi-modal balanced medical training data set: inputting the multi-modal balanced medical training data set into a medical large model to obtain a multi-modal medical diagnosis information set; inputting the multi-modal medical diagnosis information set and a multi-modal medical sample diagnosis information set into a multi-modal adversarial fair debiasing loss function corresponding to the medical large model to obtain a model loss value; in response to determining that the model loss value satisfies a preset loss condition, performing model debiasing processing on the multi-modal balanced medical test data set to obtain a debiased multi-modal medical data set. The second control unit 405 performs model training on a drug recommendation model according to the debiased multi-modal medical data set to obtain recommended drug information, and controls a drug delivery robot to deliver a recommended drug corresponding to the recommended drug information to a target patient's location.

[0111] It can be understood that the units described in the model debiasing apparatus 400 based on a medical large model correspond to the respective steps in the described method. Figure 1 Therefore, the operations, features and beneficial effects described above for the method also apply to the model debiasing apparatus 400 based on a medical large model and the units contained therein, which will not be described here again.

[0112] Reference is made below to Figure 5 which shows a structural schematic diagram of an electronic device (e.g., an electronic device) 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the function and use range of the embodiments of the present disclosure.

[0113] As Figure 5As shown, the electronic device 500 can include a processing device (e.g., a central processor, a graphics processor, etc.) 501 that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 502 or loaded into a random access memory (RAM) 503 from a storage device 508. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0114] Generally, the following devices can be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 can allow the electronic device 500 to exchange data with other devices wirelessly or wiredly. Although Figure 5 The electronic device 500 is shown with various devices, but it is understood that all of the shown devices are not required to be implemented or present. More or fewer devices can alternatively be implemented or present. Figure 5 Each block shown in the middle can represent a device or multiple devices as needed.

[0115] In particular, processes described above with reference to the flowcharts can be implemented as a computer software program according to some embodiments of the present disclosure. For example, some embodiments of the present disclosure include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for performing the methods shown in the flowcharts. In some such embodiments, the computer program can be downloaded and installed from a network through the communication devices 509, or installed from the storage devices 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-described functions defined in the methods of some embodiments of the present disclosure are performed.

[0116] Note that the computer readable medium in the above embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In some embodiments of the present disclosure, the computer readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such propagated data signals can take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that is not a storage medium and that can communicate, propagate or transport program for use by or in connection with an instruction execution system, apparatus or device. The program code contained in the computer readable medium can be transmitted by any suitable medium, including, but not limited to, wire, cable, RF (radio frequency), or the like, or any suitable combination of the foregoing.

[0117] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0118] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and not be assembled into the electronic device. The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: acquire a multi-modal medical data set from a multi-modal medical corpus, wherein the multi-modal medical data set includes the content of steps 101-105.

[0119] Computer program code for carrying out operations of some embodiments of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0120] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in some cases, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by a dedicated hardware-based system that carries out specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0121] The units described in some embodiments of the present disclosure can be implemented by means of software and / or hardware. The described units can also be provided in a processor, for example, it can be described that a processor comprises a multi-level debiasing correction unit, a data equalization grouping unit, a resource detection unit, a first control unit and a second control unit. Among them, the name of these units does not constitute a limitation to the unit itself in some cases, for example, the multi-level debiasing correction unit can also be described as "a unit for performing multi-level debiasing correction on the obtained multi-modal medical data set to obtain a multi-modal debiased medical data set".

[0122] The functionality described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program- specific Integrated Circuits (ASICs), Program- specific Standard Products (ASSPs), System-on-a-chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0123] The above description is merely exemplary of the disclosure and the application made use of the principles of the technology. It is to be understood that the application is not limited in scope to the described technical features, and that the application can be practiced with modification other than those described in the embodiment discussed in the context of the above description of the technology. For example, it will be appreciated that features described above and in the following description are not mutually exclusive and can be combined in a manner dependent upon the specific use or implementation. The scope of the application is defined by the appended claims rather than the preceding description, and all modifications that fall within the range of equivalents used in the art are intended to be embraced therein.

Claims

1. A model debiasing method based on a medical large model, comprising: performing multi-level debiasing correction on an obtained multi-modal medical data set to obtain a multi-modal debiased medical data set, wherein the multi-modal medical data set is obtained from a multi-modal medical corpus; performing data balanced grouping processing on the multi-modal debiased medical data set to obtain a multi-modal balanced medical training data set and a multi-modal balanced medical test data set; detecting resources of a server cluster, and selecting a server satisfying a resource selection condition from the server cluster as a target server; controlling the target server to perform the following model debiasing steps based on the multi-modal balanced medical training data set: inputting the multi-modal balanced medical training data set into a medical large model to obtain a multi-modal medical diagnosis information set; inputting the multi-modal medical diagnosis information set and the multi-modal medical sample diagnosis information set into a multi-modal adversarial fairness debiasing loss function corresponding to the medical large model to obtain a model loss value, comprising: performing sensitive attribute group division on the multi-modal balanced medical training data set to obtain a target medical group data set; performing feature extraction and reconstruction on the target medical group data set to obtain a target group feature vector set and a reconstructed target group feature vector set; determining a group reconstruction regularization constraint function value according to the target group feature vector set and the reconstructed target group feature vector set by using a group reconstruction regularization constraint loss function, wherein the group reconstruction regularization constraint loss function is represented as: wherein, the group reconstruction regularization constraint loss function is represented as, an index set of the target medical group data set is represented as, an i-th target group feature vector in the target group feature vector set is represented as, an i-th reconstructed target group feature vector in the reconstructed target group feature vector set is represented as; determining a medical diagnosis classification cross-entropy loss function value according to the multi-modal medical diagnosis information set and the multi-modal medical sample diagnosis information set by using a medical diagnosis classification cross-entropy loss function, wherein the medical diagnosis classification cross-entropy loss function is represented as: wherein, the medical diagnosis classification cross-entropy loss function is represented as, a quantity of multi-modal medical diagnosis information included in the multi-modal medical diagnosis information set is represented as, a quantity of classification categories of the multi-modal medical diagnosis information set is represented as, medical diagnosis information of an i-th classification category c in the multi-modal medical diagnosis information set is represented as, sample diagnosis information of an i-th classification category c in the multi-modal medical sample diagnosis information set is represented as; determining an adversarial loss function value of a fairness adversarial network included in the medical large model by using an adversarial loss function, wherein the adversarial loss function is represented as: wherein, the adversarial loss function is represented as, a predicted probability of the fairness adversarial network is represented as, a sensitive attribute in an i-th target medical group data is represented as; determining a fairness correction loss function value of a bias detector included in the medical large model by using a fairness correction loss function based on statistical parity, wherein the fairness correction loss function is represented as: wherein, the fairness correction loss function is represented as, any target medical group data in the target medical group data set is represented as, the target medical group data set is represented as, denotes an average probability of being predicted as positive class in the bias detector, an average probability of being predicted as positive class in the bias detector for a target medical population dataset; and reconstructing a fairness correction loss function value based on the average probability of being predicted as positive class in the bias detector for the target medical population dataset; and performing debiasing processing on the multi-modal balanced medical test dataset according to the fairness correction loss function value, the medical diagnosis classification cross-entropy loss function value, the adversarial loss function value, and the regularization constraint function value to generate a debiased multi-modal medical dataset. in response to determining that a model loss value satisfies a preset loss condition, performing model debiasing processing on the multi-modal balanced medical test data set to obtain a debiased multi-modal medical data set; training a drug recommendation model based on the debiased multi-modal medical data set to obtain recommended drug information, and controlling a drug delivery robot to deliver recommended drugs corresponding to the recommended drug information to a target patient location.

2. The method of claim 1, wherein, The method further comprises: in response to determining that the model loss value does not satisfy the preset loss condition, re-collecting multi-modal medical data sets of different sources as resampled multi-modal medical data sets; generating a resampled balanced medical data set based on the resampled multi-modal medical data set; performing model decision path detection processing on the medical large model based on the resampled balanced medical data set to obtain model decision path detection information; training the medical large model based on the model decision path detection information to obtain a first medical debiased large model; performing model fine-tuning on the first medical debiased large model based on obtained medical data feedback information to obtain a medical fine-tuned large model; performing model debiasing training on the medical fine-tuned large model based on obtained dialogue feedback labeling information to obtain a second medical debiased large model as a medical large model, and determining the resampled balanced medical data set as a multi-modal balanced medical training data set to execute the model debiasing steps again.

3. The method of claim 1, wherein, The multi-modal medical corpus is obtained by the following steps: collecting multi-modal regional medical data sets of different regions and different languages; determining a medical credibility set of the multi-modal regional medical data sets; performing deduplication processing on the multi-modal regional medical data sets based on a sparse attention mechanism based on the medical credibility set to obtain deduplicated medical data sets; performing multi-modal feature alignment on the deduplicated medical data sets and the multi-modal regional medical data sets after removing medical text data sets to obtain a multi-modal associated medical diagnosis information set; performing generative data augmentation on the multi-modal associated medical diagnosis information set to obtain a multi-modal augmented medical data set; performing privacy classification isolation storage on the multi-modal augmented medical data set to obtain a multi-modal medical corpus.

4. The method of claim 3, wherein, The privacy classification isolation storage of the multi-modal augmented medical data set obtains a multi-modal medical corpus, which comprises: The keyword matching is performed on the multi-modal augmented medical data set and a preset keyword set to obtain a keyword matching result set; At least one multi-modal augmented medical data corresponding to the keyword matching result set is screened out from the multi-modal augmented medical data set, and the privacy of the at least one multi-modal augmented medical data is processed and stored in a database of a medical record category included in the multi-modal medical corpus; The distance similarity matching is performed on a target multi-modal augmented medical data set and a preset dynamic medical record template set to obtain an entity template matching probability set, wherein the target multi-modal augmented medical data set is a data set obtained by removing the at least one multi-modal augmented medical data from the multi-modal augmented medical data set; At least one target multi-modal augmented medical data corresponding to an entity template matching probability greater than or equal to a preset matching probability threshold in the target multi-modal augmented medical data set is processed and stored in an electronic medical record database included in the multi-modal medical corpus after the privacy of the at least one target multi-modal augmented medical data is processed. The multi-layer bias correction is performed on the multi-modal medical data set to obtain a multi-modal bias-free medical data set, which comprises:

5. The method of claim 1, wherein, The feature information extraction is performed on the multi-modal medical data set to obtain a medical diagnosis feature information set; For each medical diagnosis feature information in the medical diagnosis feature information set, the following causal inference bias step is performed: The global perturbation processing is performed on the medical diagnosis feature information to obtain a feature sensitivity value; According to the medical diagnosis feature information set, a propensity causal confounding probability of the medical diagnosis feature information under different attribute values is determined; According to the obtained propensity causal confounding probability set and the obtained feature sensitivity value set, the regression bias is performed on the multi-modal medical data set to obtain a sensitive bias-free multi-modal medical data set; The feature local explanation bias processing is performed on the sensitive bias-free multi-modal medical data set to obtain an explanation bias-free multi-modal medical data set; The causal inference bias processing is performed on the explanation bias-free multi-modal medical data set to obtain a causal bias-free multi-modal medical data set as a multi-modal bias-free medical data set.

6. A model bias-free device based on a medical large model, comprising: a multi-layer bias correction unit configured to perform multi-layer bias correction on a multi-modal medical data set obtained to obtain a multi-modal bias-free medical data set, wherein the multi-modal medical data set is obtained from a multi-modal medical corpus; a data balanced grouping unit configured to perform data balanced grouping processing on the multi-modal bias-free medical data set to obtain a multi-modal balanced medical training data set and a multi-modal balanced medical test data set; a resource detection unit configured to perform resource detection on a server cluster and to screen a server satisfying a resource selection condition from the server cluster as a target server; and a model training unit configured to train a target model based on the multi-modal balanced medical training data set and the multi-modal balanced medical test data set. The first control unit controls the target server to perform the following model debiasing steps based on the multi-modal balanced medical training dataset: inputting the multi-modal balanced medical training dataset into the medical large model to obtain a multi-modal medical diagnosis information set; inputting the multi-modal medical diagnosis information set and a multi-modal medical sample diagnosis information set into a multi-modal adversarial fairness debiasing loss function corresponding to the medical large model to obtain a model loss value, including: performing sensitive attribute group division on the multi-modal balanced medical training dataset to obtain a target medical group dataset; performing feature extraction and reconstruction on the target medical group dataset to obtain a target group feature vector set and a reconstructed target group feature vector set; determining a group reconstruction regularization constraint function value according to the target group feature vector set and the reconstructed target group feature vector set by using a group reconstruction regularization constraint loss function, wherein the group reconstruction regularization constraint loss function is represented as: wherein, the group reconstruction regularization constraint loss function is represented as, an index set of the target medical group dataset is represented as, the i th target group feature vector in the target group feature vector set is represented as, the i th reconstructed target group feature vector in the reconstructed target group feature vector set is represented as; determining a medical diagnosis classification cross-entropy loss function value according to the multi-modal medical diagnosis information set and the multi-modal medical sample diagnosis information set by using a medical diagnosis classification cross-entropy loss function, wherein the medical diagnosis classification cross-entropy loss function is represented as: wherein, the medical diagnosis classification cross-entropy loss function is represented as, the number of multi-modal medical diagnosis information included in the multi-modal medical diagnosis information set is represented as, the number of classification categories of the multi-modal medical diagnosis information set is represented as, the medical diagnosis information of the i th classification category c in the multi-modal medical diagnosis information set is represented as, the sample diagnosis information of the i th classification category c in the multi-modal medical sample diagnosis information set is represented as; determining an adversarial loss function value of a fairness adversarial network included in the medical large model by using an adversarial loss function, wherein the adversarial loss function is represented as: wherein, the adversarial loss function is represented as, the prediction probability of the fairness adversarial network is represented as, the sensitive attribute in the i th target medical group data is represented as; determining a fairness correction loss function value of a bias detector included in the medical large model by using a fairness correction loss function based on statistical parity, wherein the fairness correction loss function is represented as: wherein, This represents the fairness correction loss function. This represents any data point of the target medical group in the target medical group dataset. This represents a dataset representing the target medical population. express Predicted as positive in the bias detector The average probability, This indicates that the target medical group dataset was predicted as positive by the bias detector. The average probability; based on the group reconstruction regularization constraint function value, the medical diagnosis classification cross-entropy loss function value, the adversarial loss function value, and the fairness correction loss function value, the multimodal balanced medical test dataset is debiased to generate a debiased multimodal medical dataset; in response to determining that the model loss value meets the preset loss condition, the multimodal balanced medical test dataset is debiased to obtain a debiased multimodal medical dataset; A second control unit performs model training on a drug recommendation model according to the de-biased multimodal medical data set to obtain recommended drug information, and controls a drug delivery robot to transport recommended drugs corresponding to the recommended drug information to a target patient location.

7. An electronic device, comprising: one or more processors; storage having stored thereon one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-5.

8. A computer readable medium having stored thereon a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-5. The computer program, when executed by a processor, implements the method of any one of claims 1-5.

Images