Model deprejudice method and device based on medical large model, equipment and medium

Abstract

The embodiment of the invention discloses a model deprejudice method and device based on a medical large model, equipment and a medium. A specific embodiment of the method comprises the following steps: carrying out multi-level prejudice removal correction on a multi-modal medical data set, and then carrying out data equalization grouping to obtain a multi-modal equalization medical training data set; performing resource detection on the server cluster; inputting the multi-modal equilibrium medical training data set into the medical large model to obtain a multi-modal medical diagnosis information set; inputting the multi-modal medical diagnosis information set and the multi-modal medical sample diagnosis information set into a multi-modal adversarial fair depolarization loss function to obtain a model loss value; and after model deprejudice processing is carried out on the multi-modal equilibrium medical test data set, model training is carried out on the drug recommendation model, and a drug delivery robot is controlled to carry out transportation. According to the embodiment, the resource load balance of the server cluster and the quality of the medical data after prejudice removal can be improved, the training speed of the model is improved, and the robot is accurately controlled to convey medicines.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method, apparatus, device, and medium for model debiasing based on a large medical model. Background Art

[0002] With the development of medical artificial intelligence, large multimodal medical models have been widely used in medical diagnosis, treatment recommendation, and other medical processes. However, the medical training data used for these models is subject to noise interference, privacy risks, and bias caused by language and regional differences, which seriously hinders the development of these models. Debiasing models based on large medical models is typically performed by using a large language model to perform large language model-based debiasing on a single data source, generating a debiased single data source and storing the debiased single data source.

[0003] However, in practice, it has been found that when the above method is used to debias a model based on a large medical model, the following technical problems often arise: due to the data bias problem of a single data source itself, and the large amount of multimodal medical data, it may not be possible to execute on a resource-constrained server or extend the training time of the large medical model, resulting in low accuracy of data debiasing and low quality of multimodal medical data, and the presence of a large amount of redundant and erroneous data, which in turn reduces the model performance of the drug recommendation model and generates less accurate recommended drug information, causing drug delivery robots to frequently transport drugs, prolonging the time it takes for drugs to be delivered to the location of patients who are unable to move, aggravating the severity of the patient's condition, and reducing the user experience.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that is not already known in this country to one of ordinary skill in the art. Summary of the Invention

[0005] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure propose a model debiasing method, apparatus, device, and medium based on a large medical model to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a model debiasing method based on a large medical model, comprising: performing multi-level debiasing correction on an acquired multimodal medical dataset to obtain a multimodal debiased medical dataset, wherein the multimodal medical dataset is acquired from a multimodal medical corpus; performing data balancing and grouping processing on the multimodal debiased medical dataset to obtain a multimodal balanced medical training dataset and a multimodal balanced medical test dataset; performing resource detection on a server cluster, and selecting a server that meets the resource selection conditions from the server cluster as a target server; controlling the target server to execute the following model based on the multimodal balanced medical training dataset. Model debiasing step: input the multimodal balanced medical training data set into the medical big model to obtain a multimodal medical diagnosis information set; input the multimodal medical diagnosis information set and the multimodal medical sample diagnosis information set into the multimodal adversarial fair debiasing loss function corresponding to the medical big model to obtain a model loss value; in response to determining that the model loss value meets the preset loss condition, perform model debiasing on the multimodal balanced medical test data set to obtain a debiased multimodal medical data set; based on the above-mentioned debiased multimodal medical data set, perform model training on the drug recommendation model to obtain recommended drug information, and control the drug delivery robot to transport the recommended drugs corresponding to the above-mentioned recommended drug information to the location of the target patient.

[0008] In a second aspect, some embodiments of the present disclosure provide a model debiasing device based on a large medical model, comprising: a multi-level debias correction unit, which performs multi-level debias correction on an acquired multimodal medical dataset to obtain a multimodal debiased medical dataset, wherein the multimodal medical dataset is acquired from a multimodal medical corpus; a data balancing grouping unit, which performs data balancing grouping processing on the multimodal debiased medical dataset to obtain a multimodal balanced medical training dataset and a multimodal balanced medical test dataset; a resource detection unit, which performs resource detection on a server cluster, and selects a server that meets the resource selection conditions from the server cluster as a target server; a first control unit, which controls the target server to perform a multimodal balanced grouping based on the multimodal balanced grouping. The multimodal balanced medical training dataset is used to perform the following model debiasing steps: the multimodal balanced medical training dataset is input into the medical big model to obtain a multimodal medical diagnosis information set; the multimodal medical diagnosis information set and the multimodal medical sample diagnosis information set are input into the multimodal adversarial fair debiasing loss function corresponding to the medical big model to obtain a model loss value; in response to determining that the model loss value meets the preset loss condition, the multimodal balanced medical test dataset is subjected to model debiasing processing to obtain a debiased multimodal medical dataset; the second control unit performs model training on the drug recommendation model according to the above-mentioned debiased multimodal medical dataset to obtain recommended drug information, and controls the drug delivery robot to transport the recommended drugs corresponding to the above-mentioned recommended drug information to the location of the target patient.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0011] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the model debiasing method based on the medical big model in some embodiments of the present disclosure can improve the resource load balancing of the server cluster, the quality of the medical data after debiasing, improve the training speed of the model and accurately control the robot to deliver medicines. Specifically, the reason for the low accuracy of the relevant data debiasing and the low quality of the multimodal medical data, the reduced model performance of the drug recommendation model, the prolonged drug delivery time, the aggravation of the severity of the patient, and the reduction of the user experience is that: due to the data bias problem of the single data source itself and the large amount of multimodal medical data, it may not be possible to execute or extend the training time of the medical big model on a resource-constrained server, resulting in low data debiasing accuracy and low quality of the multimodal medical data, the presence of a large amount of redundant and erroneous data, and thus reducing the model performance of the drug recommendation model, generating less accurate recommended drug information, causing the drug delivery robot to frequently transport drugs, prolonging the time it takes for the drugs to be delivered to the location of patients who are unable to move, aggravating the severity of the patient, and reducing the user experience. Based on this, some embodiments of the present disclosure disclose a model debiasing method based on a large medical model. First, a multi-level debiasing correction is performed on an acquired multimodal medical dataset to obtain a multimodal debiased medical dataset, wherein the multimodal medical dataset is obtained from a multimodal medical corpus. Here, obtaining a multimodal medical dataset from a multimodal medical corpus can improve the diversity of medical data and reduce bias caused by a single data source to a certain extent. The multi-level debiasing correction can include sensitivity debiasing, model interpretation debiasing, and causal relationship debiasing. Sensitivity debiasing can prevent bias caused by directly inputting minority group features into the large medical model. Model interpretation debiasing can generate local interpretations of the multimodal medical data to debias, thereby avoiding model bias caused by improper feature selection. Causal relationship debiasing can determine whether the causal relationship between the input and output diagnostic information included in the multimodal medical dataset is reasonable, thereby removing features that deviate from the causal relationship, improving the causal rationality of the input data, and reducing model bias caused by incorrect causality. Subsequently, the multimodal debiased medical dataset is subjected to data balancing and grouping processing to obtain a multimodal balanced medical training dataset and a multimodal balanced medical test dataset. This balances the input data for subsequent large medical models, preventing bias toward the majority group during model training. Subsequently, resource detection is performed on the server cluster, and servers that meet the resource selection criteria are selected from the cluster as target servers. Resource detection provides real-time visibility into the server cluster's operational status, facilitating the delivery of multimodally balanced medical training and testing datasets to servers with sufficient resources. This accelerates the training of subsequent large medical models and improves resource load balancing within the server cluster.Then, the target server is controlled to perform the following model debiasing steps based on the multimodal balanced medical training dataset: First, the multimodal balanced medical training dataset is input into the medical macro model to obtain a multimodal medical diagnostic information set. The learning capabilities of the medical macro model are utilized to improve the performance and accuracy of debiasing, reduce group differences and bias in the multimodal balanced medical training dataset, and improve the accuracy of the multimodal medical diagnostic information. Second, the multimodal medical diagnostic information set and the multimodal medical sample diagnostic information set are input into the multimodal adversarial fairness debiasing loss function corresponding to the medical macro model to obtain a model loss value. By incorporating an adversarial loss, a fairness correction term, and a group reconstruction regularization constraint into the loss function, the model's processing performance and training efficiency for multimodal medical data can be further improved. Third, in response to determining that the model loss value satisfies a preset loss condition, model debiasing is performed on the multimodal balanced medical test dataset to obtain a debiased multimodal medical dataset. This improves the accuracy of data debiasing and the quality of the debiased multimodal medical data. Finally, the drug recommendation model is trained based on the debiased multimodal medical dataset to obtain recommended drug information. The drug delivery robot is then controlled to deliver the recommended drug corresponding to the recommended drug information to the target patient's location. This improves the accuracy of the recommended drug information, reduces the delivery time and frequency of the drug delivery robot, shortens the time it takes for patients with disabilities to obtain medication, and enhances the user experience. Consequently, this model debiasing method based on a large medical model, which obtains a multimodal medical dataset from a multimodal medical corpus, can effectively reduce bias caused by a single data source. By performing debiasing during both data preprocessing and model training, it can accurately identify and correct biases and potential bias factors in the data, avoid the accidental deletion of key information, ensure data balance across different groups, and enable the model to more fairly learn the characteristics of different groups during training. This effectively reduces model bias caused by data bias, improves the training efficiency of the large medical model, enhances debiasing accuracy, increases the training speed of the large medical model and the drug recommendation model, increases cluster load, and reduces robot delivery time and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of a model debiasing method based on a large medical model according to the present disclosure; Figure 2is a schematic diagram of causal paths generated in some embodiments of the model debiasing method based on a large medical model according to the present disclosure; Figure 3 is a schematic diagram comparing the accuracy and fairness of the present disclosure and existing debiasing technologies in some embodiments of the model debiasing method based on a large medical model according to the present disclosure; Figure 4 is a schematic structural diagram of some embodiments of a model debiasing device based on a large medical model according to the present disclosure; Figure 5 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0015] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0017] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0018] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0019] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0020] Figure 1 The flowchart 100 of some embodiments of the method for model debiasing based on a large medical model according to the present disclosure is shown. The method for model debiasing based on a large medical model comprises the following steps: Step 101 : Perform multi-level debiasing correction on the acquired multimodal medical dataset to obtain a multimodal debiased medical dataset.

[0021] In some embodiments, an entity (e.g., an electronic device) executing the aforementioned model debiasing method based on a large medical model can perform multi-level debias correction on a multimodal medical dataset to obtain a debiased multimodal medical dataset. The multimodal medical dataset is obtained from a multimodal medical corpus. The multimodal medical corpus can be a database that automatically collects data of different modalities from multiple channels, such as HIS (Hospital Information System), PACS (Picture Archiving and Communication System), electronic medical record databases, and public medical databases, at preset times using a data acquisition engine through a secure and encrypted data interface. The data is then pre-processed and stored. The preset time can be a pre-set acquisition time. For example, the preset time can be the early morning of each day. The multimodal medical data in the multimodal medical dataset can include medical data representing a patient's physical condition in various data formats. The various data formats can include, but are not limited to, at least one of the following: text data, medical images, and time-series data of a user's physical condition acquired via sensors. The sensors can include, but are not limited to, at least one of the following: an electronic watch, a smartphone, or a blood pressure monitor. The multimodal debiased medical data in the multimodal debiased medical dataset may be medical data obtained by removing biased or unfair data from the multimodal medical dataset.

[0022] In some optional implementations of some embodiments, the multimodal medical corpus may be obtained by the following steps: The first step is to collect multimodal regional medical datasets in different regions and languages. The multimodal regional medical data in the multimodal regional medical dataset can be medical data in different languages ​​and formats to address regional differences in medical practices. For example, the multimodal regional medical data can include the use of ginger and brown sugar water as an adjunct to cold treatment in northern China.

[0023] The second step is to determine a medical credibility set for the multimodal regional medical dataset. The medical credibility in the medical credibility set can represent the credibility of the multimodal regional medical data. The higher the medical credibility value, the greater the quality and credibility of the multimodal regional medical data. In practice, the execution entity may first perform medical entity relationship extraction on the medical text dataset included in the multimodal regional medical dataset to obtain a medical entity set. The medical entities in the medical entity set may be triple entities related to human body examination data. For example, the medical entity may be {patient identity information: xxx, entity name: white blood cells, count: 1000, unit: 10^9 / L, context description: white blood cell count significantly increased}. The medical entity relationship extraction may be performed using a tandem BiLSTM (Bidirectional Long Short-Term Memory)-CRF (Conditional Random Fields) model and a pre-trained model for named entity recognition and relationship extraction. The pre-trained model may be BERT (Bidirectional Encoder Representations from Transformers). Then, through the above-mentioned medical entity set, a multi-dimensional indicator search is performed on the preset medical data feature library to obtain a medical entity multi-dimensional indicator group set. Among them, the medical entity multi-dimensional indicators in the above-mentioned medical entity multi-dimensional indicator group set can be indicator information used to measure the credibility assessment of the above-mentioned medical entity. The above-mentioned medical entity multi-dimensional indicators may include but are not limited to at least one of the following: normal value range of medical entity, abnormal value range, consistency of medical entity related indicators, and reliability of data source. The above-mentioned preset medical data feature library can be a pre-set knowledge base related to medical data. Finally, for each medical entity in the above-mentioned medical entity set, the following determination steps are performed: the first step is to determine the indicator weight value group of the medical entity multi-dimensional indicator group for the above-mentioned medical entity. The second step is to perform weighted summation on the above-mentioned indicator weight value group and the above-mentioned medical entity multi-dimensional indicator group to obtain medical credibility.

[0024] In the third step, according to the above medical credibility set, the above multimodal regional medical dataset is deduplicated based on the sparse attention mechanism to obtain the deduplicated medical dataset.

[0025] As an example, the execution entity may first filter at least one multimodal regional medical data set from the multimodal regional medical dataset whose corresponding medical credibility satisfies a preset credibility condition. The preset credibility condition may be a condition that the medical credibility is greater than or equal to a preset credibility threshold. The preset credibility threshold may be a pre-set minimum value for assessing the credibility of a medical entity. For example, the preset credibility threshold may be 0.6. Then, the at least one multimodal regional medical data set is rapidly filtered using the coarse-grained label compression technique within the sparse attention mechanism to obtain a filtered multimodal regional medical dataset. The rapid filtering may be a preliminary screening using the patient's user identification information as a coarse-grained label. The user identification information may be unique identifier representing the user's identity information. Then, in response to detecting the presence of a medical entity with the same user identification information in the filtered multimodal regional medical dataset, the filtered multimodal regional medical dataset is accurately compared using the fine-grained label selection technique within the sparse attention mechanism. The accurate comparison may be performed based on metrics such as visit time, diagnosis results, and treatment plan. It should be noted that this step ensures that duplicate data is accurately identified and removed, significantly improving deduplication efficiency while maintaining global context awareness and local accuracy. The aforementioned sparse attention mechanism can be a mechanism that uses attention to retain key information and reduce redundant computation. The aforementioned sparse attention mechanism may include, but is not limited to, at least one of the following: NSA (Nested Sparse Attention, native sparse attention mechanism), MOBA (Mixture of Block Assignment), and MLA (Multi-Head Latent Attention).

[0026] The fourth step is to perform multimodal feature alignment on the deduplicated medical dataset and the multimodal regional medical dataset after removing the medical text dataset to obtain a multimodal associated medical diagnostic information set. The multimodal associated medical diagnostic information in the multimodal associated medical diagnostic information set may be medical data that aligns information from different modalities to reduce differences in inter-modal heterogeneity and the semantic integrity of each modality. In practice, the execution entity may first perform multimodal data extraction on the deduplicated medical dataset and the multimodal regional medical dataset after removing the medical text dataset to obtain a multimodal medical diagnostic information set. The multimodal medical diagnostic information in the multimodal medical diagnostic information set may be medical feature information obtained by extracting features from medical data of different modalities. The multimodal data extraction may include: performing text recognition on image data using an optical character recognition algorithm, accurately segmenting image data using an image semantic segmentation algorithm to identify lesion area information, and performing part-of-speech tagging and named entity recognition on the deduplicated medical dataset using natural language processing technology. Then, cross-modal data alignment is performed on the multimodal medical diagnostic information set to obtain a multimodally correlated medical diagnostic information set. For example, the multimodal medical diagnostic information set may include identification information for a lung CT (Computed Tomography) image and the corresponding medical record description, "The patient has ground-glass opacities in the lungs, suspected to be early-stage lung cancer." This cross-modal data alignment can be achieved through the following steps: First, feature extraction is performed on each region in the CT image (for example, extracting features such as shape and density for the ground-glass opacity region) to obtain a regional feature information set. Second, semantic understanding and feature extraction are performed on the keywords "lung," "ground-glass opacities," and "early-stage lung cancer" in the medical record text to obtain a medical record feature information set. Third, multimodal alignment is performed on the regional feature information set and the medical record feature information set using a cross-modal attention mechanism to obtain a multimodally correlated medical diagnostic information set. It should be noted that this step accurately correlates data from different modalities, ensures data annotation consistency, and further enhances the large language model's ability to integrate and understand multimodal data.

[0027] The fifth step is to perform generative data augmentation on the multimodally correlated medical diagnostic information set to obtain a multimodally augmented medical dataset. The multimodally augmented medical data in the multimodally augmented medical dataset may be medical data obtained by augmenting the multimodally correlated medical diagnostic information set. This generative data augmentation may be performed using a generative adversarial network model in conjunction with a large language model to augment the multimodally correlated medical identification information set. For example, using data from patients with the rare disease cystic fibrosis as an example, the large language model first analyzes the feature distribution information of the existing cystic fibrosis patient data. This feature distribution information may include, but is not limited to, at least one of the following: patient genetic characteristics, clinical manifestations (e.g., frequency of symptoms such as dyspnea and recurrent lung infections), and treatment regimens (e.g., information and dosage of specific medications used). Then, using a generative adversarial network, new data samples are generated based on this feature distribution information to obtain a multimodally correlated augmented medical information set. This new data sample may include new gene sequence combinations, combinations of varying degrees of symptom manifestations, and corresponding treatment regimen adjustments. It should be noted that this step can balance the sample size of data from different populations through data augmentation, significantly improving sample representativeness.

[0028] The sixth step is to perform privacy-classified, isolated storage on the multimodal augmented medical dataset to obtain a multimodal medical corpus. This privacy-classified, isolated storage can be achieved through data segmentation, encrypted transmission, and isolated storage. This step can prevent data leakage and cross-contamination, ensuring the fairness and reliability of model training.

[0029] Optionally, performing privacy classification and isolated storage on the multimodal augmented medical dataset to obtain a multimodal medical corpus may include the following steps: In the first step, keyword matching is performed on the multimodal augmented medical dataset and a preset keyword set to obtain a keyword matching result set. The keyword matching results in the keyword matching result set may indicate whether the match was successful. The preset keywords in the preset keyword set may be pre-defined segmentation words. For example, the preset keywords may include, but are not limited to, at least one of the following: diagnosis, prescription, and chief complaint.

[0030] In the second step, at least one multimodal augmented medical data set corresponding to a keyword matching result indicating a successful match is selected from the multimodal augmented medical data set.

[0031] The third step is to perform privacy-enhancing processing on the at least one piece of multimodal augmented medical data and store it in a database of medical records included in the multimodal medical corpus. This privacy-enhancing processing can involve encrypting and anonymizing sensitive data in the multimodal augmented medical data using differential privacy techniques, or encrypting and anonymizing sensitive numerical data using a homomorphic encryption algorithm.

[0032] The fourth step is to perform distance similarity matching on the target multimodal amplified medical data set and the preset dynamic medical record template set to obtain an entity template matching probability set, wherein the above-mentioned target multimodal amplified medical data set is a data set obtained by removing the above-mentioned at least one multimodal amplified medical data from the above-mentioned multimodal amplified medical data set. The entity template matching probability in the above-mentioned entity template matching probability set can characterize the degree of matching between the target multimodal amplified medical data and the preset dynamic medical record template. The above-mentioned distance similarity matching can be a match performed after weighted summation of the cosine similarity based on the Euclidean distance and the semantic similarity extracted by the BERT model. The preset dynamic medical record template in the preset dynamic medical record template set can be a pre-set template about the medical record format. The preset dynamic medical record template set can be updated every month by a clustering algorithm to adapt to the template set of the newly added multimodal amplified medical information set.

[0033] In a fifth step, at least one target multimodal augmented medical data item in the target multimodal augmented medical data set whose entity template matching probability is greater than or equal to a preset matching probability threshold is privacy-protected and stored in a database of the electronic medical record category included in the multimodal medical corpus. The preset matching probability threshold may be a pre-set minimum matching probability value.

[0034] Optionally, after performing privacy processing on at least one target multimodal augmented medical data set whose corresponding entity template matching probability in the target multimodal augmented medical data set is greater than or equal to a preset matching probability threshold and storing it in a database of the electronic medical record category included in the multimodal medical corpus, the method further includes: The first step is to perform standardized verification mapping on the multimodal augmented medical dataset to be classified and the preset medical knowledge graph to determine the medical knowledge graph storage category set of the multimodal augmented medical dataset to be classified, wherein the above-mentioned multimodal augmented medical dataset to be classified is at least one target multimodal augmented medical dataset with a probability less than a preset matching threshold. The above-mentioned preset medical knowledge graph can be an existing knowledge graph related to medical data. In practice, the above-mentioned execution subject can first determine whether the medical entity set corresponding to the multimodal augmented medical dataset to be classified exists in the preset medical knowledge graph. Then, in response to the existence, according to the preset medical knowledge graph, the medical entity set corresponding to the multimodal augmented medical dataset to be classified is subjected to entity coding mapping to map non-standardized terms to standardized terms to obtain a standardized mapping entity set. Finally, the disease category of each standardized mapping entity in the standardized mapping entity set in the above-mentioned preset medical knowledge graph is determined as the medical knowledge graph storage category set.

[0035] In the second step, according to the above-mentioned medical knowledge graph storage category set, the above-mentioned multimodal amplified medical data set to be classified is privacy-processed and then stored in the database of the medical knowledge graph category included in the above-mentioned multimodal medical corpus.

[0036] The third step is to determine the data source set of each multimodal augmented medical data to be classified in the multimodal augmented medical data set to be classified.

[0037] In the fourth step, based on the data source set, the multimodal augmented medical dataset to be classified is privacy-protected and stored in a database of the data source category. The database of the data source category may include, but is not limited to, at least one of the following: medical images and physical examination reports.

[0038] In some optional implementations of some embodiments, performing multi-level debias correction on the acquired multimodal medical dataset to obtain the multimodal debiased medical dataset may include the following steps: The first step is to extract feature information from the multimodal medical dataset to obtain a medical diagnostic feature information set. Each medical diagnostic feature information in the medical diagnostic feature information set can influence the diagnostic information output by the large medical model and is a feature information of the medical attributes of the multimodal medical dataset. For example, the medical diagnostic feature information may include, but is not limited to, at least one of the following: gender feature information, place of residence feature information, and regional medical proficiency information. In practice, the execution entity may first perform anomaly detection on the multimodal medical dataset using a variational autoencoder to obtain a medical anomaly dataset. Secondly, the anomaly dataset is removed from the multimodal medical dataset to obtain a post-anomaly removal multimodal medical dataset. Thirdly, the large language model is used to extract semantically ambiguous multimodal medical data from the post-anomaly removal multimodal medical dataset to obtain a fuzzy medical dataset. Then, the fuzzy medical dataset is fuzzy mapped using a preset medical fuzzy rule base to obtain a fuzzified medical dataset. The preset medical fuzzy rule base may be a rule base that quantifies fuzzy semantics. For example, the above-mentioned preset medical fuzzy rule base may include: a rule base for quantifying "a little cough" as "mild cough", quantifying "occasional cough" as "intermittent cough", and quantifying "severe cough" as "severe cough". Afterwards, the above-mentioned fuzzy medical data set is denoised to obtain a denoised multimodal medical data set. Finally, the principal component analysis algorithm is used to extract feature information from the above-mentioned denoised multimodal medical data set to obtain a medical diagnosis feature information set. It should be noted that the denoising process based on fuzzy mapping processing performs fuzzy processing on the uncertainty degree of the multimodal medical data set, which can remove obviously erroneous data in the data while reducing the risk of mistakenly deleting potentially useful data, thereby improving the quality of the denoised multimodal medical data set and further improving the data quality of the medical diagnosis feature information.

[0039] In the second step, for each medical diagnostic feature information in the above medical diagnostic feature information set, the following causal inference debiasing steps are performed: Sub-step 1: Perform global perturbation processing on the medical diagnostic feature information to obtain a feature sensitivity value. The feature sensitivity value can represent the degree of influence on the diagnostic result when the attribute value corresponding to the medical diagnostic feature information changes.

[0040] As an example, the execution entity may first determine a set of mixed diagnostic feature information for the medical diagnostic feature information using a preset medical knowledge graph. The mixed diagnostic feature information in the set may be feature information that affects both the medical diagnostic feature information and the medical diagnostic information. Then, the medical diagnostic feature information is perturbed using a feature information perturbation algorithm to obtain perturbed medical diagnostic feature information. The feature information perturbation algorithm may be a random shuffle algorithm or a counterfactual simulation algorithm. The medical diagnostic feature information, the mixed diagnostic feature information set, and the perturbed medical diagnostic feature information are then input into a feature sensitivity analysis model to obtain a pre-perturbation diagnostic accuracy and a post-perturbation diagnostic accuracy. The feature sensitivity analysis model may be a machine learning model used to determine the accuracy of a diagnostic result output after perturbing the attribute values ​​of the medical diagnostic feature information. For example, the feature sensitivity analysis model may be a logistic regression model. Finally, the difference between the pre-perturbation diagnostic accuracy and the post-perturbation diagnostic accuracy is determined as the feature sensitivity value.

[0041] Sub-step 2: Determine, based on the medical diagnostic feature information set, the propensity for causal confounding of the medical diagnostic feature information at different attribute values. The propensity for causal confounding can represent the diagnostic accuracy before and after the perturbation of the medical diagnostic feature information, given the same confounding feature information.

[0042] As an example, the execution entity may use PSM (Propensity Score Matching) to determine the probability of propensity causal confounding of the medical diagnosis feature information under different attribute values ​​based on the medical diagnosis feature information set.

[0043] In the third step, based on the obtained set of tendency causal confounding probabilities and the obtained set of feature sensitivity values, regression debiasing is performed on the multimodal medical dataset to obtain a sensitive debiased multimodal medical dataset. The sensitive debiased multimodal medical data may be the multimodal medical data obtained after removing the medical diagnostic feature information.

[0044] As an example, the above-mentioned execution entity can use DML (Double Machine Learning) to perform regression debiasing on the above-mentioned multimodal medical dataset based on the obtained tendency causal confounding probability set and the obtained feature sensitivity value set to obtain sensitive debiased multimodal medical data.

[0045] The fourth step is to perform feature local interpretation and debiasing on the sensitive debiased multimodal medical dataset to obtain an interpreted debiased multimodal medical dataset. The interpreted debiased multimodal medical data in the interpreted debiased multimodal medical dataset may be medical data obtained by removing from the sensitive debiased multimodal medical data at least one medical diagnostic feature whose impact on other medical diagnostic feature information is greater than or equal to a preset impact difference threshold. In practice, the execution entity may first utilize the Med-PaLM (Medical Pathways Language Model) model to map the multiple medical diagnostic feature information included in the sensitive debiased multimodal medical data to a preset medical knowledge graph for knowledge representation, thereby obtaining a set of medical diagnostic feature vectors. Subsequently, utilizing the SHAP (Shapley Additive Explanations) model interpretation technique, the feature prediction contribution probability value of each of the multiple medical diagnostic feature information to the diagnostic information output by the large medical model is determined, thereby obtaining a set of feature prediction contribution probability values. The feature prediction contribution probability values ​​may be Shapley values. Afterwards, the above-mentioned feature prediction contribution probability value set is grouped and aggregated to obtain a feature prediction contribution probability value group set. Then, the group variation coefficient set of each feature prediction contribution probability value group in the above-mentioned feature prediction contribution probability value group set is determined. The group variation coefficient can be the ratio of the standard deviation and the mean of each feature prediction contribution probability value group. Finally, at least one medical diagnostic feature information whose corresponding group variation coefficient is greater than or equal to a preset group variation coefficient threshold is removed from the sensitive debiased multimodal medical data to obtain interpreted debiased multimodal medical data. The preset group variation coefficient threshold can be a pre-set minimum value for measuring the degree of difference between groups.

[0046] In the fifth step, causal inference debiasing is performed on the debiased multimodal medical dataset to obtain a causal debiased multimodal medical dataset, which serves as the debiased multimodal medical dataset. In practice, the execution entity may first perform abnormal feature causal relationship identification on the debiased multimodal medical dataset to obtain an abnormal feature causal relationship set. The abnormal feature causal relationships in the abnormal feature causal relationship set may be causal relationships that indicate abnormal causal relationships in the debiased multimodal medical data. The abnormal feature causal relationship identification may be performed using a causal graphical model (CGM) or a Peter-Clark (PC) algorithm. Then, the medical diagnostic feature information set corresponding to the abnormal feature causal relationship set is removed from the debiased multimodal medical dataset to obtain the debiased multimodal medical dataset.

[0047] In the process of adopting technical solutions to solve the technical problems mentioned in the background art, the following technical problems often arise: Due to the presence of a large amount of bias and unfair data in multimodal medical datasets, when using multimodal medical datasets to train drug recommendation models, the model performance is low and the drug recommendation accuracy is low. This causes the drug delivery robot to frequently transport and deliver drugs, prolonging the medication time and aggravating the severity of patients with limited mobility. To address the above technical problems, a conventional solution generally uses a causal inference algorithm based on correlation relationships to perform causal inference debiasing on the multimodal medical dataset to obtain sensitive debiased multimodal medical data. However, the above conventional solution still has the following problems: because the causal inference algorithm uses a single correlation relationship to debias the causal relationship in the multimodal medical dataset, the inference of the causal relationship is somewhat one-sided, resulting in low data debiasing accuracy and low quality of the sensitive debiased multimodal medical data. This further leads to low performance and low drug recommendation accuracy of the drug recommendation model, causing the drug delivery robot to frequently transport and deliver drugs, prolonging the medication time and aggravating the severity of patients with limited mobility. Considering the shortcomings of the aforementioned conventional solutions and the advantages and technological status of data processing debiasing algorithms possessed by our research institute partners in this field, we decided to adopt the following solution: Optionally, performing regression debiasing on the multimodal medical dataset based on the obtained tendency causal confounding probability set and the obtained feature sensitivity value set to obtain a sensitive debiased multimodal medical dataset, and controlling the medicine delivery robot to deliver the medicine based on the sensitive debiased multimodal medical data may include the following steps: In the first step, for each medical diagnostic feature information in the above medical diagnostic feature information set, the following causal inference debiasing steps are performed: Sub-step 1: Obtain medical diagnosis intermediary feature information that has a causal relationship with the medical diagnosis feature information. The medical diagnosis intermediary feature information may be feature information obtained from a preset medical knowledge graph that has a causal relationship with the medical diagnosis feature information. The medical diagnosis feature information may be a cause, and the medical diagnosis intermediary feature information may be a result.

[0048] Sub-step 2, constructing a causal path set for the above-mentioned medical diagnosis feature information, the above-mentioned medical diagnosis intermediary feature information, the above-mentioned mixed diagnosis feature information set and the medical diagnosis output information, wherein the above-mentioned medical diagnosis output information can be information of the diagnosis result determined in the multimodal medical data. The causal path in the above-mentioned causal path set can be a graphical representation of the existing causal relationship. The causal path set may include: mixed diagnosis feature information-medical diagnosis feature information-medical diagnosis output information, medical diagnosis feature information-medical diagnosis intermediary feature information-medical diagnosis output information. Figure 2 As shown, Figure 2 The set of causal paths constructed by medically identified influencing feature information (e.g., ethnically sensitive attributes), the above-mentioned medically identified mediating feature information (e.g., genetic attributes), the above-mentioned confounding influencing feature information (e.g., regional attributes), and medical treatment information (e.g., imaging medical features) is displayed.

[0049] Sub-step 3: Determine the diagnostic feature residual value and the diagnostic output feature residual value of the medical diagnostic feature information and the medical diagnostic output information with respect to the mixed diagnostic feature information, respectively. The diagnostic feature residual value may represent the conditional expectation of the conditional dependency relationship between the mixed diagnostic information and the medical diagnostic feature information. The diagnostic output feature residual value may represent the conditional expectation of the conditional dependency relationship between the mixed diagnostic information and the medical diagnostic output information. The determination may be performed using a random forest regression model.

[0050] Sub-step 4, determining the local causal probability of the medical diagnostic feature information and the medical diagnostic output information based on the above-mentioned diagnostic feature residual value, the above-mentioned diagnostic output feature residual value, the propensity causal confounding probability set and the feature sensitivity value set. The above-mentioned local causal probability can characterize the linear relationship between the medical diagnostic feature information and the above-mentioned medical diagnostic output information. As an example, the above-mentioned execution entity can use a linear regression model based on least squares fitting to determine the local causal probability of the above-mentioned medical diagnostic feature information and the above-mentioned medical diagnostic output information based on the above-mentioned diagnostic feature residual value and the above-mentioned diagnostic output feature residual value. The larger the local causal probability value, the stronger the causal relationship between the medical diagnostic feature information and the medical diagnostic output information.

[0051] Sub-step 5: In response to determining that the local causal probability is greater than or equal to a preset local causal probability threshold, counterfactual verification is performed on the medical diagnostic feature information to obtain causal relationship verification information. The causal relationship verification information may be verification information that explores the existence of a causal relationship between the medical feature information and the medical diagnostic output information by constructing a hypothetical scenario with contrary facts through counterfactual reasoning techniques. The preset local causal probability threshold may be a pre-set minimum value used to determine whether a causal relationship exists.

[0052] In a second step, in response to determining that the multiple causal relationship verification information obtained all indicate successful verification, feature removal is performed on the multimodal medical dataset based on the multiple local causal probabilities obtained to obtain a feature-removed multimodal medical dataset as the sensitive debiased multimodal medical dataset. The feature-removed multimodal medical dataset may be medical data obtained by removing from the multimodal medical dataset at least one medical diagnostic feature information having a local causal probability less than a preset local causal probability threshold.

[0053] The third step involves controlling the target server to train a drug recommendation model based on the sensitive, debiased, multimodal medical data to obtain recommended drug information, and controlling a drug delivery robot to deliver the recommended drug set corresponding to the recommended drug information to the target patient's location. The execution entity may input the sensitive, debiased, multimodal medical data into the drug recommendation model for training, thereby obtaining a trained drug recommendation model. The trained drug recommendation model then inputs the target patient's physical condition information into the trained drug recommendation model to obtain recommended drug information.

[0054] The above technical solution and its related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem mentioned in the background technology: "Since the causal relationship in the multimodal medical data set is debiased through a single correlation through a causal inference algorithm, there is a certain one-sidedness in the inference of the causal relationship, resulting in a low accuracy rate of data debiasing, a low quality of sensitive debiased multimodal medical data, and further resulting in a low performance of the drug recommendation model and a low accuracy of drug recommendation, causing the drug delivery robot to frequently transport and deliver drugs, prolonging the medication time and aggravating the severity of patients who do not have the ability to move." This results in a low accuracy rate of data debiasing, a low quality of sensitive debiased multimodal medical data, and a large amount of storage resources wasted. If the above factors are solved, it is possible to improve the accuracy rate of data debiasing, improve the quality and fairness of sensitive debiased multimodal medical data, and further result in a low performance of the drug recommendation model and a low accuracy of drug recommendation, causing the drug delivery robot to frequently transport and deliver drugs, prolonging the medication time and aggravating the severity of patients who do not have the ability to move. To achieve this effect, the present invention firstly obtains the propensity causal confounding probability set and feature sensitivity values ​​through PSM and DML, which can estimate the direct causal effect and remove the false association of sensitive features. Secondly, a causal path set and residual analysis including medical diagnosis intermediary feature information are constructed to separate the direct and indirect effects of the features. Then, through the residual values ​​of multi-dimensional indicators, the propensity causal confounding probability set and feature sensitivity values, cross-modal bias paths can be identified to ensure the rationality of the causal path. Afterwards, through counterfactual verification, the impact of confounding diagnostic feature information can be further reduced, the accuracy of debiasing multimodal medical data sets can be improved, and the quality and fairness of sensitive debiased multimodal medical data can be improved. Finally, the target server is controlled to train the drug recommendation model based on the sensitive debiased multimodal medical data to obtain recommended drug information, and the drug delivery robot is controlled to transport the recommended drug set corresponding to the recommended drug information to the target patient's location. This can improve the accuracy of the recommended drug information, reduce the frequency and duration of the drug delivery robot's transportation, and improve the user experience.

[0055] Step 102 : performing data balanced grouping processing on the multimodal debiased medical dataset to obtain a multimodal balanced medical training dataset and a multimodal balanced medical test dataset.

[0056] In some embodiments, the execution entity may perform data balancing and grouping processing on the multimodal debiased medical dataset to obtain a multimodal balanced medical training dataset and a multimodal balanced medical test dataset. The multimodal balanced medical training data in the multimodal balanced medical training dataset may be data used for subsequent parameter training of a large medical model and include balanced distribution of various medical diagnosis-influencing feature information that influences diagnostic information output. The medical diagnosis-influencing feature information may include, but is not limited to, at least one of the following: gender feature information, ethnicity feature information, regional feature information, and age feature information. The multimodal balanced medical test data in the multimodal balanced medical test dataset may be a dataset used for medical diagnosis and debiasing the large medical model. The execution entity may first perform data balancing processing on the multimodal debiased medical dataset using oversampling, undersampling, or the Synthetic Minority Over-sampling Technique (SMOTE) algorithm to obtain a multimodal balanced medical dataset. Then, the multimodal balanced medical dataset is grouped to obtain a multimodal balanced medical training dataset and a multimodal balanced medical test dataset. The above grouping may be a grouping in which the multimodal balanced medical training dataset and the multimodal balanced medical test dataset account for 0.8 and 0.2 of the multimodal debiased medical dataset respectively.

[0057] Step 103 : Perform resource detection on the server cluster, and select servers that meet resource selection conditions from the server cluster as target servers.

[0058] In some embodiments, the execution entity may perform resource detection on a server cluster and select a server from the server cluster that meets resource selection criteria as a target server. The server cluster may be a cluster consisting of multiple servers used to deploy and train a large medical model. The resource selection criteria may be that the server's idle resources are greater than the resources required to run the multimodal balanced medical training dataset, the multimodal balanced medical test dataset, and the large medical model. The target server may be the server with the most idle resources among at least one server that meets the resource selection criteria.

[0059] Step 104: Control the target server to perform the following model debiasing steps based on the multimodal balanced medical training dataset: Step 1041 , input the multimodal balanced medical training data set into the medical big model to obtain a multimodal medical diagnosis information set.

[0060] In some embodiments, the execution entity may input a multimodal balanced medical training dataset into a large medical model to obtain a multimodal medical diagnostic information set. The large medical model includes a multimodal extraction network, a multimodal alignment collaborative network, a meta-learning monitoring network, a bias detector, a fair adversarial network, and a multi-task learning network. The multimodal medical diagnostic information in the multimodal medical diagnostic information set may be information obtained by using the large medical model to perform disease diagnosis on the input multimodal balanced medical training dataset. The large medical model may be a deep neural network model that performs disease diagnosis on the input multimodal balanced medical training dataset and outputs diagnostic information. The multimodal extraction network may be a neural network model that integrates a Transformer model and a convolutional neural network model (e.g., a residual network model) to concurrently extract feature information from different modalities of the input multimodal balanced medical training dataset and output a multimodal feature vector set. The multimodal alignment collaborative network can be a deep neural network model that utilizes a co-training mechanism to automatically adjust the weight ratios between the different modal feature vectors included in the multimodal feature vector set output by the multimodal extraction network, and that can achieve alignment at the feature and semantic levels. The multimodal alignment collaborative network can be a SCAN (Stacked Cross Attention Network) that utilizes a co-training mechanism. The meta-learning monitoring network can be a meta-learner that combines meta-learning with hyperparameter adaptive adjustment of the medical model, responsible for updating the hyperparameters of the medical model or initializing the initial parameters of the medical model to facilitate rapid search when the data distribution changes. The bias detector can be a neural network model comprising a series of multi-layer Transformer models that detects feature distribution differences in the feature vectors output by the multimodal alignment collaborative network to determine fairness. The fair adversarial network can be a generative adversarial network that incorporates a fairness correction term and generates a set of false feature vectors based on the feature vector set output by the bias detector for adversarial training, thereby making the medical model more focused on fairness across different feature groups. The multi-task learning network described above may utilize a multi-task learning framework. While training the primary task (medical diagnosis), it also uses auxiliary tasks (such as detecting differences in group characteristics) to simultaneously optimize the model's adaptability to different groups, thereby avoiding bias caused by ignoring group differences. The bias detector described above may include a feature extraction network, a sensitive attribute classifier, and a fairness constraint network. The feature extraction network may utilize a multi-layer Transformer or convolutional neural network to extract features from the output feature vectors of the input multimodal alignment collaborative network.The sensitive attribute classifier can be a fully connected network that predicts sensitive attributes (e.g., gender or ethnicity). The fairness constraint network can be a network that calculates inter-group feature distribution differences. This inter-group feature distribution difference can be determined by determining the maximum mean difference between different groups or by measuring the difference in distance between probability distributions between different groups. The bias detector can be trained using an adversarial loss and a distribution alignment loss. The adversarial loss can be the cross-entropy loss calculated for the sensitive attribute classifier using a gradient reversal layer (GRL). The distribution alignment loss can be a loss that calculates the maximum mean difference between feature distributions of different groups. The gradient reversal layer can reverse the gradient of the sensitive attribute classifier during backpropagation, forcing the feature extractor to generate features unrelated to the sensitive attribute. The bias detector can be a detector that uses dynamic weighting (e.g., uncertainty weighting) to balance the adversarial loss and distribution alignment loss. Through the dual mechanisms of adversarial training and distribution alignment, the bias detector can ensure that the model eliminates biased features related to sensitive attributes while maintaining performance on the primary task. The output of the multimodal extraction network can be the input of the multimodal alignment collaborative network. The output of the multimodal alignment collaborative network is the input of the meta-learning monitoring network, the bias detector and the multi-task learning network executed in parallel. The output of the bias detector is the input of the fair adversarial network.

[0061] It should be noted that the multimodal extraction network can dynamically adjust the weights of different modal data to reduce the bias caused by single modal data; the multimodal alignment collaborative network can automatically optimize the alignment relationship between different models, improve the multimodal fusion effect, and reduce the bias caused by improper modal fusion; the meta-learning monitoring network can automatically detect and correct the bias caused by improper selection of training parameters, and improve the adaptability and fairness of large medical models; the bias detector can be a mechanism for joint training of fairness constraints and adversarial networks, which can improve the distribution of data from different groups in large medical models and reduce model bias; the fair adversarial network can detect the differences in predicted distributions between different groups by incorporating fairness correction terms into the loss function, so that large medical models pay more attention to the fairness of data from different groups and have less prediction bias; the multi-task learning network can detect the differences in different disease characteristics while diagnosing the disease, and can avoid ignoring the bias caused by group differences.

[0062] Step 1042 : Input the multimodal medical diagnosis information set and the multimodal medical sample diagnosis information set into the multimodal adversarial fair debiasing loss function corresponding to the large medical model to obtain the model loss value.

[0063] In some embodiments, the execution entity may input the multimodal medical diagnosis information set and the multimodal medical sample diagnosis information set into the multimodal adversarial fair debiasing loss function corresponding to the medical large model to obtain a model loss value, wherein the multimodal medical sample diagnosis information set may be the sample medical diagnosis information corresponding to the multimodal balanced medical training data set. The model loss value may be a numerical value used to measure whether the training of the medical large model is complete. The multimodal adversarial fair debiasing loss function may be a loss function including a classification cross entropy loss term for outputting medical diagnosis information, an adversarial loss term, a multimodal distribution alignment loss term, and a debiasing fairness correction loss term. The adversarial loss term may be a loss function corresponding to a fair adversarial network. The multimodal distribution alignment loss term may be a loss function corresponding to a multimodal alignment collaborative network. The debiasing fairness correction loss term may be a loss function corresponding to a bias detector.

[0064] In the process of adopting technical solutions to solve the technical problems mentioned in the background technology, the following technical problems often arise: how to accurately determine the accuracy of the diagnostic information output by the medical large model and the degree of debiasing of the input multimodal balanced medical training dataset by the medical large model. Due to the weak debiasing ability of the medical large model, the generated debiased multimodal medical dataset contains a large amount of redundant data, resulting in low performance of the drug recommendation model and low drug recommendation accuracy, causing the drug delivery robot to frequently transport and deliver drugs, and prolonging the medication time. To address the above technical problems, the conventional solution is generally to iteratively train the medical large model based on the cross-loss function value of the predicted output diagnostic information and the actual output diagnostic information obtained by the medical large model to obtain a trained medical large model. However, the conventional solution still has the following problems: it is difficult to accurately predict the output of the medical large model containing multiple sub-network modules based only on the cross-loss function value of the overall predicted output and the actual output. This results in low debiasing accuracy based on the medical large model and a large amount of redundant data in the generated debiased multimodal medical dataset. This results in low performance of the drug recommendation model and low drug recommendation accuracy, causing the drug delivery robot to frequently transport and deliver drugs, and prolonging the medication time. Considering the shortcomings of the aforementioned conventional solutions and the advantages and current state of large-scale model training technology possessed by our research institute partners in this field, we decided to adopt the following solution: In some optional implementations of some embodiments, the multimodal adversarial fair debiasing loss function may include: a group reconstruction regularization constraint loss function, a medical diagnosis classification cross entropy loss function, an adversarial loss function, and a fairness correction loss function based on statistical parity. The group reconstruction regularization constraint loss function may be a loss function that uses a regularization method to encourage the medical large model to fully learn the feature representations of the minority group. The medical diagnosis classification cross entropy loss function may be a loss function that quantifies the degree of difference between the predicted probability distribution of the medical large model and the true distribution. The adversarial loss function may be a loss function that trains the fair adversarial network to predict the debiasing ability of sensitive attributes. The fairness correction loss function based on statistical parity may be a loss function that characterizes the degree of inter-group differences corresponding to the feature information set in the predicted multimodal balanced medical training dataset.

[0065] Optionally, the above-mentioned inputting the multimodal medical diagnosis information set and the multimodal medical sample diagnosis information set into the multimodal adversarial fair debiasing loss function corresponding to the large medical model to obtain the model loss value, and controlling the target server to control the medicine delivery robot to transport the medicine according to the model loss value may include the following steps: The first step is to divide the multimodal balanced medical training dataset into sensitive attribute groups to obtain a target medical population dataset. The target medical population data in the target medical population dataset may be a group consisting of at least one multimodal balanced medical training data set whose sample size is less than a preset sample size threshold. The preset sample size threshold may be a preset minimum sample size. In practice, the execution entity determines a medical sensitive attribute set in the multimodal balanced medical training dataset using a statistical hypothesis testing method. The medical sensitive attributes in the medical sensitive attribute set may be attribute information whose impact on the accuracy of the medical diagnosis output is greater than or equal to a preset impact threshold. The preset impact threshold may be a preset minimum impact threshold. The statistical hypothesis testing method may include, but is not limited to, at least one of the following: a t-test and an analysis of variance. Then, the number of training samples in the multimodal balanced medical training data corresponding to each medical sensitive attribute in the medical sensitive attribute set is determined to obtain a training sample dataset. Finally, samples corresponding to at least one medical sensitive attribute corresponding to at least one training sample data set whose training sample size is less than or equal to the preset sample size threshold are filtered from the training sample dataset to form the target medical population dataset.

[0066] In the second step, feature extraction and reconstruction are performed on the target medical population dataset to obtain a target population feature vector set and a reconstructed target population feature vector set. The target population feature vectors in the target population feature vector set can represent the semantic and visual information of the target medical population data. The feature extraction can be performed using a multimodal extraction network and a multimodal alignment collaborative network. The reconstructed target population feature vector set can be a feature vector set obtained by reconstructing the target population feature vector set. The feature reconstruction can be performed using a reconstruction network. The reconstruction network can be an autoencoder.

[0067] The third step is to use the group reconstruction regularization constraint loss function to determine the group reconstruction regularization constraint function value based on the target group feature vector set and the reconstructed target group feature vector set. The group reconstruction regularization constraint function value can represent the ability to reconstruct the target group feature vector set, that is, the degree to which the characteristics of the target medical group dataset are accurately restored. The group reconstruction regularization constraint loss function can be expressed as: .

[0068] in, represents the group reconstruction regularization constraint loss function. A collection of indices representing the target medical population dataset. Represents the i-th target group feature vector in the target group feature vector set. Represents the i-th reconstructed target group feature vector in the reconstructed target group feature vector set.

[0069] The fourth step is to use the medical diagnosis classification cross entropy loss function to determine the value of the medical diagnosis classification cross entropy loss function based on the above multimodal medical diagnosis information set and the above multimodal medical sample diagnosis information set. The value of the medical diagnosis classification cross entropy loss function can characterize the difference between the predicted probability distribution of the quantitative medical large model and the true distribution. The medical diagnosis classification cross entropy loss function can be expressed as: .

[0070] in, Represents the cross entropy loss function for medical diagnosis classification. Indicates the number of multimodal medical diagnosis information included in the multimodal medical diagnosis information set. Represents the number of classification categories for medical diagnosis information. Represents the medical diagnosis information of the i-th classification category c in the multimodal medical diagnosis information set. Represents the sample diagnosis information of the i-th classification category c in the multimodal medical sample diagnosis information set.

[0071] The fifth step is to use the adversarial loss function to determine the adversarial loss function value of the fair adversarial network included in the above-mentioned medical large model. The adversarial loss function value can represent the ability of the trained fair adversarial network to predict sensitive attributes without bias. The adversarial loss function can be expressed as: .

[0072] in, represents the adversarial loss function. represents the predicted probability of the fair adversarial network. Represents the sensitive attributes in the i-th target medical population data.

[0073] Step 6: Determine the fairness-corrected loss function value of the bias detector included in the medical macro model using a fairness-corrected loss function based on statistical parity. The fairness-corrected loss function value can represent the degree of inter-group differences in the target medical population dataset. The fairness-corrected loss function can be expressed as: .

[0074] in, represents the fairness correction loss function. Represents any target medical population data in the target medical population dataset. Represents the target medical population dataset. Represents target medical population data Predicted as positive in the above biased detector The average probability of . Indicates that the target medical group dataset is predicted as positive in the above bias detector The average probability of .

[0075] In the seventh step, the multimodal balanced medical test dataset is debiased based on the group reconstruction regularization constraint function value, the medical diagnosis classification cross entropy loss function value, the adversarial loss function value, and the fairness correction loss function value to generate a debiased multimodal medical dataset. In practice, the execution entity may first determine a set of loss weight values ​​for the group reconstruction regularization constraint function value, the medical diagnosis classification cross entropy loss function value, the adversarial loss function value, and the fairness correction loss function value. The loss weight value set may characterize the importance of each loss function value. This determination may be made by dynamically adjusting the meta-learning monitoring network in the large medical model. Then, the group reconstruction regularization constraint function value, the medical diagnosis classification cross entropy loss function value, the adversarial loss function value, the fairness correction loss function value, and the loss weight value set are weighted and summed to obtain the model loss value. Then, in response to determining that the model loss value does not meet the preset loss condition, a multimodal balanced medical training dataset is reacquired to iteratively train the aforementioned large medical model until the model loss value corresponding to the large medical model meets the preset loss condition, thereby obtaining a trained large medical model. Finally, the multimodal balanced medical test dataset is input into the trained large medical model to perform model debiasing processing, thereby obtaining a debiased multimodal medical dataset.

[0076] In step 8, the target server is controlled to train a drug recommendation model based on the debiased multimodal medical dataset to obtain recommended drug information. The drug delivery robot is then controlled to deliver the recommended drug set corresponding to the recommended drug information to the target patient's location. The implementation of this step can be referenced to the implementation of step 105.

[0077] The above technical solution and its related contents, as an inventive point of the embodiment of the present disclosure, solve the technical problem mentioned in the background technology: "It is difficult to make accurate output predictions for a large medical model containing multiple sub-network modules only through the cross-loss function value of the overall predicted output and the actual output, resulting in low accuracy of debiasing based on the large medical model, low performance of large medical model training, and a large amount of redundant data in the generated multimodal medical data set after debiasing, resulting in low performance of the drug recommendation model, low accuracy of drug recommendation, and frequent transportation and delivery of medicines by the drug delivery robot, which prolongs the time of medication." The generated multimodal medical data set after debiasing contains a large amount of redundant data, resulting in low performance of the drug recommendation model, low accuracy of drug recommendation, and frequent transportation and delivery of medicines by the drug delivery robot, which prolongs the time of medication. If the above factors are solved, the performance and accuracy of the drug recommendation model can be improved, the number of drug delivery times by the drug delivery robot can be reduced, and the time of medication can be shortened. To achieve this effect, the present invention first divides the multimodal balanced medical training data set into sensitive attribute groups, obtains a minority medical group data set, and performs feature extraction and reconstruction. This can improve the understanding of minority group characteristics, improve the learning of minority group characteristics by the medical large model, and force the medical large model to learn the feature representation of the minority group through the group reconstruction regularization constraint loss function, so that the model can fully consider the characteristics of the minority group when predicting and reduce bias against the minority group. Secondly, the medical diagnosis classification cross entropy loss function value is calculated through the sample output and the output of the medical large model, and the medical large model is accurately grasped as a whole to improve the diagnostic accuracy of the medical large model. Then, the adversarial loss function value is determined, and feature representations unrelated to sensitive attributes can be learned through a fair adversarial network to reduce bias at the feature level. Afterwards, the fairness correction loss function value is determined, and the differences in predicted distributions between different groups can be monitored in real time, so that the medical large model pays more attention to the fairness performance of data from different groups during training and reduces prediction bias. Finally, in response to determining that the model loss value does not meet the preset loss condition, iterative model training is performed on the medical model. This can improve the accuracy of the medical model's debiasing, increase the speed of model training, and reduce the model iteration time. Finally, based on the model loss value, the target medicine delivery robot is controlled to transport the recommended medicine set corresponding to the recommended medicine information to the target patient's location. This can improve the accuracy of the recommended medicine information, reduce the frequency and duration of the medicine delivery robot's transportation, and enhance the user experience.

[0078] Step 1043 : In response to determining that the model loss value satisfies the preset loss condition, a model debiasing process is performed on the multimodal balanced medical test dataset to obtain a debiased multimodal medical dataset.

[0079] In some embodiments, the execution subject may perform model debiasing on the multimodal balanced medical test data set in response to determining that the model loss value satisfies a preset loss condition, thereby obtaining a debiased multimodal medical data set. The preset loss condition may be a condition that the model loss value is less than or equal to a preset loss threshold. The preset loss threshold may be a pre-set minimum value. Figure 3 The figure shows a comparison of model performance (accuracy) and fairness between the present disclosure and a baseline debiasing scheme based on a large language model. The horizontal axis compares different baseline debiasing schemes with the debiasing technique of the present disclosure. A baseline scheme without debiasing can be a scheme that directly debiases the model without processing the multimodal medical dataset. A simple preprocessing scheme can be a scheme that debiases the model only after data resampling or reweighting. A simple reprocessing scheme can be a scheme that debiases the model by adding a simple fairness penalty term to the loss function. The vertical axis represents the score of the evaluation metric, with higher scores being better. The diagnostic accuracy mentioned above can measure the accuracy of the model output. The fairness score can measure the fairness of the model across different groups, with higher scores indicating smaller inter-group differences. The debiasing effect of the baseline scheme without debiasing is acceptable in terms of accuracy, but poor in terms of fairness. The debiasing effect of the simple preprocessing scheme improves fairness, but often at the expense of overall accuracy. The debiasing effect of the simple reprocessing scheme achieves a certain balance between accuracy and fairness, but the effect is limited. The disclosed debiasing effect balances fairness and accuracy while simultaneously improving accuracy and fairness.

[0080] In step 105 , a drug recommendation model is trained based on the debiased multimodal medical dataset to obtain recommended drug information, and a drug delivery robot is controlled to transport the recommended drug corresponding to the recommended drug information to the location of the target patient.

[0081] In some embodiments, the execution entity may train a drug recommendation model based on the debiased multimodal medical dataset to obtain recommended drug information, and control a drug delivery robot to transport the recommended drug corresponding to the recommended drug information to the target patient's location. The recommended drug information may be information output by the drug recommendation model that is applicable to the target patient's symptoms. The drug delivery robot may be a robot for automated drug delivery. The target patient may be a patient who is unable to move. The drug recommendation model may be a neural network model that predicts drug recommendations based on the input debiased multimodal medical dataset and outputs recommended drug information. The drug recommendation model may be a TCMLLM-PR (Traditional Chinese Medicine Language Model-Prescription Recommendation) model.

[0082] Optionally, after step 105, the execution subject may further perform the following steps: In the first step, in response to determining that the above-mentioned model loss value does not meet the preset loss condition, a multimodal medical dataset from a different source is re-collected as a resampled multimodal medical dataset.

[0083] The second step is to generate a resampled balanced medical dataset based on the resampled multimodal medical dataset. For specific implementation, please refer to the implementation steps of steps 102 to 104.

[0084] In the third step, based on the resampled and balanced medical dataset, the medical model is subjected to model decision path detection to obtain model decision path detection information. The model decision path detection information may be information analyzing the decision-making process of the medical model's output on a specific sample. For example, the model decision path detection information may indicate that the medical model, when predicting drug treatment response in the output medical diagnosis information, overemphasizes the patient's ethnicity and underemphasizes more critical medical characteristics such as disease severity. As an example, the execution entity may utilize an attention mechanism visualization tool to perform model decision path detection on the medical model based on the resampled and balanced medical dataset to obtain model decision path detection information. The attention mechanism visualization tool can be a tool for analyzing model decision paths to effectively identify introduced bias. The attention mechanism visualization tool may include, but is not limited to, at least one of the following: LIME (Local Interpretable Model Explanations) or SHAP (Shapley Additive Explanations).

[0085] The fourth step is to perform model debiasing training on the medical big model according to the above-mentioned model decision path detection information to obtain the first medical debiased big model. As an example, the above-mentioned execution subject can first determine the model weight matrix of the above-mentioned medical big model through the above-mentioned model decision path detection information. Then, the above-mentioned model weight matrix is ​​regularized by L1 norm to suppress the feature weights of the model weight matrix, and the weight matrix corresponding to the medical big model with the model weight matrix removed is weight enhanced to obtain the adjusted model weight matrix, which is used as the weight matrix of the medical big model to perform model debiasing training on the medical big model to obtain the first medical debiased big model. This step can make the medical big model pay more attention to the medical-related features that really affect medical diagnosis, and reduce the bias problem caused by improper feature selection.

[0086] The fifth step is to fine-tune the first medical debiasing large model based on the obtained medical data feedback information to obtain a medical fine-tuned large model. The medical data feedback information can be the annotation feedback data that experts and scholars found that the medical large model has insufficient diagnostic accuracy for a small number of resampled multimodal medical data included in the obtained resampled multimodal medical data set. For example, the medical data feedback information can be "the medical large model's inaccurate diagnosis caused by inaccurate recognition of genetic features specific to rare diseases." This step can better learn the characteristics of minority groups, improve fairness in specific language medical scenarios, and reduce the medical large model's excessive dependence on majority group data. As an example, the above-mentioned execution entity can first perform keyword extraction on the medical data feedback information to obtain a feedback keyword set. Through the adapter fine-tuning technology, only the parameters of the neural network layer responsible for extracting the above feedback keyword set in the first medical debiasing large model are fine-tuned to obtain a medical fine-tuned large model.

[0087] In the sixth step, based on the acquired dialogue feedback annotation information, the medical fine-tuning large model is trained for model debiasing, obtaining a second medical debiasing large model as the medical large model. The resampled and balanced medical dataset is then determined as the multimodal balanced medical training dataset, allowing the model debiasing step to be repeated. The dialogue feedback annotation information may be feedback information obtained by clinical experts who annotate the conversation between the user and the model. For example, the dialogue feedback annotation information may include {Question: "I've been feeling tired lately. Is there something wrong?", Answer: "Fatigue may be caused by a variety of reasons. Further examination of thyroid function and other indicators is recommended."}. This step can optimize the fairness of the medical large model's diagnosis of complex cases, better adapting to the diversity and complexity of real-world medical scenarios. As an example, the execution entity may first input the dialogue information and dialogue feedback annotation information into the loss function corresponding to the medical fine-tuning large model to obtain a dialogue feedback loss value. The dialogue information may be information from the conversation between the medical large model and the user that does not require standard feedback from clinical experts. Then, the gradient descent optimization algorithm is used to perform model debiasing on the medical fine-tuning large model according to the dialogue feedback loss value to obtain a second medical debiased large model as the medical large model, and the resampled balanced medical dataset is determined as a multimodal balanced medical training dataset to perform the model debiasing step again.

[0088] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a model debiasing device based on a large medical model. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the model debiasing device based on the large medical model can be specifically applied to various electronic devices.

[0089] like Figure 4As shown, a model debiasing device 400 based on a large medical model includes: a multi-level debiasing correction unit 401, a data balancing and grouping unit 402, a resource detection unit 403, a first control unit 404, and a second control unit 405. The multi-level debiasing correction unit 401 performs multi-level debiasing correction on an acquired multimodal medical dataset to obtain a multimodal debiased medical dataset, wherein the multimodal medical dataset is acquired from a multimodal medical corpus. The data balancing and grouping unit 402 performs data balancing and grouping processing on the multimodal debiased medical dataset to obtain a multimodal balanced medical training dataset and a multimodal balanced medical test dataset. The resource detection unit 403 performs resource detection on a server cluster and selects servers from the server cluster that meet resource selection conditions as target servers. The first control unit 404 controls the target server to perform the following model debiasing steps based on the multimodal balanced medical training dataset: inputting the multimodal balanced medical training dataset into the medical large model to obtain a multimodal medical diagnosis information set; inputting the multimodal medical diagnosis information set and the multimodal medical sample diagnosis information set into the multimodal adversarial fair debiasing loss function corresponding to the medical large model to obtain a model loss value; and in response to determining that the model loss value meets a preset loss condition, performing model debiasing on the multimodal balanced medical test dataset to obtain a debiased multimodal medical dataset. The second control unit 405 trains the drug recommendation model based on the debiased multimodal medical dataset to obtain recommended drug information, and controls the drug delivery robot to transport the recommended drugs corresponding to the recommended drug information to the target patient's location.

[0090] It is understood that the units described in the model debiasing device 400 based on the medical model are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the model debiasing device 400 based on the medical large model and the units included therein, and will not be described in detail here.

[0091] Reference below Figure 5 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0092] like Figure 5As shown, electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or programs loaded from a storage device 508 into a random access memory (RAM) 503. RAM 503 also stores various programs and data required for the operation of electronic device 500. Processing device 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to bus 504.

[0093] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0094] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0095] It should be noted that in some embodiments of the present disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. Furthermore, in some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0096] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0097] The computer-readable medium may be included in the electronic device, or may exist independently and not incorporated into the electronic device. The computer-readable medium carries one or more programs. When executed by the electronic device, the one or more programs cause the electronic device to obtain a multimodal medical dataset from a multimodal medical corpus, where the multimodal medical dataset includes the contents of steps 101 to 105.

[0098] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] The units described in some embodiments of the present disclosure may be implemented in software or in hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a multi-level debias correction unit, a data balancing grouping unit, a resource detection unit, a first control unit, and a second control unit. The names of these units do not, in some cases, constitute a limitation on the units themselves. For example, the multi-level debias correction unit may also be described as a "unit that performs multi-level debias correction on an acquired multimodal medical data set to obtain a multimodal debiased medical data set."

[0101] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0102] The above descriptions are merely some preferred embodiments of the present disclosure and illustrate the underlying technical principles. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A model debiasing method based on a large medical model, comprising: performing multi-level debias correction on the acquired multimodal medical dataset to obtain a multimodal debiased medical dataset, wherein the multimodal medical dataset is acquired from a multimodal medical corpus; performing data balancing and grouping processing on the multimodal debiased medical dataset to obtain a multimodal balanced medical training dataset and a multimodal balanced medical test dataset; Performing resource detection on the server cluster, and selecting a server that meets the resource selection conditions from the server cluster as a target server; Control the target server to perform the following model debiasing steps based on the multimodal balanced medical training dataset: Input the multimodal balanced medical training data set into the medical big model to obtain a multimodal medical diagnosis information set; Input the multimodal medical diagnosis information set and the multimodal medical sample diagnosis information set into the multimodal adversarial fair debiasing loss function corresponding to the large medical model to obtain the model loss value; In response to determining that the model loss value satisfies a preset loss condition, performing model debiasing processing on the multimodal balanced medical test dataset to obtain a debiased multimodal medical dataset; Based on the debiased multimodal medical dataset, a drug recommendation model is trained to obtain recommended drug information, and a drug delivery robot is controlled to transport the recommended drug corresponding to the recommended drug information to the location of the target patient.

2. The method according to claim 1, wherein The method further comprises: In response to determining that the model loss value does not satisfy a preset loss condition, reacquiring a multimodal medical dataset from a different source as a resampled multimodal medical dataset; generating a resampled balanced medical dataset based on the resampled multimodal medical dataset; performing model decision path detection processing on the large medical model according to the resampled balanced medical data set to obtain model decision path detection information; performing model debiasing training on the medical large model according to the model decision path detection information to obtain a first medical debiasing large model; Fine-tune the first medical debiasing large model based on the acquired medical data feedback information to obtain a medical fine-tuned large model; Based on the acquired dialogue feedback annotation information, the medical fine-tuning large model is trained for model debiasing to obtain a second medical debiasing large model as the medical large model, and the resampled balanced medical dataset is determined as a multimodal balanced medical training dataset to perform the model debiasing step again.

3. The method according to claim 1, wherein The multimodal medical corpus is obtained by the following steps: Collect multimodal regional medical datasets in different regions and languages; determining a medical credibility set for the multimodal regional medical dataset; According to the medical credibility set, performing deduplication processing based on a sparse attention mechanism on the multimodal regional medical dataset to obtain a deduplicated medical dataset; Performing multimodal feature alignment on the deduplicated medical dataset and the multimodal regional medical dataset after removing the medical text dataset to obtain a multimodal associated medical diagnosis information set; Performing generative data augmentation on the multimodal associated medical diagnosis information set to obtain a multimodal augmented medical data set; The multimodal augmented medical dataset is privacy-classified and isolated for storage to obtain a multimodal medical corpus.

4. The method according to claim 3, wherein: The privacy classification and isolated storage of the multimodal augmented medical dataset to obtain a multimodal medical corpus includes: Performing keyword matching on the multimodal augmented medical dataset and a preset keyword set to obtain a keyword matching result set; Filtering at least one multimodal augmented medical data set corresponding to a keyword matching result indicating a successful match from the multimodal augmented medical data set; performing privacy processing on the at least one multimodal augmented medical data and storing the data in a database of the medical record category included in the multimodal medical corpus; performing distance similarity matching on a target multimodal augmented medical dataset and a preset dynamic medical record template set to obtain an entity template matching probability set, wherein the target multimodal augmented medical dataset is a dataset obtained by removing the at least one multimodal augmented medical data from the multimodal augmented medical dataset; At least one target multimodal augmented medical data in the target multimodal augmented medical data set whose corresponding entity template matching probability is greater than or equal to a preset matching probability threshold is privacy-processed and stored in a database of the electronic medical record category included in the multimodal medical corpus.

5. The method according to claim 1, wherein The multi-modal medical dataset is subjected to multi-level debias correction to obtain a multi-modal debiased medical dataset, including: Extracting feature information from the multimodal medical data set to obtain a medical diagnosis feature information set; For each medical diagnosis feature information in the medical diagnosis feature information set, the following causal inference debiasing step is performed: Performing global perturbation processing on the medical diagnostic feature information to obtain a feature sensitivity value; Determining, based on the medical diagnosis feature information set, the tendency causal confounding probability of the medical diagnosis feature information at different attribute values; performing regression debiasing on the multimodal medical dataset according to the obtained tendency causal confounding probability set and the obtained feature sensitivity value set to obtain a sensitive debiased multimodal medical dataset; performing feature local interpretation debiasing processing on the sensitive debiased multimodal medical dataset to obtain an interpretation debiased multimodal medical dataset; Performing causal inference debiasing processing on the explanation-debiased multimodal medical dataset to obtain a causal debiased multimodal medical dataset as the multimodal debiased medical dataset.

6. A model debiasing device based on a large medical model, comprising: a multi-level debias correction unit, performing multi-level debias correction on the acquired multimodal medical dataset to obtain a multimodal debiased medical dataset, wherein the multimodal medical dataset is acquired from a multimodal medical corpus; a data balancing and grouping unit, performing data balancing and grouping processing on the multimodal debiased medical dataset to obtain a multimodal balanced medical training dataset and a multimodal balanced medical test dataset; A resource detection unit, which performs resource detection on the server cluster and selects a server that meets the resource selection conditions from the server cluster as a target server; The first control unit controls the target server to perform the following model debiasing steps based on the multimodal balanced medical training dataset: inputting the multimodal balanced medical training dataset into the medical big model to obtain a multimodal medical diagnosis information set; inputting the multimodal medical diagnosis information set and the multimodal medical sample diagnosis information set into a multimodal adversarial fair debiasing loss function corresponding to the medical big model to obtain a model loss value; and in response to determining that the model loss value satisfies a preset loss condition, performing model debiasing processing on the multimodal balanced medical test dataset to obtain a debiased multimodal medical dataset; The second control unit trains a drug recommendation model based on the debiased multimodal medical data set to obtain recommended drug information, and controls the drug delivery robot to transport the recommended drug corresponding to the recommended drug information to the location of the target patient.

7. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images