Human phenotype auxiliary identification method, device, equipment, medium and product
By using a visual-language multimodal large model, combined with image features and semantic similarity calculation, the problems of misdiagnosis risk and low efficiency in human phenotypic recognition in clinical practice have been solved, achieving efficient and accurate phenotypic recognition and promoting the development of precision medicine.
Patent Information
- Application Number
- CN202511404972.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-28
- Publication Date
- 2026-01-13
AI Technical Summary
In current clinical practice, human phenotypic recognition relies on physicians' experiential visual assessment and cross-validation with textual information, which poses risks of misdiagnosis and low efficiency.
By employing a visual-linguistic multimodal large model, and calculating the cosine similarity between image feature vectors and the training set, as well as the cosine similarity between semantic vectors, and combining this with a finely tuned and optimized visual-linguistic multimodal large model, automatic recognition of human phenotypes can be achieved.
It has improved the accuracy and efficiency of human phenotypic recognition, reduced human error, and promoted the development of precision medicine.
Smart Images
Figure CN121330737A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of human phenotypic recognition, and in particular to a method, apparatus, device, medium and product for human phenotypic auxiliary recognition. Background Technology
[0002] In current clinical practice, phenotypic identification mainly relies on physicians' experiential visual assessment and cross-validation of textual information. This human-dominated model has two limitations: first, it is prone to cognitive bias when faced with complex phenotypic combinations, which may lead to the risk of misdiagnosis of rare cases; second, the efficiency of phenotypic feature-genotype association analysis is low, which seriously affects the timeliness of treatment.
[0003] Therefore, there is an urgent need for a human phenotype-assisted identification method to assist doctors in phenotype identification and improve identification efficiency and accuracy. Summary of the Invention
[0004] The purpose of this application is to provide a method, device, equipment, medium, and product for assisting in human phenotypic identification, which can assist doctors in phenotypic identification and improve identification efficiency and accuracy.
[0005] To achieve the above objectives, this application provides the following solution:
[0006] Firstly, this application provides a method for human phenotype-assisted identification, including:
[0007] Calculate the cosine similarity between the feature vector of the image of the human phenotype to be identified and the feature vector of each image in the training set; the training set includes the images corresponding to each observable human phenotype term and the ground truth labels of each image; the ground truth labels of the images include: all observable human phenotype terms corresponding to the images;
[0008] The feature vectors of the human phenotypic images to be identified are sorted from largest to smallest by the cosine similarity between them and the feature vectors of each image in the training set. The true labels of the images corresponding to the top K cosine similarity scores are selected to obtain a preliminary candidate label set.
[0009] Calculate the cosine similarity between the feature vector of the human phenotypic image to be identified and the semantic vector of each label in the extended label set; the extended label set is obtained by multi-dimensional expansion of the initial candidate label set;
[0010] The feature vectors of the human phenotype images to be identified are sorted from largest to smallest by the cosine similarity between the feature vectors and the semantic vectors of each label in the extended label set. The labels corresponding to the top N cosine similarities are selected to obtain the initial screening results of human phenotypes.
[0011] The labels from the initial screening of human phenotypes and the images of human phenotypes to be identified are input into a finely tuned and optimized visual-language multimodal large model to obtain the human phenotype terms corresponding to the images of human phenotypes to be identified.
[0012] In one implementation, the process of constructing the training set specifically includes:
[0013] Filter all observable human phenotypic terms from the human phenotypic database;
[0014] Search for images corresponding to each observable human phenotypic term to obtain an image dataset;
[0015] A training set is obtained by adding labels to each image in the image dataset.
[0016] In one embodiment, the fine-tuning and optimization process of the visual-language multimodal large model specifically includes:
[0017] The initial set of negative classes is obtained by removing the true labels from the initial screening results of human phenotypes.
[0018] Calculate the similarity between each label in the initial negative class set and each true label in the training set;
[0019] Sort the similarity between each label in the initial negative class set and each real label in the training set from largest to smallest, and select the labels in the initial negative class set corresponding to the top N similarity scores to obtain semantically similar negative samples.
[0020] Obtain the sampled label set by acquiring all labels in the training set that are semantically unrelated to the deleted true labels;
[0021] The target labels are removed from the sampled label set to obtain semantically irrelevant negative samples; the target labels are the labels in the semantically similar negative samples.
[0022] Using the training set as positive samples and semantically similar and semantically unrelated negative samples as negative samples, the visual-language multimodal large model is fine-tuned and optimized.
[0023] In one implementation, all observable human phenotypic terms are selected from a human phenotypic database, specifically including:
[0024] For any human phenotype term in the human phenotype database, input both the preset prompt word and the human phenotype term into the Qwen3-8B model to obtain the first judgment result;
[0025] Input the preset prompt words and the human phenotypic terms into the DeepSeek-V3 model to obtain the second judgment result;
[0026] If both the first and second judgment results are yes, then the human phenotypic term is determined as an observable human phenotypic term.
[0027] In one implementation, images corresponding to each observable human phenotypic term are searched to obtain an image dataset, specifically including:
[0028] Using terms representing observable human phenotypes as search terms, an initial set of images was obtained through a search engine.
[0029] Images that do not meet the preset conditions in the initial image set are deleted to obtain the image dataset.
[0030] In one implementation, the training set is used as positive samples, and semantically similar negative samples and semantically unrelated negative samples are used as negative samples to fine-tune and optimize the large-scale visual-language multimodal model. Specifically:
[0031] We employ a low-rank adaptive technique, using the training set as positive samples and semantically similar and semantically unrelated negative samples as negative samples, to fine-tune and optimize the large-scale visual-language multimodal model.
[0032] Secondly, this application provides a human phenotype-assisted identification device, comprising:
[0033] The first cosine similarity calculation module is used to calculate the cosine similarity between the feature vector of the human phenotypic image to be identified and the feature vector of each image in the training set; the training set includes the images corresponding to each observable human phenotypic term and the true labels of each image; the true labels of the images include: all observable human phenotypic terms corresponding to the images;
[0034] The preliminary candidate label set determination module is used to sort the feature vectors of the human phenotypic images to be identified with the feature vectors of each image in the training set from largest to smallest, select the true labels of the images corresponding to the top K cosine similarities, and obtain the preliminary candidate label set.
[0035] The second cosine similarity calculation module is used to calculate the cosine similarity between the feature vector of the human phenotypic image to be identified and the semantic vector of each label in the extended label set; the extended label set is obtained by multi-dimensional expansion of the preliminary candidate label set.
[0036] The human phenotype preliminary screening result determination module is used to sort the cosine similarity between the feature vector of the human phenotype image to be identified and the semantic vector of each label in the extended label set from largest to smallest, select the labels corresponding to the top N cosine similarities, and obtain the human phenotype preliminary screening result.
[0037] The human phenotypic terminology recognition module is used to input the labels from the initial screening results of human phenotypes and the human phenotypic image to be identified into a finely tuned and optimized visual-language multimodal large model to obtain the human phenotypic terms corresponding to the human phenotypic image to be identified.
[0038] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the human phenotype-assisted identification method described in any of the preceding claims.
[0039] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the human phenotype-assisted identification method described in any of the preceding claims.
[0040] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the human phenotype-assisted identification method described in any of the preceding claims.
[0041] According to the specific embodiments provided in this application, this application has the following technical effects:
[0042] This application provides a method, device, equipment, medium, and product for assisting in the identification of human phenotypes. In recent years, with the rapid development of deep learning and artificial intelligence technologies, especially the rise of visual-language multimodal large models, their widespread application in the medical field has provided innovative pathways to overcome the aforementioned clinical dilemmas. Visual-language multimodal large models can efficiently fuse image data (such as patient facial features and physical appearance) with language data (such as medical records, family history, and gene mutation information), providing a more accurate tool for automatic phenotype identification. Through multimodal models trained on large-scale image and text data and human-machine collaborative decision-making, the system can assist doctors in quickly identifying uncertain phenotypes, thereby significantly improving the early detection and intervention of genetic diseases. This application uses a finely tuned and optimized visual-language multimodal large model to improve recognition efficiency through an end-to-end automated processing flow, and improves the accuracy of recognition results through the cross-modal understanding capabilities of the visual-language multimodal large model.
[0043] This application utilizes medical imaging to automatically identify external phenotypic features associated with common Mendelian genetic diseases, providing assistance to doctors, greatly improving the efficiency of genetic disease diagnosis, reducing human error, and promoting the realization of precision medicine. With the deepening application of artificial intelligence technology in the medical field, this application not only has significant academic value but will also have a profound impact on the clinical diagnosis and treatment of genetic diseases, promoting the rapid development of precision medicine. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 A flowchart illustrating a human phenotype-assisted identification method provided in an embodiment of this application;
[0046] Figure 2 An image used as an example in this application;
[0047] Figure 3 Image used as another example in this application;
[0048] Figure 4 A schematic diagram of the functional modules of a human phenotypic auxiliary identification device provided in an embodiment of this application;
[0049] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0051] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0052] This application provides a method for assisting in the identification of human phenotypes, such as... Figure 1 As shown, it includes:
[0053] Step 201: Calculate the cosine similarity between the feature vector of the image of the human phenotype to be identified and the feature vector of each image in the training set; the training set includes the images corresponding to each observable human phenotype term and the true labels of each image; the true labels of the images include: all observable human phenotype terms corresponding to the images.
[0054] Step 202: Sort the cosine similarity between the feature vector of the human phenotypic image to be identified and the feature vector of each image in the training set from largest to smallest, select the true labels of the images corresponding to the top K cosine similarity scores, and obtain the preliminary candidate label set.
[0055] Step 203: Calculate the cosine similarity between the feature vector of the human phenotypic image to be identified and the semantic vector of each label in the extended label set; the extended label set is obtained by multi-dimensional expansion of the preliminary candidate label set.
[0056] Step 204: Sort the cosine similarity between the feature vector of the human phenotype image to be identified and the semantic vector of each label in the extended label set from largest to smallest, and select the labels corresponding to the top N cosine similarities to obtain the initial screening results of the human phenotype. N can be 20.
[0057] Step 205: Input the labels from the initial screening results of human phenotypes and the human phenotype image to be identified into the finely tuned and optimized visual-language multimodal large model to obtain the human phenotype terms corresponding to the human phenotype image to be identified.
[0058] To avoid providing the entire human phenotypic vocabulary to the model for processing, this application proposes an efficient candidate phenotypic selection method. This method quickly identifies and selects the top-N most relevant candidate labels by fusing image feature similarity with semantic similarity of human phenotypic terms, providing a foundation for accurate judgment in subsequent large-scale vision-language multimodal models.
[0059] In another exemplary embodiment of this application, the process of constructing the training set specifically includes:
[0060] All observable human phenotypic terms were selected from the human phenotypic database.
[0061] The image dataset was obtained by searching for images corresponding to each observable human phenotypic term.
[0062] A training set is obtained by adding labels to each image in the image dataset.
[0063] In another exemplary embodiment of this application, visually observable phenotypic terms are selected from a human phenotypic database using natural language processing techniques. Specifically, this includes:
[0064] For any human phenotypic term in the human phenotypic database, both the preset prompt and the human phenotypic term are input into the Qwen3-8B model to obtain the first judgment result. The specific preset prompt is "I will give you the name of a human phenotypic term. Please judge whether the term refers to a phenotypic feature that a clinician can directly judge through visual observation. Return only 'yes' or 'no'", which serves as the basis for model classification.
[0065] The preset prompt words and the human phenotypic terms are both input into the DeepSeek-V3 model to obtain the second judgment result.
[0066] If both the first and second judgment results are "yes", then the human phenotype term is determined as an observable human phenotype term. The human phenotype database can be the Human Phenotype Ontology (HPO) database, version 2025-05-06, available at https: / / www.hpo.jax.org. It categorizes over 18,000 terms in the HPO database, retaining approximately 4,000 terms for which both models result in "yes", covering phenotypes that can be identified through images, such as facial deformities, craniofacial asymmetry, limb abnormalities, and postural features.
[0067] In practical applications, both the Qwen3-8B model and the DeepSeek-V3 model are large language models finely tuned with biomedical and clinical corpora. These two models are used as classifiers to determine observable human phenotypic terms.
[0068] In another exemplary embodiment of this application, images corresponding to various observable human phenotypic terms are searched to obtain an image dataset, specifically including:
[0069] Using observable human phenotypic terms as search terms, an initial image set was obtained through search engines. Specifically, based on approximately 4,000 selected observable human phenotypic terms, images were retrieved using search engines such as Google Image Search, with these terms serving as keywords.
[0070] Images that do not meet the preset conditions in the initial image set are deleted to obtain the image dataset.
[0071] In practical applications, images that do not meet the preset conditions in the initial image set are deleted to obtain the image dataset. Specifically, manual screening involves reviewing the search results and filtering according to the following criteria:
[0072] i. The image resolution and quality meet the requirements of computational analysis. Specifically, the image resolution, whether length or width, is not less than 256.
[0073] ii. The phenotype explicitly described on the source webpage is consistent with the query terms.
[0074] iii. Images are labeled as suitable for academic research or originating from licensed platforms (such as Wikimedia Commons, medical education websites).
[0075] IV. Copyright Filtering: Images with unclear or restricted copyright were excluded. The filtering yielded approximately 10,000 images, each associated with at least one observable human phenotypic term.
[0076] In practical applications, a training set is obtained by adding labels to each image in the image dataset, specifically including:
[0077] Step 1: Data preprocessing.
[0078] Image segmentation: The composite image is segmented using an edge detection algorithm (identifying continuous black and white dividing bars with a width > 5 pixels) to generate independent sub-images.
[0079] Edge cropping: Detect the percentage of continuous white pixels at the boundary (threshold > 80%) and automatically crop redundant areas.
[0080] Manual review: Filter out subgraphs that do not match the phenotype after segmentation to ensure data quality.
[0081] This step yields Dataset 1 (single-label dataset), consisting of 13,000 images, each labeled with one HPO term.
[0082] Step 2: Multi-label candidate image recognition.
[0083] 1. Data extraction: Obtain genetic disease-phenotype association data from the Online Mendelian Inheritance in Man (OMIM) database (each genetic disease entry corresponds to multiple HPO terms).
[0084] 2. Co-occurrence probability calculation: Statistically analyze the co-occurrence frequency of each pair of HPO terms in the same genetic disease, and calculate the conditional probability (e.g., P(phenotype B|phenotype A), which represents the probability that phenotype B also exists when phenotype A exists).
[0085] 3. Screening of highly correlated phenotypes: Phenotype pairs with a conditional probability ≥ 0.9 are retained (i.e., when phenotype A appears, the probability of phenotype B appearing is extremely high), resulting in 2000 highly correlated additional phenotypes.
[0086] 4. Manual review: Locate the images corresponding to phenotype A in dataset 1, and check whether each image has phenotype B. Construct dataset 2 based on images with multiple phenotypes. Dataset 2 includes 3000 images, each labeled with multiple confirmed HPO terms.
[0087] This step yields Dataset 2 (a multi-label dataset), consisting of 3000 images, each labeled with multiple HPO terms.
[0088] Step 3: Combine Dataset 1 and Dataset 2, which are labeled with HPO terms, into a training set.
[0089] In practical applications, before calculating the cosine similarity between the feature vector of the image of the human phenotype to be identified and the feature vectors of each image in the training set, the following steps are also included:
[0090] Image feature extraction: High-dimensional visual feature representations are extracted from the human phenotype image to be identified using the CLIP image encoder (clip-vit-large-patch14) to obtain the feature vector of the human phenotype image to be identified.
[0091] In practical applications, the cosine similarity between the feature vector of the human phenotypic image to be identified and the feature vectors of each image in the training set is sorted from largest to smallest. The real labels of the images corresponding to the top K cosine similarity are selected to obtain a preliminary candidate label set. Specifically, the cosine similarity between the feature vector of the human phenotypic image to be identified and the feature vectors of all images in the training set is calculated, and the human phenotypic labels corresponding to the Top-K most similar images are obtained to form a preliminary candidate label set.
[0092] In practical applications, each term (node) in HPO forms a hierarchical structure through "is_a" relationships, constituting a directed acyclic graph. Each node has: Parent node: a more abstract higher-level concept; Sibling nodes: parallel concepts that share a direct parent node; Co-occurring nodes: high-frequency terms that appear simultaneously with the node, mined from the OMIM database. Before calculating the cosine similarity between the feature vector of the image of the human phenotype to be identified and the semantic vectors of each label in the extended label set, the following steps are also taken:
[0093] Step A: Semantic Expansion of the Candidate Set: Based on the human phenotype ontology knowledge base, the initial candidate tag set is expanded in multiple dimensions to obtain an expanded tag set, specifically including:
[0094] Parent node expansion (enhancing abstract expression capabilities).
[0095] Sibling nodes at the same level (resolves confusion between concepts at the same level).
[0096] High-frequency co-occurrence nodes (utilizing data-driven correlations).
[0097] Step B: Precise candidate set filtering: Apply CLIP text encoder to extract semantic features of each tag in the extended tag set to obtain semantic vectors.
[0098] To enhance the accuracy and robustness of the visual-language multimodal large model in recognizing potential human phenotypes in clinical images, this application designs two types of high-quality negative samples for model fine-tuning. These negative samples challenge the model's phenotype recognition capabilities from different dimensions, prompting the model to form more refined discrimination boundaries. In another exemplary embodiment of this application, the fine-tuning and optimization process of the visual-language multimodal large model specifically includes:
[0099] The initial set of negative classes is obtained by removing the true labels from the initial screening results of human phenotypes.
[0100] Calculate the similarity between each label in the initial negative class set and each true label in the training set;
[0101] Sort the similarity between each label in the initial negative class set and each real label in the training set from largest to smallest, and select the labels in the initial negative class set corresponding to the top N similarity scores to obtain semantically similar negative samples.
[0102] Obtain the sampled label set by acquiring all labels in the training set that are semantically unrelated to the deleted true labels;
[0103] The target labels are removed from the sampled label set to obtain semantically irrelevant negative samples; the target labels are the labels in the semantically similar negative samples.
[0104] Using the training set as positive samples and semantically similar and semantically unrelated negative samples as negative samples, the visual-language multimodal large model is fine-tuned and optimized.
[0105] In another exemplary embodiment of this application, the true labels in the initial screening results of human phenotypes are deleted to obtain an initial negative class set; the similarity between each label in the initial negative class set and the true labels is calculated; the similarity between each label in the initial negative class set and each true label in the training set is sorted from largest to smallest, and the labels in the initial negative class set corresponding to the top N similarities are selected to obtain semantically similar negative samples, specifically:
[0106] The candidate set obtained from the human phenotype candidate label screening step, i.e., the initial screening results of human phenotypes, is used as the initial pool. Real labels from clinical images are then removed from the initial pool to form the initial negative class set. From the initial negative class set, based on the semantic relationships between observable human phenotype terms in the HPO database, the Jiang & Conrath semantic similarity between each label in the initial negative class set and the real labels in the training set is calculated. Labels with higher similarity in the initial negative class set are prioritized as "hard negative samples." These samples, because they are closer to the real phenotypes in the HPO semantic space, can effectively train the model to capture subtle phenotypic differences, significantly improving the model's ability to distinguish similar phenotypes.
[0107] In another exemplary embodiment of this application, a sampled label set is obtained by acquiring all labels in the training set that are semantically unrelated to the deleted true labels; the target labels in the sampled label set are then deleted to obtain semantically irrelevant negative samples, specifically as follows:
[0108] Labels that are semantically irrelevant to the deleted true labels (i.e., observable human phenotypic terms whose similarity to the deleted true labels is less than a set threshold) are randomly sampled from the training set. Simultaneously, it is ensured that the sampled labels do not appear in semantically similar negative samples, which have significant semantic differences from the deleted true labels. By introducing negative samples with larger semantic margins, the model is encouraged to build more robust decision boundaries, improving its overall ability to identify phenotypic categories.
[0109] In another exemplary embodiment of this application, the training set is used as positive samples, and semantically similar negative samples and semantically unrelated negative samples are used as negative samples to fine-tune and optimize the large-scale visual-language multimodal model, specifically as follows:
[0110] Low-Rank Adaptation (LoRA) is employed, using the training set as positive samples and semantically similar and semantically irrelevant negative samples as negative samples, to fine-tune and optimize the large-scale visual-language multimodal model. Specifically, LoRA is used for targeted optimization of the large-scale visual-language multimodal model:
[0111] Technical framework: Llama-Factory open-source tool.
[0112] Fine-tuning method: full-layer LoRA adaptation (lora_target:all), parameter configuration rank=64, alpha=64;
[0113] Training configuration: maximum sequence length 4096 tokens, batch size 2 (gradient accumulation 8 steps), learning rate 1.0e-4 (cosine scheduling), 3 training rounds (warm-up ratio 0.1), bf16 mixed precision enabled.
[0114] In another exemplary embodiment of this application, the labels in the initial screening results of human phenotypes and the image of the human phenotype to be identified are input into a finely tuned and optimized visual-language multimodal large model to obtain the human phenotype terms corresponding to the image of the human phenotype to be identified, specifically:
[0115] Validation template construction: Combine the human phenotype image to be identified with each label in the initial screening results of human phenotype according to the template format of the image-phenotype consistency validation experiment.
[0116] Model inference: Input the constructed validation template into the fine-tuned and optimized visual-language multimodal large model Qwen2.5-VL-Instruct.
[0117] Judgment result acquisition: The model outputs a binary judgment result (match / no match) for each combination, and obtains the human phenotypic terminology corresponding to the human phenotypic image to be identified based on the judgment result.
[0118] In practical applications, the training set is dynamically updated. A dataset expansion mechanism is used to dynamically update the training set to cover newly added observable human phenotypic terms in the human phenotypic ontology database. The visual-language multimodal large model is updated along with the training set. The specific steps are as follows:
[0119] Step 1.1: Regularly monitor the latest version of the HPO database, and execute Step 1.2 when new observable human phenotypic terms are extracted.
[0120] Step 1.2: Screening of new observable human phenotypic terms.
[0121] Input the preset prompts and the newly added observable human phenotypic terms into the Qwen3-8B model and the DeepSeek-V3 model (both fine-tuned with biomedical corpus); if both models output "yes", then they are identified as scalable observable human phenotypic terms.
[0122] Step 1.3: Scalable image acquisition and filtering of observable human phenotypic terms
[0123] Images were acquired through the following channels, using scalable, observable human phenotypic terms as search terms:
[0124] Publicly accessible medical image databases, search engines (screening images with clear copyright), and clinical partner institutions (obtaining anonymized patient clinical images requires ethical approval).
[0125] Image selection criteria:
[0126] Resolution: ≥256 pixels in length or width;
[0127] Relevance: The image content clearly corresponds to the search terms;
[0128] Copyright: Meets the requirements for use in academic research.
[0129] Step 1.4: New image labeling and verification.
[0130] Automatic pre-labeling: The pre-trained CLIP model is used to automatically label images and output candidate labels.
[0131] Manual review: Candidate labels are reviewed by two or more clinical experts to correct labeling errors and obtain a new image dataset with correct labels (each image is associated with one or more scalable terms).
[0132] Step 1.5: Semantic expansion of new tags.
[0133] Based on the HPO knowledge base, the real labels of each new image in the new image dataset are semantically expanded in multiple dimensions to obtain the expanded new image label set (which is consistent with the logic of step A above).
[0134] Step 1.6: Generate new negative samples.
[0135] For the ground truth labels in the expanded new image label set, new semantically similar negative samples and new semantically irrelevant negative samples are generated, following the same steps as those mentioned above for generating semantically similar and semantically irrelevant negative samples. The ground truth labels for each new image in the new image dataset are the training set mentioned above, and the expanded new image label set is the initial screening result of human phenotypes mentioned above.
[0136] Step 1.7: Incremental fine-tuning.
[0137] A low-rank adaptive technique (consistent with the original fine-tuning parameters) is adopted. The new image dataset with correct annotation is used as positive samples, and the new semantically similar negative samples and the new semantically unrelated negative samples are used as negative samples to input the visual-language multimodal large model for incremental fine-tuning.
[0138] This application designs three complementary benchmark experimental systems covering common clinical application scenarios, aiming to systematically evaluate the model's ability to accurately identify and discriminate human phenotypes. These experiments evaluate the model's phenotypic consistency judgment, multiple-choice matching, and false-positive exclusion ability, respectively. The specific experimental design is as follows:
[0139] 1. Image-phenotype consistency verification experiment.
[0140] Input composition: a single clinical image and a single human phenotypic term to be validated.
[0141] Task objective: Determine whether there is a valid match between an image and a phenotypic term (output "yes" or "no").
[0142] Sample construction strategy:
[0143] Positive samples: constructed based on real image labels and their direct parent nodes in the human phenotype ontology, reflecting the hierarchical inheritance characteristics of clinical phenotype classification.
[0144] Negative samples: generated using the negative sample generation strategy described above (i.e., methods for generating semantically similar negative samples and semantically unrelated negative samples).
[0145] Evaluation method: The accuracy rate is used as the core indicator, which is the percentage of samples that correctly identify matching relationships out of the total test samples.
[0146] 2. Experiment on accurate identification of multiple candidate phenotypes.
[0147] Input composition: a single clinical image and a set of multiple candidate class phenotypic terms (including a mixture of positive and negative samples).
[0148] Task objective:
[0149] Single-label scenario: accurately identify the only correct phenotypic term from the candidate set.
[0150] Multi-label scenario: Identify all matching phenotypes from the candidate set (the output must be completely consistent with the real label set).
[0151] Sample construction strategy:
[0152] Positive samples: In single-label scenarios, one real label (the image's real label or its direct parent node) is included; in multi-label scenarios, the number of real labels is dynamically adjusted based on the number of real labels in the image—when the number of labels is >5, 2-5 are randomly selected as positive samples for the current task, and when the number of labels is ≤5, all are included.
[0153] Negative samples: generated through negative sample generation strategies (i.e., methods for generating semantically similar negative samples and semantically unrelated negative samples).
[0154] Evaluation method: The strict matching accuracy is used, which is the proportion of samples whose predicted phenotype set is completely consistent with the actual phenotype set (including the extension of the parent node of the ontology).
[0155] 3. Phenotypic false positive exclusion test.
[0156] Input composition: a single clinical image and a set of pure negative sample phenotypes (strictly excluding the true label and its parent node).
[0157] Task objective: Determine "no match" to reject all invalid candidate phenotypes.
[0158] Negative samples: generated using the negative sample generation strategy described above (i.e., methods for generating semantically similar negative samples and semantically unrelated negative samples).
[0159] Evaluation method: The rejection accuracy is used, which is the proportion of samples where the model correctly outputs "no match".
[0160] The three types of experiments serve two purposes: first, to act as an evaluation system for model performance; and second, to serve as a data source for model fine-tuning. First, question-answer pairs are generated based on the three types of experiments. A portion of these question-answer pairs is used to fine-tune the model, resulting in the fine-tuned model. Then, the remaining portion is used to test the model and evaluate its performance.
[0161] Mendelian genetic diseases, as typical hereditary disorders caused by single-gene mutations, are often accompanied by highly identifiable physical characteristics, including distinctive facial features, abnormal skin texture, and limb deformities. These visual phenotypes are not only key biomarkers for disease diagnosis but also the core basis for early intervention. This application proposes a human phenotypic-assisted identification method based on a visual-linguistic multimodal model. This method can automatically identify and extract potential human phenotypic features from unannotated clinical images to assist clinicians in diagnosis, and efficiently identify Mendelian genetic disease-related phenotypic features by leveraging cross-modal understanding capabilities.
[0162] This application represents the first technical solution specifically designed for the assisted identification and extraction of human phenotypes from clinical images, and is groundbreaking in this field, offering the following technical advantages:
[0163] 1. High accuracy in human phenotype recognition.
[0164] This application presents a human phenotypic-assisted recognition method based on a visual-language multimodal large model, achieving an accuracy of 80.5% in clinical image phenotypic recognition tasks. This achievement is primarily due to the following technical reasons:
[0165] Dedicated Dataset Construction: This application constructs the first clinical image-phenotype dataset specifically for human phenotype recognition. This dataset has been carefully labeled and validated, providing high-quality data for model training.
[0166] Leveraging the cross-modal understanding capabilities of large models: By fully utilizing the cross-modal understanding capabilities of visual-language multimodal large models, the recognition performance of human phenotypes is improved.
[0167] Targeted optimization strategy: Using the training set as positive samples and semantically similar negative samples and semantically unrelated negative samples as negative samples, fine-tuning and optimizing the visual-language multimodal large model to achieve contrastive learning. A contrastive learning method that considers the hierarchical structure of human phenotypic terms and image similarity is adopted, combined with a special model fine-tuning strategy, so that the model can better understand the subtle differences and intrinsic relationships of phenotypic features.
[0168] 2. This application possesses excellent scalability, enabling further expansion of the recognition range and improvement of recognition accuracy based on practical applications. The main advantages of this technology stem from:
[0169] Infrastructure advantages: The transfer learning framework based on a vision-language multimodal large model inherits the powerful generalization ability of pre-trained models and significantly reduces the dependence on new data scales.
[0170] Dataset expansion mechanism: Expand the clinical image-phenotype dataset by updating the training set in real time, and continuously enrich the model's knowledge base through incremental learning;
[0171] Efficient fine-tuning strategy: Low-rank adaptation technique is used to achieve efficient parameter updates without retraining the entire model from scratch.
[0172] 3. High processing efficiency.
[0173] This application can complete a single phenotypic recognition task within 1 second, while also possessing batch processing capabilities. It can identify potential human phenotypic features in clinical images in a short time, which relies on:
[0174] Automated recognition process: The end-to-end automated processing improves recognition efficiency.
[0175] Parallel processing capability: The core architecture of Qwen2.5VL is Transformer, which leverages the parallel computing advantages of the Transformer architecture to support the simultaneous processing of multiple clinical images.
[0176] This application also provides several specific examples of applying the human phenotypic-assisted identification method proposed in the above embodiments:
[0177] Example 1:
[0178] Input image is Figure 2 .
[0179] The input text is: "As a medical expert with over 10 years of experience and expertise in Human Phenotypic Ontology (HPO), please carefully observe the provided human medical images and determine whether they show a phenotype of [finger hypoplasia / dysplasia]. Please respond with only capital letters: [YES / NO]".
[0180] The answer is YES.
[0181] Example 2:
[0182] Input image is Figure 2 .
[0183] The input text is: "As a medical expert with over 10 years of experience and expertise in human phenotypic ontology, please accurately identify and annotate all observable phenotypic features from the provided medical imaging data based on the following HPO terminology list: overlapping toes; fifth finger joint contracture; finger dysplasia / hypoplasia; cutaneous syndactyly; fifth finger dysplasia / hypoplasia; bulbous apex of the toe; broad palm; missing distal interphalangeal creases; finger dysplasia; coxa vara; spindle-shaped fingers; none of the above; return only matching HPO terminology names. If no match is found, return 'None of the above.'."
[0184] Answer: Incomplete or underdeveloped finger development
[0185] Example 3:
[0186] Input image is Figure 2 .
[0187] The input text is: "As a medical expert with over 10 years of experience and expertise in Human Phenotypic Ontology (HPO), please accurately identify and annotate all observable phenotypic features from the provided medical imaging data based on the following list of HPO terms: overlapping toes; finger hypoplasia; wide palm; fifth finger hypoplasia / dysplasia; single transverse palmar fold; cutaneous syndactyly; dorsum of the hand edema; fingertip pad protrusion; micrognathia; bulbiform toe; thumb hypoplasia / dysplasia; none of the above; return only matching HPO terminology names. If no match is found, return 'none of the above'."
[0188] Answer: None of the above.
[0189] Example 4:
[0190] Input image is Figure 3 .
[0191] The input text is: "As a medical expert with over 10 years of experience and expertise in Human Phenotypic Ontology (HPO), please accurately identify and annotate all observable phenotypic features from the provided medical imaging data based on the following HPO terminology list: coxa vara; finger dysplasia / hypoplasia; metacarpal dysplasia / hypoplasia; sandal interpeduncle; ulnar deviation of the thumb; phalangeal irregularities; fifth finger dysplasia / hypoplasia; finger deviation; finger dysplasia; hand phalangeal dysplasia / hypoplasia; none of the above; only return matching HPO terminology names. If no match is found, return 'None of the above.'."
[0192] Answer: Fifth finger underdevelopment / dysplasia; finger underdevelopment; metacarpal bone underdevelopment / dysplasia; finger bone underdevelopment / dysplasia; finger underdevelopment.
[0193] Based on the same inventive concept, this application also provides a human phenotype auxiliary identification device for implementing the aforementioned human phenotype auxiliary identification method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the human phenotype auxiliary identification device provided below can be found in the limitations of the human phenotype auxiliary identification method described above, and will not be repeated here.
[0194] In one exemplary embodiment, such as Figure 4 As shown, a human phenotype-assisted identification device is provided, comprising:
[0195] The first cosine similarity calculation module is used to calculate the cosine similarity between the feature vector of the human phenotypic image to be identified and the feature vector of each image in the training set; the training set includes the images corresponding to each observable human phenotypic term and the true labels of each image; the true labels of the images include: all observable human phenotypic terms corresponding to the images;
[0196] The preliminary candidate label set determination module is used to sort the feature vectors of the human phenotypic images to be identified with the feature vectors of each image in the training set from largest to smallest, select the true labels of the images corresponding to the top K cosine similarities, and obtain the preliminary candidate label set.
[0197] The second cosine similarity calculation module is used to calculate the cosine similarity between the feature vector of the human phenotypic image to be identified and the semantic vector of each label in the extended label set; the extended label set is obtained by multi-dimensional expansion of the preliminary candidate label set.
[0198] The human phenotype preliminary screening result determination module is used to sort the cosine similarity between the feature vector of the human phenotype image to be identified and the semantic vector of each label in the extended label set from largest to smallest, select the labels corresponding to the top N cosine similarities, and obtain the human phenotype preliminary screening result.
[0199] The human phenotypic terminology recognition module is used to input the labels from the initial screening results of human phenotypes and the human phenotypic image to be identified into a finely tuned and optimized visual-language multimodal large model to obtain the human phenotypic terms corresponding to the human phenotypic image to be identified.
[0200] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores human phenotypic aid-based identification data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a human phenotypic aid-based identification method.
[0201] Those skilled in the art will understand that Figure 5The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.
[0202] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.
[0203] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.
[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0205] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0206] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.
[0207] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0208] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for human phenotypic-assisted identification, characterized in that, The human phenotype-assisted identification method includes: Calculate the cosine similarity between the feature vector of the image of the human phenotype to be identified and the feature vector of each image in the training set; the training set includes the images corresponding to each observable human phenotype term and the ground truth labels of each image; the ground truth labels of the images include: all observable human phenotype terms corresponding to the images; The feature vectors of the human phenotypic images to be identified are sorted from largest to smallest by the cosine similarity between them and the feature vectors of each image in the training set. The real labels of the images corresponding to the top K cosine similarity scores are selected to obtain a preliminary candidate label set. Calculate the cosine similarity between the feature vector of the human phenotypic image to be identified and the semantic vector of each label in the extended label set; the extended label set is obtained by multi-dimensional expansion of the initial candidate label set; The feature vectors of the human phenotype images to be identified are sorted from largest to smallest by the cosine similarity between the feature vectors and the semantic vectors of each label in the extended label set. The labels corresponding to the top N cosine similarities are selected to obtain the initial screening results of human phenotypes. The labels from the initial screening of human phenotypes and the images of human phenotypes to be identified are input into a finely tuned and optimized visual-language multimodal large model to obtain the human phenotype terms corresponding to the images of human phenotypes to be identified.
2. The human phenotype-assisted identification method according to claim 1, characterized in that, The process of constructing the training set specifically includes: Filter all observable human phenotypic terms from the human phenotypic database; Search for images corresponding to each observable human phenotypic term to obtain an image dataset; A training set is obtained by adding labels to each image in the image dataset.
3. The human phenotype-assisted identification method according to claim 1, characterized in that, The fine-tuning and optimization process of the large-scale vision-language multimodal model specifically includes: The initial set of negative classes is obtained by removing the true labels from the initial screening results of human phenotypes. Calculate the similarity between each label in the initial negative class set and each true label in the training set; Sort the similarity between each label in the initial negative class set and each real label in the training set from largest to smallest, and select the labels in the initial negative class set corresponding to the top N similarity scores to obtain semantically similar negative samples. Obtain the sampled label set by acquiring all labels in the training set that are semantically unrelated to the deleted true labels; The target labels are removed from the sampled label set to obtain semantically irrelevant negative samples; the target labels are the labels in the semantically similar negative samples. Using the training set as positive samples and semantically similar and semantically unrelated negative samples as negative samples, the visual-language multimodal large model is fine-tuned and optimized.
4. The human phenotype-assisted identification method according to claim 2, characterized in that, The database of human phenotypes was used to select all observable human phenotype terms, specifically including: For any human phenotype term in the human phenotype database, input both the preset prompt word and the human phenotype term into the Qwen3-8B model to obtain the first judgment result; Input the preset prompt words and the human phenotypic terms into the DeepSeek-V3 model to obtain the second judgment result; If both the first and second judgment results are yes, then the human phenotypic term is determined as an observable human phenotypic term.
5. The human phenotype-assisted identification method according to claim 2, characterized in that, The image dataset was obtained by searching for images corresponding to various observable human phenotypic terms, specifically including: Using terms representing observable human phenotypes as search terms, an initial set of images was obtained through a search engine. Images that do not meet the preset conditions in the initial image set are deleted to obtain the image dataset.
6. The human phenotype-assisted identification method according to claim 3, characterized in that, Using the training set as positive samples and semantically similar and semantically unrelated negative samples as negative samples, the large-scale visual-language multimodal model is fine-tuned and optimized, specifically as follows: We employ a low-rank adaptive technique, using the training set as positive samples and semantically similar and semantically unrelated negative samples as negative samples, to fine-tune and optimize the large-scale visual-language multimodal model.
7. A human phenotypic auxiliary recognition device, characterized in that, The human phenotype-assisted identification device includes: The first cosine similarity calculation module is used to calculate the cosine similarity between the feature vector of the human phenotypic image to be identified and the feature vector of each image in the training set; the training set includes the images corresponding to each observable human phenotypic term and the true labels of each image; the true labels of the images include: all observable human phenotypic terms corresponding to the images; The preliminary candidate label set determination module is used to sort the feature vectors of the human phenotypic images to be identified with the feature vectors of each image in the training set from largest to smallest, select the true labels of the images corresponding to the top K cosine similarities, and obtain the preliminary candidate label set. The second cosine similarity calculation module is used to calculate the cosine similarity between the feature vector of the human phenotypic image to be identified and the semantic vector of each label in the extended label set; the extended label set is obtained by multi-dimensional expansion of the preliminary candidate label set. The human phenotype preliminary screening result determination module is used to sort the cosine similarity between the feature vector of the human phenotype image to be identified and the semantic vector of each label in the extended label set from largest to smallest, select the labels corresponding to the top N cosine similarities, and obtain the human phenotype preliminary screening result. The human phenotypic terminology recognition module is used to input the labels from the initial screening results of human phenotypes and the human phenotypic image to be identified into a finely tuned and optimized visual-language multimodal large model to obtain the human phenotypic terms corresponding to the human phenotypic image to be identified.
8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the human phenotype-assisted identification method according to any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the human phenotype-assisted identification method as described in any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the human phenotype-assisted identification method as described in any one of claims 1-6.