Drug image recognition management method based on multi-modal learning

Abstract

The invention relates to the technical field of information processing, in particular to a medicine image recognition management method based on multi-modal learning, which comprises the following steps: S1, collecting medicine image data and text data of corresponding medicine names, chemical components, major functions and instructions; s2, cleaning the medicine image data, removing noise, classifying and labeling the medicine image data, and extracting and labeling key information of the text data by utilizing a natural language processing method; and S3, extracting visual features of the medicine image through a convolutional neural network or a Transform visual model. According to the method, the drug information can be comprehensively represented by integrating the visual features of the drug image, the semantic features of the drug text data and the drug chemical features. The multi-modal feature fusion not only enhances the robustness of drug identification, but also significantly improves the accuracy of identification.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a drug image recognition management method based on multimodal learning. Background Art

[0002] Currently, existing drug image recognition management methods generally rely on computer vision and natural language processing technologies, combined with machine learning and deep learning algorithms. The following are some common methods and technologies: 1. Computer Vision Technology: Image preprocessing: This includes steps such as image denoising, cropping, scaling, and normalization to improve image quality and standardize input data.

[0003] Feature extraction: Extract high-level features from images using techniques such as convolutional neural networks (CNNs). CNNs are able to identify local patterns in images and reduce the spatial dimensionality of features through pooling operations.

[0004] Image classification: Train classifiers to classify drug images. Commonly used classification algorithms include support vector machines (SVM), random forests (RF), and deep learning models such as ResNet and Inception.

[0005] 2. Natural Language Processing Technology: Text cleaning: remove noise data in the text, such as stop words, punctuation marks, etc.

[0006] Keyword extraction: Use algorithms such as TF-IDF and TextRank to extract keywords from the text. These keywords can help identify the name, ingredients, functions, etc. of the drug.

[0007] Text classification: Use traditional machine learning algorithms (such as Naive Bayes, Logistic Regression) or deep learning models (such as LSTM, GRU, BERT) to classify text to help identify the type of drug and its related information.

[0008] However, the above-mentioned prior arts usually rely on a single modality (such as images or text), which is prone to misidentification when faced with complex or similar drugs. Single modality information is not sufficient to fully characterize drug characteristics, resulting in low recognition accuracy, especially when the drugs are similar in appearance or the text description is insufficient; and the prior art lacks consistency in the representation of drug information features in different modalities, making it difficult to capture the intrinsic connection between different modalities during the fusion process. The inconsistency of feature representation will reduce the depth of understanding of drug information by the computer system for drug image recognition and management, affecting the final recognition performance. In addition, the lack of a mechanism to dynamically adjust the importance of features in different modalities makes the system less interpretable.

[0009] Therefore, there is a need for a drug image recognition management method based on multimodal learning that can solve the above problems. Summary of the invention

[0010] The present invention provides a drug image recognition management method based on multimodal learning. By integrating the visual features of drug images, the semantic features of drug text data and the drug chemical features, the present invention can comprehensively characterize drug information. This fusion of multimodal features not only enhances the robustness of drug recognition, but also significantly improves the accuracy of recognition.

[0011] The technical solution adopted by the present invention to solve the above technical problems is: a drug image recognition management method based on multimodal learning, comprising the following steps: Step S1: collecting drug image data and corresponding text data of drug name, chemical composition, function and indication, and instructions for use; Step S2: cleaning the drug image data, removing noise and classifying and labeling it, and extracting and labeling key information of the text data using a natural language processing method; Step S3: extracting visual features of the drug image through a convolutional neural network or a Transformer visual model; Step S4: extracting semantic features of the text data using a pre-trained language model; Step S5: extracting chemical features from the chemical composition data of the drug, including the drug molecular structure and pharmacological properties; Step S6: Use the attention mechanism to dynamically adjust the importance of different modal features, and integrate the visual features of drug images, the semantic features of drug text data, and the drug chemical features through a multi-layer heterogeneous graph model; Step S7: narrowing the differences among the visual features of the drug image, the semantic features of the drug text data, and the drug chemical features by a comparative learning method, and obtaining drug multimodal feature representation data; Step S8: Input the drug multimodal feature representation data into a computer system for drug image recognition and management to optimize drug image recognition and text understanding tasks.

[0012] Furthermore, the drug image data and the corresponding text data of the drug name, chemical composition, function and indications and instructions for use collected in step S1 include: Step S1-1: Data source acquisition: Obtain drug image data from hospitals, pharmacies, drug manufacturers, and public databases; Step S1-2: Image data collection: collect the front, back and side angle images of the drug; Step S1-3: Text data collection: collect the drug name, chemical composition, function and indication text data corresponding to the drug image; Step S1-4: Data pairing: Pairing the collected drug image data with its corresponding text data to form a one-to-one correspondence; Step S1-5: Diversity enhancement: Increase the diversity of drug image data through data enhancement.

[0013] Furthermore, in step S2, the drug image data is cleaned, noise is removed, and classification and annotation are performed, and key information of the text data is extracted and annotated using a natural language processing method, including: Step S2-1: performing denoising processing on the drug image data to remove random noise in the image; Step S2-2: performing cropping, scaling, and normalization preprocessing operations on the drug image data; Step S2-3: classify and label the preprocessed drug image data and mark the drug category to which it belongs; Step S2-4: Clean the text data to remove irrelevant characters, punctuation marks and stop words; Step S2-5: Stemming and lemmatization step: performing stemming and lemmatization processing on the cleaned text data; Step S2-6: extracting key information from the text data using a natural language processing method; Step S2-7: Annotate the extracted key information and generate corresponding label data.

[0014] Furthermore, the key information in the text data in step S2-5 includes the drug name, drug chemical composition, drug function, drug instructions, drug adverse reactions, drug storage conditions, drug production information and drug expiration date.

[0015] Furthermore, extracting visual features of the drug image by using a convolutional neural network or a Transformer visual model in step S3 includes: Step S3-1: Loading preprocessed drug image data; Step S3-2: Initialize the convolutional neural network or Transformer visual model; Step S3-3: extracting high-level features of the drug image through a convolutional neural network or a Transformer visual model; Step S3-4: Process the extracted features to generate standardized feature vectors; Step S3-5: Store the extracted feature vector for subsequent use.

[0016] Furthermore, extracting the semantic features of the text data using the pre-trained language model in step S4 includes: Step S4-1: Loading pre-processed drug text data; Step S4-2: Initialize the pre-trained language model; Step S4-3: Encode the text data using a pre-trained language model to extract semantic features; Step S4-4: Processing the extracted semantic features to generate a standardized feature vector; Step S4-5: Store the extracted semantic feature vector for subsequent use.

[0017] Furthermore, the chemical features extracted from the chemical composition data of the drug in step S5 include the molecular structure and pharmacological properties of the drug, including: Step S5-1: Loading the chemical composition data of the drug; Step S5-2: extracting the molecular structure information of the drug from the chemical composition data; Step S5-3: extracting the pharmacological property information of the drug from the chemical composition data; Step S5-4: Encode the extracted molecular structure and pharmacological property information to generate a feature vector; Step S5-5: Store the encoded feature vector for subsequent use.

[0018] Furthermore, in step S6, the attention mechanism is used to dynamically adjust the importance of different modal features, and the visual features of drug images, the semantic features of drug text data, and the drug chemical features are integrated through a multi-layer heterogeneous graph model, including: Step S6-1: Loading the extracted visual features of the drug image, the semantic features of the drug text data, and the drug chemical features; Step S6-2: Initialize the attention mechanism module and the multi-layer heterogeneous graph model; Step S6-3: Use the attention mechanism to dynamically adjust the importance of different modal features; Step S6-4: construct a heterogeneous graph containing nodes of different modalities; Step S6-5: propagating feature information on the heterogeneous graph; Step S6-6: Integrate feature information of different modalities to generate the final multimodal feature representation; Step S6-7: Store the integrated multimodal feature vector for subsequent use.

[0019] Furthermore, in step S7, the differences between the visual features of the drug image, the semantic features of the drug text data, and the drug chemical features are reduced by a contrast learning method to obtain drug multimodal feature representation data including: Step S7-1: Loading the integrated visual features of the drug image, the semantic features of the drug text data, and the drug chemical features; Step S7-2: Initialize the contrastive learning framework; Step S7-3: embedding features of different modalities into the same latent space; Step S7-4: Select positive samples of different modalities; Step S7-5: Select negative samples of different modalities; Step S7-6: Calculate the comparison loss between positive samples and negative samples to reduce feature differences; Step S7-7: updating feature representation according to contrast loss; Step S7-8: Store the updated multimodal feature representation data.

[0020] The advantages of the present invention are: 1. Multimodal feature fusion improves the accuracy of drug identification: The present invention can comprehensively characterize drug information by integrating the visual features of drug images, the semantic features of drug text data, and the drug chemical features. This fusion of multimodal features not only enhances the robustness of drug recognition, but also significantly improves the accuracy of recognition, especially in the case of complex or similar drugs.

[0021] 2. Enhanced consistency and interpretability of drug information: The present invention utilizes a pre-trained language model and contrastive learning method to not only improve the consistency between different modal features, but also reduces feature differences through contrastive learning and enhances the consistency of cross-modal features. In addition, the importance of different modal features is dynamically adjusted through the attention mechanism, so that the final multimodal feature representation is more in line with the needs of actual drug information, thereby improving the interpretability of the system.

[0022] 3. Provides comprehensive drug information management and identification solutions: The present invention covers the entire process from data collection, preprocessing, feature extraction to feature integration and multi-task learning, providing a complete solution for drug image recognition and text understanding. This method can not only cope with the limitations of single-modality data, but also effectively handle the correlation between multiple modality data, providing an efficient and intelligent platform for drug management. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Embodiment 1: The present invention provides a drug image recognition management method based on multimodal learning, comprising the following steps: Step S1: collecting drug image data and corresponding text data of drug name, chemical composition, function and indication, and instructions for use; The drug image data and the corresponding drug name, chemical composition, function and indication text data collected in step S1 include: Step S1-1: Data source acquisition: Obtain drug image data from hospitals, pharmacies, drug manufacturers, and public databases; Step S1-2: Image data collection: collect the front, back and side angle images of the drug; Step S1-3: Text data collection: collect the drug name, chemical composition, function and indication text data corresponding to the drug image; Step S1-4: Data pairing: Pairing the collected drug image data with its corresponding text data to form a one-to-one correspondence; Step S1-5: Diversity enhancement: Increase the diversity of drug image data through data enhancement (such as rotation, scaling, brightness adjustment).

[0025] Specifically, the data source acquisition step in step S1-1 includes: Obtain drug image data from patient medication records in hospital information systems; Obtaining drug image data from pharmacy sales records; Obtain drug image data from drug manufacturers’ product catalogs and instructions; Obtain drug image data from public drug databases (such as the FDA database).

[0026] The image data collection step in step S1-2 includes: Collect images of the front, back, and side of the drug boxes; Collect high-resolution images of drug bottles or pills; Make sure to capture images of your medication under different lighting conditions, including indoor lighting, outdoor lighting, and different backgrounds.

[0027] The text data collection step in step S1-3 includes: Collect drug names, including generic and trade names; Collect a list of the drug's chemical components; Collect information on the functions and indications of drugs; Collect instructions for use of medications, including details such as dosage and usage.

[0028] The data pairing step in step S1-4 includes: Pair each drug image with its corresponding drug name, chemical composition, main function and instructions for use; Ensure the one-to-one correspondence between each set of data to avoid data mismatch.

[0029] The diversity enhancement step in step S1-5 includes: Use image enhancement techniques, such as rotation, scaling, flipping, brightness adjustment, etc., to increase the diversity of drug image data; Through these enhancement techniques, the robustness of the model under different conditions is improved.

[0030] Through these specific steps, we ensure that the collected drug image data and text data are of high quality and diversity, laying a solid foundation for subsequent feature extraction and model training.

[0031] Step S2: Clean the drug image data, remove noise, classify and annotate it, and use natural language processing methods to extract and annotate key information of the text data, specifically including: Step S2-1: performing denoising processing on the drug image data to remove random noise in the image; Step S2-2: performing cropping, scaling, and normalization preprocessing operations on the drug image data; Step S2-3: classify and label the preprocessed drug image data and mark the drug category to which it belongs; Step S2-4: Clean the text data to remove irrelevant characters, punctuation marks and stop words; Step S2-5: Stemming and lemmatization step: performing stemming and lemmatization processing on the cleaned text data; Step S2-6: extracting key information from the text data using a natural language processing method, where the key information in the text data includes drug name, drug chemical composition, drug function, drug instructions, drug adverse reactions, drug storage conditions, drug production information, and drug expiration date; Step S2-7: Annotate the extracted key information and generate corresponding label data.

[0032] Step S3: extracting visual features of the drug image through a convolutional neural network or a Transformer visual model, specifically, including: Step S3-1: Data preparation step: load the preprocessed drug image data, ensure that the image has undergone preprocessing steps such as cropping, scaling, and normalization; convert the image into a data format suitable for the input model (such as Tensor).

[0033] Step S3-2: Model initialization step: Initialize the convolutional neural network (such as ResNet, VGG) or Transformer visual model (such as ViT); set model parameters such as input size, output dimension, etc.; if a pre-trained model is used, load the pre-trained weights.

[0034] Step S3-3: Feature extraction step: input the preprocessed drug image into the initialized convolutional neural network or Transformer visual model; extract high-level features of the image, such as output through the middle layer of the model; obtain the feature map before the last layer of the model as the visual feature of the drug image.

[0035] Step S3-4: Feature processing step: process the extracted features, such as average pooling, maximum pooling, etc.; convert the processed features into a vector of fixed length; normalize the feature vector to ensure consistent distribution of the features.

[0036] Step S3-5: Feature storage step: store the extracted feature vector in a suitable data structure (such as a list, array); save the feature vector for subsequent feature fusion and model training.

[0037] Step S4: extracting semantic features of the text data using a pre-trained language model (such as BERT, RoBERTa), specifically including: Step S4-1: Data preparation: loading the cleaned and preprocessed drug text data; Ensure that the text data is formatted correctly, such as removing stop words and punctuation marks.

[0038] Step S4-2: Model initialization: Initialize a pre-trained language model, such as BERT, RoBERTa, DistilBERT; set model parameters, such as the maximum length of input text, batch size, etc. (if a pre-trained model is used, load the pre-trained weights).

[0039] Step S4-3: Text encoding: input the preprocessed drug text data into the pre-trained language model; use the output of the last layer or a specific layer of the model to extract the semantic features of the text; obtain the feature vector output by the model as the semantic representation of the drug text.

[0040] Step S4-4: Feature processing: Process the extracted semantic features and perform normalization to ensure consistent feature distribution; optionally, perform dimensionality reduction processing (such as PCA, t-SNE) to reduce the dimension of the feature vector.

[0041] Step S4-5: Feature storage: Store the extracted semantic feature vector in a suitable data structure (such as a list, array); save the feature vector for subsequent feature fusion and model training.

[0042] Step S5: Extract chemical features from the chemical composition data of the drug, including the drug molecular structure and pharmacological properties, specifically including: Step S5-1: Data preparation: loading the sorted drug chemical composition data; Step S5-2: Molecular structure extraction: Use chemical informatics tools (such as RDKit) to parse the drug's SMILES string or MOL file to extract molecular structure information; generate molecular fingerprints such as ECFP and MACCS; extract other molecular properties such as molecular weight, LogP value, and the number of hydrogen bond donors and acceptors.

[0043] Step S5-3: Pharmacological property extraction: extract the pharmacological property information of the drug from the chemical composition data, including mechanism of action, target, and metabolic pathway; organize the pharmacological property data to ensure that the information is complete and accurate.

[0044] Step S5-4: Feature encoding: Encode the extracted molecular structure information to generate a feature vector; encode the extracted pharmacological property information to generate a feature vector; optionally, standardize the feature vector to ensure consistent feature distribution.

[0045] Step S5-5: Feature storage: Store the encoded feature vector in a suitable data structure (such as a list, array); save the feature vector for subsequent feature fusion and model training.

[0046] Step S6: Use the attention mechanism to dynamically adjust the importance of different modal features, and integrate the visual features of drug images, the semantic features of drug text data, and the drug chemical features through a multi-layer heterogeneous graph model. Specifically, it includes: Step S6-1: Data preparation: Load the extracted visual features of drug images, semantic features of drug text data, and drug chemical features; ensure that all feature vectors have been standardized and are ready for feature fusion.

[0047] Step S6-2: Feature fusion initialization: Initialize the attention mechanism module, such as using Self-Attention or Multi-Head Attention; initialize the multi-layer heterogeneous graph model, such as using Graph Attention Networks (GAT) or Graph Convolutional Networks (GCN).

[0048] Step S6-3: Application of attention mechanism: Input the features of different modalities into the attention mechanism module; Calculate the correlation between different modal features; dynamically adjust the importance of different modal features to generate a weighted feature matrix.

[0049] Step S6-4: Heterogeneous graph construction: Construct a heterogeneous graph, where each node represents a modal feature; establish connections between nodes of different types (visual, semantic, chemical); define edge types to represent the relationship between different modalities.

[0050] Step S6-5: Graph feature propagation: propagate feature information on heterogeneous graphs, and update node features through graph convolution or other graph neural network algorithms; ensure that features can flow between different types of nodes and capture the interaction between multimodal information.

[0051] Step S6-6: Feature integration: Integrate feature information of different modalities to generate the final multimodal feature representation; feature vectors can be integrated by concatenation, weighted summation, etc.; ensure that the integrated feature vector has good representation ability.

[0052] Step S6-7: Feature storage: Store the integrated multimodal feature vector in a suitable data structure (such as a list or array); save the feature vector for use in subsequent multi-task learning models.

[0053] Step S7: narrowing the differences between the visual features of the drug image, the semantic features of the drug text data, and the drug chemical features by a contrastive learning method to obtain drug multimodal feature representation data, specifically including: Step S7-1: Data preparation: load the integrated visual features of drug images, semantic features of drug text data, and drug chemical features; ensure that all feature vectors are ready for comparative learning.

[0054] Step S7-2: Contrastive learning initialization: Initialize the contrastive learning framework, for example, using SimCLR, MoCo, or BYOL methods; set hyperparameters such as temperature coefficient and batch size.

[0055] Step S7-3: Feature embedding: embed features of different modalities into the same latent space; use a shared encoder or a specific encoder to map features of different modalities to the same dimensional space.

[0056] Step S7-4: Positive sample selection: For each sample, different modal features of the same drug are selected as positive samples; for example, for the image features of a certain drug, its corresponding semantic features and chemical features are positive samples.

[0057] Step S7-5: Negative sample selection: different modal features of different drugs are selected as negative samples; for example, for the image features of a certain drug, the semantic features and chemical features of other drugs are negative samples.

[0058] Step S7-6: Contrastive loss calculation: Calculate the contrastive loss, the goal is to shorten the distance between positive samples and increase the distance between negative samples; use a contrastive loss function (such as InfoNCE loss) to calculate the loss value.

[0059] Step S7-7: Feature update: Update the feature representation according to the calculated contrast loss; use optimization methods such as gradient descent to adjust the model parameters to reduce the difference between features.

[0060] Step S7-8: Feature storage: Store the updated multimodal feature representation data in a suitable data structure (such as a list, array); save the feature vector for use in subsequent multi-task learning models.

[0061] Step S8: Input the drug multimodal feature representation data into a computer system for drug image recognition and management to optimize drug image recognition and text understanding tasks.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A drug image recognition management method based on multimodal learning, characterized in that: The following steps are involved: Step S1: collecting drug image data and corresponding text data of drug name, chemical composition, function and indication, and instructions for use; Step S2: cleaning the drug image data, removing noise and classifying and labeling it, and extracting and labeling key information of the text data using a natural language processing method; Step S3: extracting visual features of the drug image through a convolutional neural network or a Transformer visual model; Step S4: extracting semantic features described by the text data using a pre-trained language model; Step S5: extracting chemical features from the chemical composition data of the drug, including the drug molecular structure and pharmacological properties; Step S6: Use the attention mechanism to dynamically adjust the importance of different modal features, and integrate the visual features of drug images, the semantic features of drug text data, and the drug chemical features through a multi-layer heterogeneous graph model; Step S7: narrowing the differences among the visual features of the drug image, the semantic features of the drug text data, and the drug chemical features by a comparative learning method, and obtaining drug multimodal feature representation data; Step S8: Input the drug multimodal feature representation data into a computer system for drug image recognition and management to optimize drug image recognition and text understanding tasks.

2. According to claim 1, a drug image recognition management method based on multimodal learning is characterized in that: The drug image data and the corresponding drug name, chemical composition, function and indication text data collected in step S1 include: Step S1-1: Data source acquisition: Obtain drug image data from hospitals, pharmacies, drug manufacturers, and public databases; Step S1-2: Image data collection: collect the front, back and side angle images of the drug; Step S1-3: Text data collection: collect the drug name, chemical composition, main function and instructions text data corresponding to the drug image; Step S1-4: Data pairing: Pairing the collected drug image data with its corresponding text data to form a one-to-one correspondence; Step S1-5: Diversity enhancement: Increase the diversity of drug image data through data enhancement.

3. The drug image recognition management method based on multimodal learning according to claim 1, characterized in that: In step S2, the drug image data is cleaned, noise is removed, and classification and annotation are performed, and key information of the text data is extracted and annotated using a natural language processing method, including: Step S2-1: performing denoising processing on the drug image data to remove random noise in the image; Step S2-2: performing cropping, scaling, and normalization preprocessing operations on the drug image data; Step S2-3: classify and label the preprocessed drug image data and mark the drug category to which it belongs; Step S2-4: Clean the text data to remove irrelevant characters, punctuation marks and stop words; Step S2-5: Stemming and lemmatization step: performing stemming and lemmatization processing on the cleaned text data; Step S2-6: extracting key information from the text data using a natural language processing method; Step S2-7: Annotate the extracted key information and generate corresponding label data.

4. The drug image recognition management method based on multimodal learning according to claim 3 is characterized in that: The key information in the text data in step S2-5 includes the drug name, drug chemical composition, drug function, drug usage instructions, drug adverse reactions, drug storage conditions, drug production information and drug expiration date.

5. The drug image recognition management method based on multimodal learning according to claim 1, characterized in that: Extracting visual features of the drug image by using a convolutional neural network or a Transformer visual model in step S3 includes: Step S3-1: Loading preprocessed drug image data; Step S3-2: Initialize the convolutional neural network or Transformer visual model; Step S3-3: extracting high-level features of the drug image through a convolutional neural network or a Transformer visual model; Step S3-4: Process the extracted features to generate standardized feature vectors; Step S3-5: Store the extracted feature vector for subsequent use.

6. The drug image recognition management method based on multimodal learning according to claim 1 is characterized in that: The step S4 of extracting the semantic features of the text data using the pre-trained language model includes: Step S4-1: Loading pre-processed drug text data; Step S4-2: Initialize the pre-trained language model; Step S4-3: Encode the text data using a pre-trained language model to extract semantic features; Step S4-4: Processing the extracted semantic features to generate a standardized feature vector; Step S4-5: Store the extracted semantic feature vector for subsequent use.

7. The drug image recognition management method based on multimodal learning according to claim 1 is characterized in that: The chemical features extracted from the chemical composition data of the drug in step S5 include the molecular structure and pharmacological properties of the drug, including: Step S5-1: Loading the chemical composition data of the drug; Step S5-2: extracting the molecular structure information of the drug from the chemical composition data; Step S5-3: extracting the pharmacological property information of the drug from the chemical composition data; Step S5-4: Encode the extracted molecular structure and pharmacological property information to generate a feature vector; Step S5-5: Store the encoded feature vector for subsequent use.

8. The drug image recognition management method based on multimodal learning according to claim 1, characterized in that: In step S6, the attention mechanism is used to dynamically adjust the importance of different modal features, and the visual features of drug images, the semantic features of drug text data, and the drug chemical features are integrated through a multi-layer heterogeneous graph model, including: Step S6-1: Loading the extracted visual features of the drug image, the semantic features of the drug text data, and the drug chemical features; Step S6-2: Initialize the attention mechanism module and the multi-layer heterogeneous graph model; Step S6-3: Use the attention mechanism to dynamically adjust the importance of different modal features; Step S6-4: construct a heterogeneous graph containing nodes of different modalities; Step S6-5: propagating feature information on the heterogeneous graph; Step S6-6: Integrate feature information of different modalities to generate a final multimodal feature representation; Step S6-7: Store the integrated multimodal feature vector for subsequent use.

9. The drug image recognition management method based on multimodal learning according to claim 1, characterized in that: In step S7, the differences between the visual features of the drug image, the semantic features of the drug text data, and the drug chemical features are reduced by a contrast learning method to obtain drug multimodal feature representation data including: Step S7-1: Loading the integrated visual features of the drug image, the semantic features of the drug text data, and the drug chemical features; Step S7-2: Initialize the contrastive learning framework; Step S7-3: embedding features of different modalities into the same latent space; Step S7-4: Select positive samples of different modalities; Step S7-5: Select negative samples of different modalities; Step S7-6: Calculate the comparison loss between positive samples and negative samples to reduce feature differences; Step S7-7: updating feature representation according to contrast loss; Step S7-8: Store the updated multimodal feature representation data.