Multi-modal deep learning model system and method for multi-classification diagnosis of laryngeal and hypopharyngeal lesions

Abstract

The invention belongs to the technical field of medical diagnosis, and discloses a multi-modal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions, a model is established by adopting laryngoscope photos under white light imaging, and laryngoscope equipment for collecting the photos is equipped in most domestic primary hospitals. Meanwhile, a laryngoscope picture for training covers most lesions of the throat and the hypopharynx and is not limited to the vocal cord level. In order to test the effect of the model on patients in primary hospitals, three medical institutions are found for external verification. The accuracy rate of the method is still 80.00-83.87%, which can be compared with the experience level of endoscopic physicians in 3-10 years. Moreover, the model can generate a heat map, namely, the possible position of the lesion on the image can be displayed, and certain interpretability is achieved. Therefore, popularization and application of the model in primary hospitals are facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical diagnosis, and particularly relates to a multimodal deep learning model system and method for multi-classification diagnosis of laryngeal and hypopharyngeal lesions. Background Art

[0002] In 2022, the newly diagnosed cases of laryngeal cancer and hypopharyngeal cancer globally were 188,960 and 86,276 respectively. Early diagnosis is crucial for saving patients' lives and protecting basic daily functions such as breathing, swallowing, and vocalization. Laryngopharyngeal endoscopy is the main tool for diagnosing all laryngopharyngeal diseases, including laryngopharyngeal cancer. However, training endoscopists to achieve accurate diagnosis is a time-consuming process. Nevertheless, the above problems are likely to be solved by deep learning in the near future. Deep learning can process and perceive various forms of information and is gradually incorporated into the research of various scientific problems, including the classification of laryngopharyngeal diseases and the prediction of lesion histology (mainly benign / malignant).

[0003] Previous studies have shown that the overall accuracy of deep learning models in evaluating laryngeal diseases ranges from 0.806 to 0.997. However, all models have a major flaw - they only rely on laryngopharyngeal endoscopy images for judgment, which may cause the models to lose some information helpful for diagnosis. During real endoscopic examinations, the clinical characteristics of patients (such as age and gender) are obvious, and the purpose of the examination or medical history is usually known in advance, which provides additional useful information for doctors in the diagnosis of endoscopic images.

[0004] Secondly, in previous studies, the diagnostic accuracy of binary classification tasks (such as only classifying as benign or malignant) was relatively high, but the diagnostic accuracy in multi-classification tasks was relatively low. And some models with high accuracy need to use narrowband imaging (NBI). Compared with laryngoscopy using conventional white light imaging, it has an advantage in the ability to detect malignant lesions. However, the required equipment platform and the training time of endoscopists for operating such endoscopic equipment are both relatively high. At present, most primary hospitals in China do not have such equipment, so it is difficult to popularize deep learning models using NBI technology.

[0005] In addition, most of the laryngoscopy photos selected in previous literature focused on the larynx or even only on the vocal cord area. In fact, real laryngoscopy examinations need to observe the laryngeal cavity including the vocal cords and the hypopharyngeal cavity, which also hinders the popularization of previously published models.

[0006] Through the above analysis, the problems and defects of the existing technology are as follows:

[0007] At present, most primary hospitals in China do not have such equipment, so it is difficult to popularize deep learning models using NBI technology. Summary of the Invention

[0008] Aiming at the problems existing in the prior art, the present invention provides a multimodal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions.

[0009] The present invention is implemented as follows. A multimodal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions includes:

[0010] An acquisition module, a training module, a testing module, and a verification module;

[0011] The acquisition module, connected to the training module and the verification module, is used to collect laryngoscope photos under white light imaging through a laryngoscope device, and at the same time record the corresponding patient clinical characteristics (gender and age) and medical history information;

[0012] The training module, connected to the acquisition module, the testing module, and the verification module, is used to train the laryngoscope photos; the laryngoscope photos cover most of the lesions of the larynx and hypopharynx, not limited to the vocal cord level;

[0013] The testing module, connected to the training module and the verification module, is used to test the effect of the model on the patients in this central hospital (i.e., as internal verification);

[0014] The verification module, connected to the acquisition module, the training module, and the testing module, is used for external verification by three medical institutions.

[0015] Furthermore, the training module:

[0016] Is used for training the model; its three types of modalities are laryngoscope images, patient complaints, and clinical characteristics; at the same time, according to its final diagnosis, it is marked as the normal (Normal), non-cancerous lesion (Non Cancer), and cancer (Cancer) groups, and is artificially scored as Normal 0 points, Non Cancer 1.0 - 1.3 points (its subgroup of benign lesions is 1.0, low-grade precancerous lesions is 1.1, high-grade precancerous lesions is 1.3), and Cancer 2.0 points.

[0017] Another object of the present invention is to provide a method for constructing a multimodal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions, including:

[0018] Step 1, use the acquisition module to collect laryngoscope photos under white light imaging through a laryngoscope device, and at the same time record the corresponding patient clinical characteristics (gender and age) and medical history information;

[0019] Step 2, train the laryngoscope photos through the training module; the laryngoscope photos cover most of the lesions of the larynx and hypopharynx, not limited to the vocal cord level;

[0020] Step 3, testing the effect of the model on patients in this central hospital through the test module;

[0021] Step 4: Use three medical institutions as external validation through the validation module.

[0022] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for constructing a multimodal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions.

[0023] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the method for constructing a multimodal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions.

[0024] Another object of the present invention is to provide an information data processing terminal, which is used to implement the multimodal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions.

[0025] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0026] Current mainstream research focuses on visual modeling of laryngoscopic images, but systematically ignores the integration of clinical information associated with the images (such as patient demographic characteristics and medical history data). This technical limitation directly restricts the application potential of diagnostic models in multi-dimensional information integration and hinders the effective improvement of diagnostic accuracy. Especially when faced with: (1) multi-classification diagnostic scenarios, such as the three-classification discrimination of no lesions / benign lesions / malignant lesions; (2) full throat field of view analysis, that is, a wide area assessment beyond the single structure of the vocal cords, the diagnostic efficiency of existing models shows a significant attenuation. How to break through the integration bottleneck of existing multimodal information, optimize the data preprocessing process and build an accurate diagnostic model has become a technical bottleneck that needs to be overcome.

[0027] The technical solution of the present invention is based on a multimodal deep learning framework, which integrates laryngoscope photos under white light imaging with the patient's clinical characteristic data (including gender, age) and outpatient medical history information to build the first multi-classification diagnostic model for laryngeal and hypopharyngeal lesions that integrates multimodal information. Through the synergy of the acquisition module, training module, test module and verification module, this solution achieves efficient data acquisition, alignment, feature extraction and fusion, thereby ensuring that different modal data are jointly learned in the same feature space, providing a unified and accurate input signal for subsequent three-classification detection (distinguishing between normal, non-cancerous lesions and cancer).

[0028] During the model training process, the present invention adopts advanced structures such as convolutional neural networks and multi-layer perceptrons to perform end-to-end joint optimization on image, numerical, and text data. During the testing process (i.e., internal verification), the accuracy, precision, recall, and F1 score of the model of the present invention in the three-class diagnosis of laryngeal and hypopharyngeal lesions are respectively as high as 91.67%, 91.67%, 91.87%, and 91.82%, all exceeding the average level of laryngoscopy experts with more than 10 years of experience (accuracy 86.24%, precision 86.23%, recall 87.49%, F1 score 86.50%). This result proves that through multi-modal data integration, the present invention has greatly improved the discriminant ability of the model for lesion classification and significantly narrowed the gap between clinical experts and the model.

[0029] To verify the applicability of the model of the present invention in primary hospitals, the research team introduced external data from three sub-center primary medical institutions for independent verification. The results showed that the accuracy of the model on the external verification set was still around 80%, comparable to the level of endoscopists with 30 years of clinical experience. In addition, the model also has the function of generating heat maps, which can intuitively display the possible positions of lesions on laryngoscopy images, improving the interpretability of the model and helping physicians understand and adopt the model prediction results, thus promoting its popularization and application in primary hospitals.

[0030] The present invention has achieved a technical breakthrough in multi-modal data integration, feature alignment, and joint learning methods, solving the limitations brought by traditional visual models relying only on single image information. By introducing clinical features and medical history texts, the model constructed by the present invention has higher generalization ability and robustness, and can adapt to data differences collected by different hospitals and different devices. In addition, the collaborative processing of multi-modal data not only improves the accuracy and stability of diagnosis, but also enables the model to achieve real-time and efficient diagnosis in low-resource environments, thus providing an innovative and efficient solution for the early detection of laryngeal lesions.

[0031] The technical solution of the present invention has significant expected benefits and commercial value in clinical applications. By realizing the efficient integration and joint training of multi-modal data, the model of the present invention is expected to be widely promoted in primary medical institutions, reducing the medical risks brought by misdiagnosis or missed diagnosis and improving the cure rate of patients. At the same time, this technical solution provides new ideas for the automated diagnosis of laryngeal and hypopharyngeal lesions, breaking through the technical bottleneck of relying on expert experience for diagnosis for a long time, and has high market application prospects and economic benefits. In the future, the model can also be extended to the multi-class diagnosis of other related lesions, providing further technical support for the application of multi-modal deep learning in the field of medical image diagnosis. Brief Description of the Drawings

[0032] Figure 1It is a block diagram of a multi-modal deep learning model system for multi-class diagnosis of laryngeal and hypopharyngeal lesions provided by an embodiment of the present invention.

[0033] Figure 2 It is a flowchart of a method for constructing a multi-modal deep learning model system for multi-class diagnosis of laryngeal and hypopharyngeal lesions provided by an embodiment of the present invention.

[0034] Figure 3 It is a multi-modal patient sample example provided by an embodiment of the present invention. It involves three modalities, namely patient characteristics, patient chief complaints, and laryngoscope images.

[0035] Figure 4 It is a structural diagram of a multi-modal patient information interaction model provided by an embodiment of the present invention.

[0036] Figure 5 It is a flowchart of data collection and partitioning provided by an embodiment of the present invention.

[0037] Figure 6 It is a visualization attention map (heat map) of multi-modal sample feature extraction provided by an embodiment of the present invention, as well as the score maps given by the model and doctors at all levels.

[0038] Figure 1 Among them: 1. Acquisition module; 2. Training module; 3. Testing module; 4. Verification module. Specific embodiments

[0039] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the following further details the present invention in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] Embodiment 1: Model training and testing based on internal data of the main central hospital

[0041] In this embodiment, the internal data collected by the main central hospital is used for model training and testing. First, laryngoscope photos under white light imaging are collected by the laryngoscope equipment installed in the main central hospital, and the age, gender, and medical history text information of the patients are synchronously recorded to form a data set containing three modalities: images, numerical values, and texts.

[0042] The data collected by the acquisition module is transmitted to the training module through a data transmission interface. The training module uses a convolutional neural network (CNN) to extract features from the laryngoscope images, and at the same time uses a multi-layer perceptron (MLP) and an embedded text encoding model to encode the clinical features and medical history text data respectively.

[0043] Next, the training module aligns the three-modal data through a preset feature alignment unit and fuses them into a unified high-dimensional feature vector. The fused features are used as input for the multi-classification training of the model.

[0044] After training is completed, the testing module receives data from the same internal dataset, tests the trained model, and collects the model output results and relevant metric data.

[0045] Finally, by comparing the actual pathological diagnosis results with the model prediction results, the performance of the model on the internal dataset is verified and the parameters are tuned, and the best model is selected as the preliminary result.

[0046] Example 2: Model Verification Based on External Data from Grassroots Hospitals in Sub-centers **

[0047] In this example, the best model is verified using external data collected from three grassroots hospitals in independent sub-centers. First, the grassroots hospitals in the sub-centers collect laryngoscope photos under white light imaging through their respective laryngoscope devices, and record the clinical characteristics and medical history information of the patients to form an external dataset.

[0048] The acquisition module sends the external data to the verification module through a secure data transmission interface. The verification module performs image preprocessing, numerical normalization, and text vectorization on the external data using the same preprocessing process as the training module.

[0049] After preprocessing, the external data is converted into a unified format with the internal data through the feature alignment unit to form a high-dimensional feature vector that can be directly input into the model.

[0050] The best model then predicts the external data, and the verification module collects and records the classification results of the model for various laryngeal and hypopharyngeal lesions.

[0051] By comparing the actual clinical diagnosis results corresponding to the external data of the sub-center hospitals with the model prediction output, the verification module conducts a statistical analysis of the generalization ability of the model to evaluate the final performance of the model under data from different medical institutions.

[0052] As Figure 1 shown, a multi-modal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions provided by an embodiment of the present invention includes:

[0053] An acquisition module 1 (divided into internal data from the main center hospital and external data from grassroots hospitals in sub-centers), a training module 2, a testing module 3, and a verification module 4;

[0054] The acquisition module 1 collects laryngoscope photos under white light imaging through a laryngoscope device, and inputs clinical feature data such as the age and gender of the patient, as well as the corresponding medical history information. Among them, the internal data from the main central hospital is connected to the training module 2 and the testing module 3 for model training and internal testing; the external data from three primary hospitals of independent sub-centers is connected to the verification module 4 for the final verification of the model performance;

[0055] The training module 2 is connected to the acquisition module 1, the testing module 3, and the verification module 4, and is used for the model to align and train three different formats of modal information: laryngoscope images, clinical features, and medical history texts; the laryngoscope images cover most lesions of the larynx and hypopharynx, not limited to the vocal cord level;

[0056] The testing module 3 is connected to the training module 2 and the verification module 4, and is used to test the effect of the model on the patients in the hospital where the central unit is located, and adjust appropriate parameters according to the results to finally select the best model;

[0057] The verification module 4 is connected to the acquisition module 1, the training module 2, and the testing module 3. The best model constructed by module 3 is used for external verification in three primary medical institutions independent of the main center to evaluate the final performance of the model.

[0058] A multi-modal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions provided by an embodiment of the present invention focuses on the collaborative work of four main modules: the acquisition module 1, the training module 2, the testing module 3, and the verification module 4. These modules are interconnected to jointly form a complete diagnostic process, aiming to improve the diagnostic accuracy and efficiency of laryngeal and hypopharyngeal lesions through deep learning technology.

[0059] The acquisition module 1 first obtains laryngoscope photos under white light imaging through a laryngoscope device, and at the same time inputs the clinical feature data (such as age, gender) of the patient and the corresponding medical history text information. These multi-modal data are stored in the forms of image signals, numerical signals, and text signals respectively. The system performs preliminary preprocessing on the collected images, including image denoising, contrast adjustment, and region cropping, to ensure that the image quality is suitable for subsequent feature extraction; performs normalization processing on the numerical clinical data; and performs word segmentation, stop word filtering, and vectorization processing on the medical history text data to form feature vectors in a unified format, which is convenient for subsequent alignment and fusion between modalities.

[0060] The training module 2 is responsible for aligning and fusing features of data from different sources and modalities. This process first uses a convolutional neural network (CNN) to extract deep features from laryngoscope images, and then maps the image features to the same feature space as the clinical data and text data through a fully connected layer. At the same time, for clinical numerical features and text vectors, a multi-layer perceptron (MLP) and an embedding model are respectively used for feature encoding. Subsequently, the system uses a feature fusion algorithm (such as weighted concatenation or attention mechanism) to fuse the three-modal data, forming a comprehensive high-dimensional feature vector, providing a unified input signal for model training, and ensuring that the information between different data modalities can be synergistically expressed.

[0061] The multi-modal signals after fusion processing are used as input and enter the deep learning model through the training module 2 for multi-classification diagnosis training. During the training process, the model jointly learns the image, numerical, and text features in an end-to-end manner, optimizes using the cross-entropy loss function and other multi-task losses, and continuously updates the model weights using the gradient descent method. At the same time, the testing module 3 evaluates the performance of the current model on the internal dataset in real time, collects various key metrics (such as accuracy, recall, etc.) and feeds them back to the training module to dynamically adjust the model hyperparameters and network structure, realizing the continuous optimization and adaptive enhancement of signal features, and ensuring that the model can fully capture the multi-dimensional information of various lesions in the larynx and hypopharynx.

[0062] After internal training and parameter tuning, the verification module 4 applies the best model to external data from primary hospitals in independent sub-centers for verification. In this process, the system first processes the externally collected data according to the same preprocessing and feature fusion process as the internal data, and then inputs the processed signals into the trained model for diagnostic prediction. The verification module calculates various performance metrics by statistically analyzing the differences between the prediction results and the actual diagnostic results, and comprehensively evaluates the performance of the model under different modal signals. Finally, by comparing the data of internal testing and external verification, the generalization ability and robustness of the model are further confirmed, providing an objective and reliable basis for multi-classification diagnosis, and at the same time providing feedback data for the continuous optimization of future models.

[0063] First, the acquisition module 1 is responsible for acquiring laryngoscope photos under white light imaging through a laryngoscope device. These photos cover most types of lesions in the larynx and hypopharynx regions, not limited to lesions at the vocal cord level. At the same time, clinical feature data such as the patient's age and gender, as well as text information on the corresponding medical history, are entered as the other two modalities of information in addition to the laryngoscope images. The acquisition module 1 is connected to the training module 2 and the verification module 4 to ensure that the acquired image data can be transmitted to the training and verification systems in a timely manner. Relatively clear image acquisition that can distinguish the characteristics of lesions is the basis for the performance of the entire model. Therefore, the acquisition module 1 plays a key role in data preprocessing and image quality control to ensure that the subsequent training and diagnosis processes can be carried out based on accurate and comprehensive image information.

[0064] The training module 2 is responsible for training a deep learning model on the acquired laryngoscope photos. This module is not only connected to the acquisition module 1 to obtain the latest image data, but also maintains close contact with the test module 3 and the verification module 4 for real-time evaluation and optimization during the model training process. The training module 2 uses image data covering various laryngeal and hypopharyngeal lesions to perform feature extraction and classification learning through a multi-layer neural network, gradually improving the model's recognition ability and classification accuracy. During the training process, the model continuously adjusts its parameters to minimize the prediction error and ensure its efficiency and reliability in practical applications.

[0065] The role of the test module 3 is to evaluate the actual application effect of the trained deep learning model in patients in primary hospitals. This module is connected to the training module 2 and the verification module 4, and can obtain the trained model and test it in a real medical environment. By testing on the actual patient data in primary hospitals, the test module 3 can examine the generalization ability and adaptability of the model to ensure that it can still maintain a high level of diagnostic accuracy under different medical conditions. In addition, the test module 3 can also feedback the test results to the training module 2 to guide the further optimization and improvement of the model.

[0066] Finally, the verification module 4 undertakes the external verification task of the model by connecting the acquisition module 1, the training module 2, and the test module 3. This module conducts independent verification in three different medical institutions to ensure that the model still performs excellently under diverse medical environments and different equipment conditions. The verification module 4 evaluates the stability and consistency of the model by comparing the data and diagnostic results of different institutions, further enhancing its credibility and practicality in clinical applications. Overall, the multi-modal deep learning model of the present invention realizes the full-process optimization from data acquisition, model training, effect testing to external verification through the close cooperation of the four modules, significantly improving the multi-classification diagnosis ability of laryngeal and hypopharyngeal lesions.

[0067] The training module provided by the embodiment of the present invention:

[0068] For the training of the model; its three types of modalities are laryngoscopy images, patient complaints, and clinical features; at the same time, according to its final diagnosis, it is labeled as the normal (Normal), non-cancerous lesion (Non Cancer), and cancer (Cancer) groups, and artificially scored as 0 points for Normal, 1.0 - 1.3 points for Non Cancer (its subgroups are 1.0 for benign lesions, 1.1 for low-grade precancerous lesions, and 1.3 for high-grade precancerous lesions), and 2.0 points for Cancer.

[0069] As Figure 2 shown, a method for constructing a multimodal deep learning model system for multi-class diagnosis of laryngeal and hypopharyngeal lesions provided by an embodiment of the present invention includes:

[0070] S101, using a laryngoscope device by the acquisition module to acquire laryngoscopy photos under white light imaging, and simultaneously inputting the corresponding patient's clinical features (gender and age) and medical history information;

[0071] S102, training the laryngoscopy photos by the training module; the laryngoscopy photos cover most lesions of the larynx and hypopharynx, not limited to the vocal cord level;

[0072] S103, testing the effect of the model on patients in the main central hospital by the testing module;

[0073] S104, using three medical institutions as external verification by the verification module.

[0074] Another object of the present invention is to provide a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for constructing a multimodal deep learning model system for multi-class diagnosis of laryngeal and hypopharyngeal lesions.

[0075] Another object of the present invention is to provide a computer-readable storage medium, storing a computer program, and when the computer program is executed by a processor, the processor executes the steps of the method for constructing a multimodal deep learning model system for multi-class diagnosis of laryngeal and hypopharyngeal lesions.

[0076] Another object of the present invention is to provide an information data processing terminal, the information data processing terminal is used to implement the multimodal deep learning model system for multi-class diagnosis of laryngeal and hypopharyngeal lesions.

[0077] The specific implementation of the present invention:

[0078] The multimodal deep learning framework integrates clinical features and outpatient medical histories into laryngoscope images taken in white light mode for detecting throat lesions. Finally, 1424 laryngoscope images of patients from Zhejiang First Hospital and their corresponding information (such as Figure 3 ) were used for model training. Its three modalities are laryngoscope images, patient complaints, and clinical features (i.e., gender, age). At the same time, according to its final diagnosis, it is labeled as the normal (Normal), non-cancerous lesion (Non Cancer), and cancer (Cancer) groups, and artificially scored as Normal 0 points, Non Cancer 1.0 - 1.3 points (its subgroup of benign lesions is 1.0, low-grade precancerous lesions is 1.1, high-grade precancerous lesions is 1.3), and Cancer 2.0 points.

[0079] BEIT3 is used as the backbone network for multimodal data perception, and this network exhibits good characteristics in modal feature extraction, modal interaction, and generalization. In terms of modal interaction, BEIT3 utilizes a unified multi-head self-attention layer shared by different modalities, thus endowing the model with the ability of multimodal information interaction. Moreover, a fusion module is designed to combine the patient's clinical features (gender and age) with the BEIT3 model. At the same time, using the contrastive learning framework of contrastive language-image pretraining (CLIP), sufficient contrast information of unlabeled multimodal data pairs is constructed in the embedding spaces of different modalities, thereby achieving the alignment between multimodal information. To further explore the cross-modal matching information in the multimodal dataset, this study uses CLIP as a pre-training task to enhance the model's ability to perform multimodal tasks. This study adopted supervised contrastive learning (SCL) on the patient-level feature embedding space Z to improve the model's performance. The above diagnostic model architecture for laryngeal and hypopharyngeal diseases based on multimodal information fusion and feature space contrast interaction is as Figure 4 .

[0080] Its performance was verified on separate internal and external datasets, and the process is shown in Figure 5 .

[0081] Several high-performance backbone models widely used in the fields of medicine and artificial intelligence were tested, including ResNet, Vit, Swin Transformer, and CLIP. In addition, text modality information was incorporated using BERT and compared with the proposed architecture. The diagnostic performance of the model on the internal validation set was the highest, with the accuracy, precision, recall, and F1 score reaching as high as 91.67%, 91.67%, 91.87%, and 91.82% respectively, as shown in Table 1.

[0082] The model was compared with endoscopists at different levels on internal and external validation sets. Generally, it could achieve the diagnostic ability of doctors at the Senior level with 310 years of laryngoscope working experience, and far exceeded that of doctors at the Expert level with 10 years of experience on the internal validation set, as shown in Table 2.

[0083] Figure 6 It shows that the model has a certain interpretability and can indicate the location of the lesion on the laryngoscope image. And it can avoid the interference of pharyngeal secretions ( Figure 6 a). Compared with human doctors, it can accurately judge the nature of the lesion on relatively blurred laryngoscope photos ( Figure 6 b), and can also make more accurate judgments on the interfering information provided in the medical history ( Figure 6 c).

[0084]

[0085] Table 1 Performance comparison of models with different modalities on the dataset

[0086]

[0087] Table 2 Performance comparison of the model with laryngology experts with different years of experience in the three-class classification task.

[0088] Note: Bold, underlined, and italic respectively indicate that the performance of the model exceeds that of experts, senior doctors, and junior doctors.

[0089] Example 1: Multi-classification diagnostic system for laryngeal and hypopharyngeal lesions based on Hospital A

[0090] In Hospital A, the multi-modal deep learning model of the present invention was deployed to assist otolaryngologists in diagnosing laryngeal and hypopharyngeal lesions. The system includes an acquisition module 1, a training module 2, a testing module 3, and a verification module 4, which are respectively deployed on different hardware devices to ensure efficient operation. The acquisition module 1 is installed on the laryngoscope device in the hospital, and it can collect the laryngoscope photos of patients in real time through white light imaging and transmit the image data to the training module 2 and the verification module 4.

[0091] ENT doctors in Hospital A use a high-resolution laryngoscope device to examine patients and collect laryngoscope photos under white light imaging. These photos cover various lesion types, including vocal cord edema, laryngeal cancer, laryngeal polyps, and hypopharyngeal ulcers. The acquisition module 1 preprocesses the images, such as denoising, contrast enhancement, and image cropping, to ensure that the image quality meets the requirements of training and diagnosis. At the same time, the corresponding patient clinical characteristics (gender and age) and medical history information are entered.

[0092] The training module 2 receives the preprocessed image data and uses a deep convolutional neural network (CNN) to extract features and classify laryngeal and hypopharyngeal lesions. The IT team in Hospital A regularly inputs newly collected data into the model for retraining to improve the model's recognition accuracy and adaptability. During the training process, the model ensures high accuracy and low false alarm rates on different lesion types through cross-validation and hyperparameter optimization.

[0093] The testing module 3 tests the trained model in the actual clinical environment of Hospital A. By diagnosing and testing the actual cases of other patients in Hospital A, the performance of the model in a real medical scenario is evaluated. The test results show that the model can accurately distinguish different types of laryngeal and hypopharyngeal lesions and assist doctors in making more accurate diagnostic decisions.

[0094] The verification module 4 compares and analyzes the diagnostic results of Hospital A with the data of three other collaborating medical institutions (Hospitals B, C, and D). Through cross-institutional data verification, the stability and consistency of the model in different devices and medical environments are confirmed. According to the verification results, the training module 2 is further optimized to ensure that the model maintains high performance in a wider range of application scenarios.

[0095] Embodiment 2: A multi-classification diagnosis system for laryngeal and hypopharyngeal lesions based on a telemedicine platform

[0096] On a telemedicine platform, the multi-modal deep learning model of the present invention is deployed to provide efficient laryngeal and hypopharyngeal lesion diagnosis support for primary medical institutions in remote areas. The system includes an acquisition module 1, a training module 2, a testing module 3, and a verification module 4, and realizes data transmission and collaborative work among the modules through cloud computing and network connection.

[0097] The primary medical institutions are equipped with networked laryngoscope devices, and the acquisition module 1 uploads the laryngoscope photos under white light imaging to the cloud server in real time through a wireless network. The uploaded data is encrypted to ensure patient privacy and data security. The cloud server performs preliminary preprocessing on the images, including image normalization and format conversion, to meet the processing requirements of the subsequent deep learning model.

[0098] The training module 2 runs in the cloud and uses large-scale laryngoscope image data from grass-roots medical institutions to train the model. Through distributed computing resources, the training module 2 can quickly process massive data and optimize the parameters and structure of the deep learning model. After the model training is completed, it is automatically deployed and updated to all connected grass-roots medical institutions to ensure that each node uses the latest diagnostic model.

[0099] The testing module 3 conducts real-time testing on the model in the cloud and evaluates the diagnostic effect of the model through simulated and actual case data. During the use process, doctors in grass-roots medical institutions can transmit the diagnostic results and clinical feedback back to the cloud. The testing module 3 continuously evaluates the model performance according to the feedback information and provides improvement suggestions. In this way, the model can continuously learn and improve in actual applications and adapt to the differences in different regions and devices.

[0100] The verification module 4 is connected to three different medical institutions (including Hospital A, Hospital B, and a grass-roots medical point). Through cross-institutional data comparison and analysis, it verifies the performance of the model in diverse medical environments. The verification results help identify the advantages and disadvantages of the model under different conditions and guide the training module 2 to conduct targeted optimization and adjustment to ensure the high accuracy and stability of the model in wide applications.

[0101] The telemedicine platform provides a user-friendly interface. Doctors in grass-roots medical institutions can upload laryngoscope photos through simple operation steps and obtain instant diagnostic results and suggestions. The system also provides detailed diagnostic reports and visual analysis to help doctors understand the judgment basis of the model and enhance the confidence and accuracy of diagnosis. At the same time, the platform supports multi-language and multi-device access to adapt to the usage needs of different regions.

[0102] The above two embodiments show the specific implementation methods of the present invention in different application scenarios. Embodiment 1 focuses on the integrated application within a single medical institution. Through the collaborative work of localized modules, it realizes efficient multi-classification diagnosis of laryngeal and hypopharyngeal lesions. Embodiment 2 extends to the telemedicine platform and uses cloud computing and network technologies to provide real-time and accurate diagnostic support for grass-roots medical institutions in remote areas. Both of these implementation methods give full play to the advantages of the multi-modal deep learning model of the present invention, significantly improve the efficiency and accuracy of laryngeal and hypopharyngeal lesion diagnosis, and meet the actual needs in different medical environments.

[0103] Figure 6 The laryngoscope images and outpatient text information of several external units are selected for effect demonstration, and the evaluation results of the model and the scoring results of doctors at all levels are also listed together to show its corresponding performance. Figure 6 The laryngoscope heat map of a shows that its attention can be more concentrated on the vocal cord area where the lesion is located (warm color enrichment), while some common pharyngeal secretions under the laryngoscope are cleverly avoided (no warm color enrichment).Figure 6 Figure b shows the evaluation result of a somewhat blurred laryngoscope image. The lesion in the right vocal cord area is of concern to the model (warm color enrichment), and a prediction result (predicted value of 1.97) close to the true pathological result (cancer, true value of 2) is given. Among 10 physicians of different levels, only 2 expert-level physicians gave an estimate of 2.0. Figure 6 In Figure c, the medical history gives a confounding piece of information - "The patient's previous biopsy record at another hospital was leukoplakia of the vocal cord", which led 5 out of 10 doctors to give a wrong benign diagnosis, while the model gave a prediction tending to malignancy (predicted value of 1.97). Of course, there are also situations where the model is more prone to error than humans. Figure 6 Figure d is a counterexample where the model makes an incorrect judgment.

[0104] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated designed hardware. Those of ordinary skill in the art can understand that the above devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code is provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and their modules of the present invention can be implemented by hardware circuits of programmable hardware devices such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above hardware circuits and software such as firmware.

[0105] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention shall be covered by the protection scope of the present invention.

Claims

1. A multi-modal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions, characterized in that, Comprising: A collection module, which includes an input device for collecting laryngoscope photos under white light imaging, and for recording patient age, gender, and medical history information; A training module, connected to the collection module through a data transmission interface, the training module receiving the laryngoscope photos, clinical feature data, and medical history text data; A testing module, connected to the training module through a data transmission interface, there being a data transfer device between the testing module and the training module; A verification module, connected to the collection module, training module, and testing module through a data transmission interface, the verification module receiving external data from different medical institutions.

2. The multimodal deep learning model according to claim 1, wherein The training module further includes: An image feature extraction unit, a clinical feature encoding unit, and a medical history text encoding unit; A feature alignment unit, which forms an integrated structure with the image feature extraction unit, clinical feature encoding unit, and medical history text encoding unit, and each unit is interconnected through a data transmission interface.

3. The multi-modal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions according to claim 1, wherein The training module includes: Three types of modalities, namely laryngoscope images, patient complaints, and clinical features; According to the final diagnosis, the data is marked as the normal (Normal), non-cancerous lesion (Non Cancer), and cancer (Cancer) groups; An artificial scoring system, where the normal group is marked as 0 points, the non-cancerous lesion group is marked as 1.0 to 1.3 points, and the cancer group is marked as 2.0 points, specifically subdivided into 1.0 point for benign lesions, 1.1 points for low-grade precancerous lesions, and 1.3 points for high-grade precancerous lesions.

4. The multimodal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions according to claim 1, characterized in that, The collection module further includes an image preprocessing unit for denoising, contrast enhancement, and image cropping of the collected laryngoscope photos.

5. The multimodal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions according to claim 1, characterized in that, The verification module is configured to use data from three different medical institutions for model verification.

6. The multi-modal deep learning model system for multi-class diagnosis of laryngeal and hypopharyngeal lesions according to claim 1, wherein The testing module is configured to apply the trained model to the actual patient data of primary hospitals to evaluate the diagnostic performance of the model.

7. A method for constructing a multimodal deep learning model system for multi-classification diagnosis of laryngeal and hypopharyngeal lesions implementing the multimodal deep learning model according to any one of claims 1 to 16, characterized in that, Comprising: Step 1, using the laryngoscope device by the collection module to collect laryngoscope photos under white light imaging, and simultaneously inputting the corresponding patient clinical features and medical history information; Step 2, training the laryngoscope photos through the training module; the laryngoscope photos cover most lesions of the larynx and hypopharynx, not limited to the vocal cord level; Step 3, testing the effect of the model on the patients in this central hospital through the testing module; Step 4, using three medical institutions as external verification through the verification module.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multi-modal deep learning model system construction method for multi-class diagnosis of laryngeal and hypopharyngeal lesions as claimed in claim 7.

9. A computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor executes the steps of the multi-modal deep learning model system construction method for multi-class diagnosis of laryngeal and hypopharyngeal lesions as claimed in claim 7.

10. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the multi-modal deep learning model system for multi-class diagnosis of laryngeal and hypopharyngeal lesions as claimed in any one of claims 16.

Images