Otolaryngology image intelligent recognition method based on multi-modal artificial intelligence

Abstract

The embodiment of the application discloses an otological image intelligent identification method based on a multi-modal artificial intelligence, and comprises the following steps: acquiring an otological image and clinical text information of a patient; fusing the otological image and the clinical text information to obtain multi-modal features; and identifying the multi-modal clinical text information features by using a multi-model integrated architecture to automatically determine the tympanic membrane state of the patient, wherein the tympanic membrane state comprises a normal tympanic membrane, a perforation, an inward protrusion, an effusion, a suppurative otitis media, calcification and otosclerosis. The embodiment realizes automatic intelligent identification of the tympanic membrane state.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical image processing technology, and in particular to an intelligent recognition method for otological images based on multimodal artificial intelligence. Background Technology

[0002] Various tympanic membrane perforations, retractions, effusions, purulent otitis media, tympanic membrane calcification, otosclerosis, and cerumen impaction are common otolaryngological diseases. Early and accurate identification of the eardrum condition is crucial for patient recovery. Currently, identification relies on manual interpretation of endoscopic images, and primary care facilities lack audiometry and soundproof rooms, exhibiting at least one of the following problems: First, there is a significant shortage of professional ENT specialists and poor equipment at the grassroots level, which can easily lead to missed or incorrect identifications. Second, manual identification is affected by experience level, resulting in poor consistency and low efficiency, which cannot meet the needs of large-scale screening. Third, existing technologies are mostly limited to single image recognition and do not integrate multimodal data such as clinical text, resulting in insufficient accuracy in recognizing early and minor lesions; Fourth, due to limitations in deployment equipment, existing AI recognition models struggle to balance recognition accuracy and model size. Models with high recognition accuracy tend to be large in size, making them difficult to deploy in grassroots equipment; while models with simple structures have limited recognition accuracy or are unable to comprehensively identify various lesion types. Summary of the Invention

[0003] This invention provides an intelligent otological image recognition method based on multimodal artificial intelligence to solve at least one of the above-mentioned problems.

[0004] In a first aspect, embodiments of the present invention provide an intelligent image recognition method for ear diseases based on multimodal artificial intelligence, comprising: Acquire otological images and clinical text information from patients; The otological images and clinical text information are fused to obtain multimodal features; The multi-model integrated architecture is used to identify the state features of the multi-model clinical text information and automatically determine the patient's tympanic membrane status, which includes normal tympanic membrane, perforation, retraction, effusion, purulent otitis media, calcification, and otosclerosis.

[0005] Secondly, embodiments of the present invention also provide an electronic device, the electronic device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent otology image recognition method based on multimodal artificial intelligence as described in any embodiment.

[0006] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent otology image recognition method based on multimodal artificial intelligence as described in any embodiment.

[0007] In summary, this embodiment provides an intelligent image recognition method for otology based on multimodal artificial intelligence. It integrates functions such as endoscopic data acquisition, storage, multimodal intelligent analysis, and diagnostic result output, enabling automatic identification of normal tympanic membranes and common lesions, improving recognition efficiency and accuracy, and filling the gap in recognition capabilities at the grassroots level. Specifically, the method of this embodiment can achieve the following effects: 1. Comprehensive identification coverage, accurately distinguishing between normal tympanic membranes and 6 types of core lesions, solving the problem of limited identification types in existing technologies; 2. Multimodal feature fusion improves recognition accuracy by 10%-15% compared to single image recognition, enhancing the accuracy of early, minor lesion identification; 3. By integrating multiple model network structures and making full use of the recognition advantages of various network models, it is possible to accurately distinguish between normal tympanic membranes and 6 types of core lesions. Through model pruning guided by the consistency of multi-model recognition, both recognition accuracy and model size are taken into account, making it easier to deploy the model in primary equipment and improving the applicability of the recognition method. 4. It achieves automated intelligent recognition of eardrum condition, which can solve the problems of large shortage of professional ENT personnel and poor equipment configuration at the grassroots level, which easily leads to missed recognition and false recognition, as well as the problems of poor consistency and low efficiency of manual recognition due to the influence of experience level, which cannot meet the needs of large-scale screening. Attached Figure Description

[0008] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0009] Figure 1 This is a flowchart of an intelligent otology image recognition method based on multimodal artificial intelligence provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a multi-model ensemble network provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0011] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0012] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0013] Figure 1 This is a flowchart illustrating an intelligent image recognition method for ear diseases based on multimodal artificial intelligence, provided by an embodiment of the present invention. This method can be applied to scenarios such as ENT clinical auxiliary recognition, primary healthcare screening recognition, or remote medical auxiliary recognition, achieving automated and precise tympanic membrane lesion recognition based on endoscopic ear images. This method is executed by an electronic device, such as... Figure 1 As shown, the specific steps include the following: S110. Obtain the patient's otological images and clinical text information.

[0014] Otological images include endoscopic images. Optionally, the aforementioned electronic device can be connected to the endoscopic device via wired / wireless means, supporting high-definition image acquisition with a resolution of ≥1920×1080, synchronously acquiring shooting parameters (focal length, exposure, shooting angle), and acquiring clinical text information such as patient age, gender, and symptom description as a data source for tympanic membrane status recognition.

[0015] After data collection is complete, a standardized otoscopy database can be built using a distributed storage architecture, comprising sub-libraries for image data, patient information, lesion annotation, and model training. Data is anonymized before being imported into the database, complying with medical data security standards. The database supports categorized retrieval and batch export, providing data support for subsequent model iterations.

[0016] S120. The otology images and clinical text information are fused to obtain multimodal features.

[0017] In one specific embodiment, the otological image can be preprocessed to extract visual features from the image. Optionally, the image can be processed by denoising, enhancement, size normalization, etc., to initially extract visual features such as texture, color, and shape.

[0018] Then, the clinical text information can be converted into gated attention features with the same channel dimension as the visual features, providing channel-level attention weights for the visual features to distinguish the importance of different channels in the image, thus avoiding direct modification of the image's texture structure. Optionally, the clinical text information can first be converted into embedded features using natural language processing techniques. Then, a gating network is used to convert the embedded features into a gating mask with the same channel dimension as the preprocessed image. : (1) in, Each element in the formula is a coefficient between 0 and 1, used to control which image channels are "strong" and which are "weak".

[0019] Finally, the visual features are fused with the gated attention features to obtain multimodal features. Optionally, the visual features are gated and weighted: (2) in, This indicates that the data is multiplied channel by channel, not added or stitched together, in order to preserve the texture information in the image.

[0020] S130. The multimodal features are identified using a multi-model integration architecture to automatically determine the patient's tympanic membrane status, which includes normal tympanic membrane, perforation, retraction, effusion, purulent otitis media, calcification, and otosclerosis.

[0021] To accommodate different tympanic membrane states and improve recognition accuracy, this embodiment employs a multi-model ensemble architecture to comprehensively determine the tympanic membrane state. For example, Inception V3, ResNet50, and Unet networks can be used to identify the tympanic membrane state separately, and then the tympanic membrane state is determined by combining the recognition results of each network. This multi-model parallel recognition and comprehensive judgment integration method can combine the advantages of multiple models, but in some cases, significant discrepancies and difficulties in reconciling the recognition results of different models may arise. Therefore, this embodiment improves upon the traditional multi-model ensemble architecture by proposing a serial-parallel combined multi-model ensemble method, which fully utilizes the advantages of various models while fundamentally ensuring the consistency of the results from each model.

[0022] In one specific implementation, the following can be used: Figure 2 The network structure shown first utilizes the parallel convolutional network of Inception V3 to extract global features of the multimodal features, achieving preliminary localization of the lesion region. Optionally, the complete Inception V3 network includes multiple parallel convolutional network branches and a classification layer. This embodiment omits the classification layer, inputting the multimodal features in parallel into each convolutional network branch for processing, and then concatenating the output features of each convolutional network branch by channel to obtain the global features of the multimodal features. The advantage of the Inception V3 network is its ability to capture the global characteristics of the image, enabling preliminary localization of the lesion region.

[0023] Then, this embodiment improves the multi-stage residual block group of ResNet50 (or ResNet with more convolutional layers) into a parallel structure, extracting deep texture features of the global features in parallel under different receptive fields to reflect lesion features at different scales. Traditional ResNet50 multi-stage residual block groups are serial (sequentially from Stage 1 to Stage N, with the receptive field gradually increasing and the output dimension gradually decreasing). This embodiment, in order to utilize its residual structure advantages and take into account the scale tendency of different lesion features, improves the originally serial multi-stage residual block group into a parallel structure. Under the guidance of global features, it further extracts deep texture features under different receptive fields, maintaining consistency with the global features of Inception V3 on the one hand, and fully extracting deep features using the residual structure on the other. Optionally, since the input dimensions of each stage residual block group are inconsistent, a lightweight feature transformation network can be used to convert the global features into dimensions suitable for each stage residual block group. Optionally, the feature transformation network can sample within the convolutional plane through local pooling operations to achieve dimensionality transformation within the convolutional plane, and achieve channel number transformation through 1*1*channel number convolution.

[0024] Subsequently, combined Figure 2The global features and deep texture features under different receptive fields are input into encoders of different scales in Unet. The output of each encoder is concatenated with the corresponding deep texture feature channel and then fed into the next encoder layer. The output of each encoder is then connected to the corresponding decoder in Unet, concatenated with the output channel of the next decoder layer, and decoded. The top decoder outputs the fine segmentation features of the tympanic membrane state. The Unet model can achieve end-to-end output, providing pixel-level segmentation of minute lesions. Therefore, this embodiment uses the deep texture features extracted by the ResNet50 network and further segmentes them using the Unet model to finally output the fine segmentation features of the tympanic membrane state. Optionally, the encoder of the Unet model can use a regular convolutional network or a MobileNetV2 network to ensure smooth operation between the encoder and decoder. Each encoder is used to perform downsampling and upsampling operations. The output of the global features is directly input into encoder 1. The output of the residual block group of Stage 1-Stage N in the ResNet network is concatenated with the output of the previous encoder layer along the channel dimension and then used as the input of encoder 2-Encoder N+1. Encoder N+1 is the top-level encoder. Its output can be the same as the global features or the same as the visual feature dimension of the entire network input. The output includes the eardrum state type and confidence level of each pixel. According to the scale correspondence, it can be converted into the eardrum state type and confidence level of each pixel in the preprocessed image.

[0025] Finally, the global features, deep texture features under different receptive fields, and fine segmentation features are fused to comprehensively evaluate the tympanic membrane state type and confidence level. Optionally, feature fusion can be performed by feature splicing, and the fused features are input into a classification layer such as an MLP (i.e., the recognition layer in the figure) to obtain the final recognition result. This result can include the confidence level of the patient's tympanic membrane belonging to one of the seven states: normal tympanic membrane, perforation, retraction, effusion, purulent otitis media, calcification, and otosclerosis.

[0026] As described above, to ensure recognition accuracy, this embodiment constructs a novel multi-model fusion architecture based on three network prototypes: Inception V3, ResNet50, and Unet, to fully capture the features of different lesions. Under this architecture, to avoid inconsistent recognition results among the models, this embodiment improves the traditional parallel multi-model recognition to a combined serial and parallel approach. Based on the advantages of the three models in feature extraction, a serial branch is constructed: Inception V3 -> ResNet50 -> Unet. This ensures deep texture feature extraction and refined segmentation guided by global features, avoiding deviation from the global localization direction of lesion features. Simultaneously, this embodiment improves the multi-stage serial residual blocks of ResNet into a parallel structure. This allows for integration with Unet's multi-layer encoder, where the residual block group and encoder jointly extend the feature extraction depth at each scale, while retaining the advantages of the residual algorithm and accommodating the scale tendencies of different lesion features. Finally, the features extracted based on the three network prototypes are fused in parallel, further leveraging the advantages of multi-model fusion while ensuring consistency.

[0027] Furthermore, it can be seen that the above network structure fully utilizes the recognition advantages of various model structures, resulting in a relatively complex and large network structure. Therefore, this embodiment proposes a model pruning method. After the model training is complete, while ensuring recognition accuracy, the network structure of subsequent stages is simplified through the guidance of global features. In a specific implementation, this process may include the following steps: Step 1: Calculate the similarity between the output of each ResNet residual block group and the global feature, and prune residual block groups with similarity less than the pruning threshold. In this embodiment, after model training is complete, a batch of samples is used to determine the pruning range. Combined with... Figure 2Since global features are the foundation for subsequent feature extraction, this embodiment does not prune the Inception V3 network structure. Instead, based on the consistency of feature extraction, it analyzes whether each subsequent processing branch has learned information that matches the guiding direction of the global features, taking each residual block group (e.g., ResNet50's stage1, stage2, stage3, and stage4) in parallel as a unit. Optionally, feature similarity can be used to measure the degree of deviation of each residual block group from the guiding direction of the global features. Taking residual block group stage1 as an example, the similarity between the output of residual block group stage1 and the global features is calculated. The greater the similarity, the more consistent the features learned by residual block group stage1 are with the guiding direction of the global features, which conforms to the feature consistency design of the network and should be retained during pruning. The smaller the similarity, the more serious the deviation of the features learned by residual block group stage1 from the guiding direction of the global features, which violates the feature consistency goal of the entire network, and should be pruned first. Similarly, the similarity between the output of residual block group stage2 and the global features, the similarity between the output of residual block group stage3 and the global features, and the similarity between the output of residual block group stage4 and the global features are calculated respectively. If a certain similarity is lower than the pruning threshold, the residual block corresponding to that similarity is pruned.

[0028] Furthermore, when calculating similarity, if the output dimension of the residual block is the same as the dimension of the global feature, feature similarity, such as cosine similarity or Euclidean distance, can be calculated directly. However, if the output of the residual block differs from the dimension of the global feature, the dimension of the residual block's output in the two-dimensional plane can first be transformed to match the global feature using linear interpolation or average pooling. The transformed output of the residual block can be denoted as... Then, the Xavier uniform initialization method is used to generate... The weight matrix, where, This indicates the channel dimension (i.e., the number of channels) of the output of the residual block group. The channel dimension represents the global feature; the weight matrix is ​​used to... The vector channel transformation at each two-dimensional plane position is consistent with the global feature, while maintaining the variance information of the transformed data. The output of the residual block group after channel transformation can be denoted as... After the above transformation, The dimension of the vector is completely consistent with the dimension of the global feature, thus allowing the calculation of their vector similarity. It should be noted that the dimension transformation operation here is only used to determine the pruning range and does not participate in the model application phase. Therefore, the parameters do not require special training. As long as the data information of the feature remains as unchanged as possible before and after the transformation, this embodiment uses average pooling, linear interpolation, and Xavier uniform initialization to maintain the stability of data information diffusion or compression, reflecting as realistically as possible whether the data information learned by the residual block group deviates from the direction of the global feature data information.

[0029] Step 2: Prune the portions of the residual block groups that inherit the features from the pruned residual block groups in each encoder and decoder of UNet. The remaining network structure is used to perform subsequent multimodal feature recognition. After pruning residual block groups with low similarity, the portions of the corresponding encoders that involve the output of the residual block groups will also be pruned (e.g., by setting the mask to 0), thereby reducing computation and model size. For example, if residual block group Stage1 is pruned, the data in encoder 2 that originally represented the output of Stage1 will all be set to 0 by masking. Since the scale of the ear lesion features to be identified cannot be determined in advance in this embodiment, depth texture features of all scales are retained in ResNet. Here, pruning guided by global features deletes branches that deviate from the direction of global feature guidance, reducing computation and eliminating interference from redundant branches, achieving a good balance between model size and recognition accuracy.

[0030] Step 3: Utilize the retained network structure to continue multimodal feature recognition, obtaining new fine segmentation features. If the segmentation similarity between the new fine segmentation features and the annotated tympanic membrane lesion image is lower than a preset threshold, reverse-locate the portion of the pruned structure related to the decrease in segmentation similarity, restore the function of that portion, and lower the pruning threshold of the residual block group corresponding to that portion. This step uses a batch of new annotated samples to verify the segmentation effect of the pruned image. The fine segmentation features output by the Unet network can be restored to a segmentation map of the same size as the patient's otoscope image through scaling transformation, etc. This segmentation map is compared with the annotated sample segmentation map, for example, by calculating the Dice coefficient. If the segmentation similarity is lower than a preset threshold (e.g., 85%), it is determined as "over-pruning". At this point, the parts of the pruned structure that significantly impact segmentation similarity can be traced back. Optionally, the entire pruned structure can be restored, and the restored structure can be divided into several smaller parts (e.g., each pruned residual module is a part, each pruned operation in the encoder is a part, and each pruned operation in the decoder is a part). Each part is masked using a local mask, and the part with the largest reduction in segmentation similarity after masking is taken as the result of reverse localization. The function of this part and its corresponding residual block group is restored, and the pruning threshold of the corresponding residual block group is lowered. For example, if the pruned operation in encoder 2 is the localization result, the feature mask is set to 1 to restore this part and the corresponding residual block group stage1, and the pruning threshold of residual block group stage1 is lowered. In this step, the pruning threshold of each residual block group is adjusted differentially through pruning feedback. While maintaining the global feature guidance direction, the guiding algorithm retains as many residual block groups as possible that have a significant impact on segmentation accuracy, achieving adaptive judgment of branch importance.

[0031] Step 4: Recalculate the similarity between the output of each residual block group after recovery and the global feature, and prune the residual block groups with similarity less than their respective pruning thresholds, as well as the corresponding portions in each encoder and decoder. This step then uses a new batch of samples to prune again, and redetermines the pruning range based on the differential pruning threshold. At this point, the over-pruned portions may be retained.

[0032] Step 5: Return to the previous step and continue performing multimodal feature recognition using the retained network structure to obtain new fine-grained segmentation features until the segmentation similarity reaches a preset threshold, completing the pruning feedback correction. Repeat this process, and through repeated verification with a large number of samples, obtain a pruning scheme that guarantees segmentation similarity. The pruning thresholds determined for each residual block group at this point can also be used as preferred initial values ​​in subsequent model update pruning.

[0033] Furthermore, after automatically determining the patient's tympanic membrane status by identifying the multimodal features using a multi-model ensemble architecture, a pixel-level recognition map of the tympanic membrane status can be displayed. Each pixel represents the tympanic membrane status, including normal tympanic membrane, perforation, retraction, effusion, purulent otitis media, calcification, or otosclerosis, along with its confidence score (i.e., a vector composed of seven confidence scores). Optionally, a visual interface can be used to display the original otoscopy image, the lesion area annotation map (obtained from the output of the Unet network), the recognition results, and diagnostic suggestions, supporting electronic report generation, printing, and delivery to the doctor's workstation.

[0034] Furthermore, as the database accumulates, when patient data reaches a certain scale, new samples can be used to iteratively update the complete recognition model. After the update, the model is pruned again and redeployed, maintaining the dynamic updating of model parameters and structure with the data. The re-pruning process can utilize the differentiated pruning thresholds optimized in the previous part-time job, improving the rationality of the pruning.

[0035] To verify the effectiveness of the method in this embodiment, clinical otoscopy images were collected, including images of normal tympanic membranes, various lesions, and their corresponding audiograms and diagnostic texts. These images were double-blindly annotated by three senior otolaryngologists and divided into training, validation, and test sets in an 8:1:1 ratio. A data processing module was then developed using Python, and a multimodal ensemble model was trained using the TensorFlow framework. The database employed a hybrid architecture of MySQL and MongoDB, and a visual interactive interface was developed using Vue.js. Test set validation results showed that the model achieved an average recognition accuracy of 96.2% for six types of lesions, with an area under the ROC curve (AUC) ≥0.98, and diagnostic consistency with senior physicians exceeding 92%. Simultaneously, clinical trials were conducted in three primary hospitals, processing a total of 1000 otoscopy images. The diagnostic efficiency was improved by three times compared to manual methods, and the missed diagnosis rate was reduced to 1.2%. This fully demonstrates the effectiveness of the method in this embodiment.

[0036] In summary, this embodiment provides an intelligent image recognition method for otology based on multimodal artificial intelligence. It integrates functions such as endoscopic data acquisition, storage, multimodal intelligent analysis, and diagnostic result output, enabling automatic identification of normal tympanic membranes and common lesions, improving recognition efficiency and accuracy, and filling the gap in recognition capabilities at the grassroots level. Specifically, the method of this embodiment can achieve the following effects: 1. Comprehensive identification coverage, accurately distinguishing between normal tympanic membranes and 6 types of core lesions, solving the problem of limited identification types in existing technologies; 2. Multimodal feature fusion improves recognition accuracy by 10%-15% compared to single image recognition, enhancing the accuracy of early, minor lesion identification; 3. By integrating multiple model network structures and making full use of the recognition advantages of various network models, it is possible to accurately distinguish between normal tympanic membranes and 6 types of core lesions. Through model pruning guided by the consistency of multi-model recognition, both recognition accuracy and model size are taken into account, making it easier to deploy the model in primary equipment and improving the applicability of the recognition method. 4. It achieves automated intelligent recognition of eardrum condition, which can solve the problems of large shortage of professional ENT personnel and poor equipment configuration at the grassroots level, which easily leads to missed recognition and false recognition, as well as the problems of poor consistency and low efficiency of manual recognition due to the influence of experience level, which cannot meet the needs of large-scale screening.

[0037] It should be noted that all user data involved in this application is information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0038] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the device includes a processor 60, a memory 61, an input device 62, and an output device 63; the number of processors 60 in the device can be one or more. Figure 3 Taking a processor 60 as an example; the processor 60, memory 61, input device 62, and output device 63 in the device can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0039] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the intelligent otological image recognition method based on multimodal artificial intelligence in this embodiment of the invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 61, thereby realizing the aforementioned intelligent otological image recognition method based on multimodal artificial intelligence.

[0040] The memory 61 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 61 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include memory remotely located relative to the processor 60, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0041] Input device 62 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 63 may include display devices such as a display screen.

[0042] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the otology image intelligent recognition method based on multimodal artificial intelligence of any embodiment.

[0043] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0044] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0045] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0046] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent recognition of otological images based on multimodal artificial intelligence, characterized in that, include: Acquire otological images and clinical text information from patients; The otological images and clinical text information are fused to obtain multimodal features; The multi-model integration architecture is used to identify the multimodal features and automatically determine the patient's tympanic membrane status, which includes normal tympanic membrane, perforation, retraction, effusion, purulent otitis media, calcification, and otosclerosis.

2. The intelligent recognition method for otological images based on multimodal artificial intelligence as described in claim 1, characterized in that, The fusion of the otological images and clinical text information to obtain multimodal features includes: Extract visual features from the ear images; The clinical text information is converted into gated attention features with the same dimension as the visual feature channels; Based on the gated attention features, the visual features are subjected to channel attention weighting to obtain multimodal features.

3. The intelligent otological image recognition method based on multimodal artificial intelligence as described in claim 1, characterized in that, The method of using a multi-model ensemble architecture to identify the multimodal features and automatically determine the patient's tympanic membrane status includes: The parallel convolutional network of Inception V3 is used to extract global features of the multimodal features to achieve preliminary localization of the lesion area; The multi-stage residual block group of ResNet is converted into a parallel structure, and the deep texture features of the global features under different receptive fields are extracted in parallel to reflect the lesion features at different scales. The global features and deep texture features under different receptive fields are respectively input into encoders of different scales in Unet. The output of each encoder is concatenated with the corresponding deep texture feature channel and then fed into the next encoder layer. The output of each encoder is connected to the corresponding scale decoder in Unet, and after being concatenated with the output channel of the next layer decoder, it is decoded, and the fine segmentation features of the tympanic membrane state are output by the top layer decoder. The global features, deep texture features under different receptive fields, and fine segmentation features are fused to comprehensively evaluate the tympanic membrane state type and confidence level.

4. The intelligent recognition method for otological images based on multimodal artificial intelligence according to claim 3, characterized in that, After identifying the multimodal features using a multi-model ensemble architecture and automatically determining the patient's tympanic membrane state, the method further includes: Calculate the similarity between the output of each ResNet residual block group and the global feature, and prune residual block groups with similarity less than the pruning threshold. The residual block features in each encoder and decoder of Unet are partially pruned, and the remaining network structure is used to perform subsequent multimodal feature recognition.

5. The intelligent recognition method for otological images based on multimodal artificial intelligence according to claim 4, characterized in that, The step of calculating the similarity between the output of each ResNet residual block group and the global feature, and pruning residual block groups with similarity less than the pruning threshold, includes: When the output of the residual block group is different from the dimension of the global feature, the dimension of the output of the residual block group in the two-dimensional plane is transformed to be consistent with the dimension of the global feature by linear interpolation or average pooling. Generate using Xavier uniform initialization method The weight matrix, where, and These represent the output of the residual block group and the channel dimension of the global feature, respectively. The weight matrix is ​​used to transform the vector channel at each two-dimensional plane position in the output of the transformed residual block group into one that is consistent with the global feature, while keeping the variance information of the transformed data unchanged. Calculate the similarity between the output of the residual block group after channel transformation and the global feature.

6. The intelligent recognition method for otological images based on multimodal artificial intelligence according to claim 4, characterized in that, After partially pruning the residual block group features that are being pruned in each encoder and decoder of the Unet, the method further includes: The retained network structure is used to continue multimodal feature recognition, resulting in new, refined segmentation features; If the segmentation similarity between the new fine segmentation feature and the tympanic membrane lesion annotation image is lower than a preset threshold, the part of the pruned structure related to the decrease in segmentation similarity is located in reverse, the function of the part is restored, and the pruning threshold of the residual block group corresponding to the part is lowered. The similarity between the output of each residual block group after recovery and the global feature is recalculated, and the residual block groups with similarity less than their respective pruning thresholds, as well as the corresponding parts in each encoder and decoder, are pruned. Returning to the operation of using the retained network structure to continue multimodal feature recognition and obtain new fine segmentation features, until the segmentation similarity reaches a preset threshold, the pruning feedback correction is completed.

7. The intelligent recognition method for otological images based on multimodal artificial intelligence according to claim 3, characterized in that, Unet's encoder uses the MobileNetV2 architecture.

8. The intelligent recognition method for otological images based on multimodal artificial intelligence according to claim 1, characterized in that, After automatically determining the patient's tympanic membrane state by identifying the multimodal features using a multi-model ensemble architecture, the method further includes: The image displays a pixel-level identification of the tympanic membrane condition. Each pixel represents one of the following conditions: normal tympanic membrane, perforation, retraction, effusion, purulent otitis media, calcification, otosclerosis, and its confidence level.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent otology image recognition method based on multimodal artificial intelligence as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the intelligent otology image recognition method based on multimodal artificial intelligence as described in any one of claims 1-8.

Images