Abnormal feature extraction method and device for otology multi-modal data and medium

Abstract

The invention discloses an abnormal feature extraction method and device for otology multi-mode data and a medium, and the method comprises the steps: carrying out the abnormal feature extraction of the otology multi-mode data of a user according to a comprehensive model, and obtaining the abnormal features of the otology data; wherein the comprehensive model is obtained by training the initial model according to positive and negative sample pairs and pseudo labels, and the positive and negative sample pairs are generated by pairing joint representation vectors of different users. According to the abnormal feature extraction method and device for otology multi-modal data and the medium, the combined representation vector is generated to integrate the image and the text data so as to break the modal barrier, and the positive and negative sample pairs and the pseudo labels are combined to train the initial model, so that the obtained comprehensive model can master the multi-modal association logic, and the accuracy of the abnormal feature extraction of the otology multi-modal data is improved. The abnormal features can be directly and efficiently extracted, and the problems that the abnormal feature extraction efficiency is low and the rapid processing requirement of the otology multi-modal data is difficult to meet due to the lack of an analysis mechanism for efficiently integrating the otology multi-modal data can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical imaging, in particular to an otology multi-modal data abnormal feature extraction method, device and medium. BACKGROUND

[0002] In otology clinical practice, otology image data, otology text data and function-related data and other multi-modal data are generated. These data have significant differences in information dimensions and expression forms, and complex internal relationships exist among them. In clinical scenarios, there is a clear need for efficient processing of otology multi-modal data and accurate extraction of abnormal features. The results can directly serve clinical information integration analysis, subsequent operation scheme reference generation and other links. Therefore, constructing an effective otology multi-modal data abnormal feature detection mechanism is of great significance to improve the efficiency of clinical data processing and strengthen the clinical application value of multi-modal information. In the feature extraction stage, the existing technology generally follows the single-modal feature independent extraction logic. Specifically, image features rely on CNN, ResNet and other models to extract spatial structure information, and text features rely on TF-IDF, LSTM and other technologies to extract semantic information. In the modal fusion link, the existing technology mostly adopts a shallow fusion strategy, which has two ways: one is to directly concatenate the feature vectors of each modality, and the other is to only vote or calculate the mean value of the independent detection results of each modality.

[0003] However, the existing technology adopts single-modal independent extraction logic in the feature extraction stage, does not establish cross-modal correlation, and cannot capture the internal relationship of the data. In the modal fusion link, it relies on a shallow strategy to integrate data by feature concatenation or detection result voting and mean value calculation, which cannot dynamically adapt to feature importance and is difficult to mine data complementary value. This limitation makes the whole lack of an analysis mechanism for efficiently integrating otology multi-modal data, ultimately resulting in low efficiency of abnormal feature extraction and difficulty in meeting the clinical demand for rapid processing of otology multi-modal data. SUMMARY

[0004] The present application provides an otology multi-modal data abnormal feature extraction method, device and medium to solve the problem of lack of an analysis mechanism for efficiently integrating otology multi-modal data, resulting in low efficiency of abnormal feature extraction and difficulty in meeting the demand for rapid processing of otology multi-modal data.

[0005] To achieve the above-mentioned purpose, the present application provides an otology multi-modal data abnormal feature extraction method, which comprises:

[0006] Obtaining otology multi-modal data of a user;

[0007] According to the comprehensive model, the otological multi-modal data is subjected to abnormal feature extraction, and otological data abnormal features are obtained; wherein, the comprehensive model is obtained by multi-stage training of an initial model according to positive and negative sample pairs and pseudo label pairs, the positive and negative sample pairs are generated by pairing joint representation vectors of different users, and the joint representation vectors are obtained by feature extraction on historical otological image data and otological text data; the pseudo label is generated by generating a joint representation vector from historical otological multi-modal data without labeled lesion information, and filtering the prediction result with a confidence greater than a preset value by the trained initial model; and the initial model is a multi-modal generative AI system based on a Transformer architecture.

[0008] According to the comprehensive model, the otological multi-modal data is subjected to abnormal feature extraction, and otological data abnormal features are obtained; wherein, the comprehensive model is obtained by multi-stage training of an initial model according to positive and negative sample pairs and pseudo label pairs, the positive and negative sample pairs are generated by pairing joint representation vectors of different users, and the joint representation vectors are obtained by feature extraction on historical otological image data and otological text data; the pseudo label is generated by generating a joint representation vector from historical otological multi-modal data without labeled lesion information, and filtering the prediction result with a confidence greater than a preset value by the trained initial model; and the initial model is a multi-modal generative AI system based on a Transformer architecture.

[0009] According to the comprehensive model, the otological multi-modal data is subjected to abnormal feature extraction, and otological data abnormal features are obtained; wherein, the comprehensive model is obtained by multi-stage training of an initial model according to positive and negative sample pairs and pseudo label pairs, the positive and negative sample pairs are generated by pairing joint representation vectors of different users, and the joint representation vectors are obtained by feature extraction on historical otological image data and otological text data; the pseudo label is generated by generating a joint representation vector from historical otological multi-modal data without labeled lesion information, and filtering the prediction result with a confidence greater than a preset value by the trained initial model; and the initial model is a multi-modal generative AI system based on a Transformer architecture.

[0010] As a preferred solution, the joint representation vector is obtained by feature extraction on historical otological image data and otological text data, specifically:

[0011] obtain historical otology image data and historical otology text data of a plurality of users;

[0012] extract image feature vectors from the historical otology image data through a ConvNeXt comprehensive model;

[0013] encode the historical otology text data and extract structured features to obtain text feature vectors;

[0014] fuse the image feature vectors and the text feature vectors according to a preset attention mechanism to obtain the joint representation vector.

[0015] The preferred scheme extracts image features through a ConvNeXt comprehensive model to obtain image feature vectors, encodes text data to obtain text feature vectors, and fuses the two features using an attention mechanism to generate a joint representation vector, effectively solving the modal difference problem of otology image and text data. Moreover, the introduction of the attention mechanism enables dynamic focusing on the relevance features of the image and the text, improving the relevance and effectiveness of multi-modal data fusion, providing more comprehensive and accurate basic feature support for subsequent abnormal feature extraction and diagnosis, and enhancing the recognition ability of the model for conductive hearing loss related features.

[0016] As a preferred scheme, the image feature vectors and the text feature vectors are fused according to a preset attention mechanism to obtain the joint representation vector, specifically:

[0017] a query matrix of image features is established according to the image feature vectors, and a key matrix and a value matrix of text features are established according to the text feature vectors;

[0018] based on a Query-Key-Value attention mechanism, attention weights are calculated according to the query matrix and the key matrix, and weighted summation is performed in combination with the value matrix, and at the same time, the region features in the image feature vectors and the corresponding semantic descriptions in the text feature vectors are semantically aligned to calculate the joint representation vector.

[0019] The preferred scheme calculates the relationship between the image query matrix and the text key-value matrix, and the model can adaptively assign attention weights to highlight the feature information that is critical for diagnosis. The semantic alignment operation further eliminates the semantic gap between the image regions and the text descriptions, ensuring the accurate correspondence between the image lesion regions and the text diagnostic descriptions. This mechanism enables the joint representation vector to not only retain the spatial details of the image, but also fuse the clinical semantics of the text, providing higher quality feature input for subsequent model training and abnormal recognition, and improving the accuracy and reliability of auxiliary diagnosis.

[0020] As a preferred solution, the pseudo label is generated according to a joint representation vector of historical ear multi-modal data without labeled lesion information, and is generated by filtering a prediction result with a confidence greater than a preset value through an initial model that has been trained, specifically as follows:

[0021] Based on historical ear multi-modal data without labeled abnormal technical features or lesion-related information, a first joint representation vector is generated according to a ConvNeXt comprehensive model, the first joint representation vector is input into an initial model pre-trained based on the positive and negative sample pairs, and an abnormal prediction result set containing a confidence is output by the model;

[0022] An abnormal prediction result with a confidence greater than a preset value in the abnormal prediction result set is defined as the pseudo label.

[0023] This preferred solution uses a pre-trained model to predict the joint representation vector of unlabeled data, and filters the result with a confidence higher than a preset value as a pseudo label, which not only expands the training sample size, but also ensures the reliability of the pseudo label through the confidence threshold, avoiding the interference of low-quality labels on model training. This semi-supervised learning strategy can fully exploit the implicit abnormal information in unlabeled data, reduce the dependence on large-scale labeled data, and is particularly suitable for scenarios with high clinical data labeling costs and scarce samples, which can enhance the generalization ability and practical clinical application value of the model.

[0024] As a preferred solution, the comprehensive model is obtained by multi-stage training of an initial model based on positive and negative sample pairs and pseudo labels, specifically as follows:

[0025] Based on a preset loss function, the initial model is trained by contrastive learning based on the positive and negative sample pairs, to obtain a pre-trained initial model; wherein the loss function optimizes the initial model by narrowing the distance of positive sample features and widening the distance of negative sample features;

[0026] The pseudo label and the historical ear multi-modal data without labeled lesion information are combined into new samples, which are expanded to a training set containing the historical ear image data and ear text data, to obtain an expanded training set, and the pre-trained initial model is trained again based on the expanded training set to obtain the comprehensive model.

[0027] The preferred scheme adopts a multi-stage training strategy, first performs contrast learning through positive and negative sample pairs, and then performs secondary training using a training set expanded by pseudo labels, thereby significantly improving the diagnostic performance of the comprehensive model. Contrast learning narrows the distance between positive samples and widens the distance between negative samples, enabling the model to more accurately distinguish between normal and abnormal features, thereby laying a good foundation for feature discrimination. Secondary training using a training set expanded by pseudo labels further utilizes the effective information in unlabeled data, thereby refining the model's learning of complex lesion characteristics. The multi-stage training enables the model to first acquire basic discrimination ability and then optimize detail recognition accuracy, thereby ensuring the stability of the model and improving the model's ability to capture subtle abnormalities related to conductive hearing loss, thereby ultimately improving the accuracy and robustness of auxiliary diagnosis.

[0028] As a preferred scheme, the ConvNeXt comprehensive model is established in the following manner:

[0029] Based on the preset ConvNeXt model, the ConvNeXt comprehensive model is generated by integrating a ConvNeXt Block module, a Downsample module, and a CBAM attention module.

[0030] The ConvNeXt Block module is established based on depthwise separable convolution and layer normalization, the Downsample module is established by downsampling the feature map, and the CBAM attention module is established by dynamically adjusting the feature weights through channel attention and spatial attention.

[0031] The preferred scheme integrates the ConvNeXt Block, Downsample, and CBAM attention modules to construct a comprehensive model, thereby fully leveraging the technical advantages of each module and improving the efficiency and accuracy of otic image feature extraction. The ConvNeXt Block is based on depthwise separable convolution and layer normalization, which reduces computational complexity while enhancing feature extraction capability. The Downsample module effectively reduces the dimensionality of the feature map through downsampling, thereby improving model operation efficiency. The CBAM attention module dynamically adjusts the feature weights of the channel and spatial dimensions, thereby enabling the model to focus on key regions related to conductive hearing loss in otic images. The synergistic effect of the three modules enables the model to efficiently process multi-modal image data and accurately capture lesion characteristics, thereby providing high-quality image feature input for subsequent multi-modal fusion and abnormality recognition and enhancing the clinical applicability of the model.

[0032] The present application also provides an abnormal feature extraction device for otic multi-modal data, comprising a data module and an extraction module.

[0033] The data module is configured to obtain otic multi-modal data of a user.

[0034] The extraction module is configured to extract abnormal features from the otological multi-modal data according to a comprehensive model to obtain otological data abnormal features; wherein the comprehensive model is obtained by training an initial model in multiple stages according to positive and negative sample pairs and pseudo-label pairs, the positive and negative sample pairs are generated by pairing joint representation vectors of different users, and the joint representation vectors are obtained by extracting features from historical otological image data and otological text data; the pseudo-labels are generated by generating joint representation vectors from historical otological multi-modal data without labeled lesion information, and filtering the prediction results with a confidence greater than a preset value by the trained initial model; and the initial model is a multi-modal generative AI system based on a Transformer architecture.

[0035] As a preferred solution, the extraction module includes a data unit, an image unit, a text unit, and a fusion unit.

[0036] The data unit is configured to obtain historical otological image data and historical otological text data of a plurality of users.

[0037] The image unit is configured to extract features from the historical otological image data by a ConvNeXt comprehensive model to obtain an image feature vector.

[0038] The text unit is configured to encode the historical otological text data and extract structured features to obtain a text feature vector.

[0039] The fusion unit is configured to fuse the image feature vector and the text feature vector according to a preset attention mechanism to obtain the joint representation vector.

[0040] As a preferred solution, the fusion unit includes a matrix sub-unit and a vector sub-unit.

[0041] The matrix sub-unit is configured to establish a query matrix of image features according to the image feature vector, and establish a key matrix and a value matrix of text features according to the text feature vector.

[0042] The vector sub-unit is configured to calculate attention weights according to the query matrix and the key matrix based on a Query-Key-Value attention mechanism, and perform weighted summation by combining the value matrix, while performing semantic alignment between region features in the image feature vector and corresponding semantic descriptions in the text feature vector to calculate the joint representation vector.

[0043] The application also provides a storage medium having a computer program stored thereon, wherein the computer program is invoked and executed by a computer to implement the otological multi-modal data abnormal feature extraction method described above. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flow diagram of an ear multi-modal data abnormal feature extraction method provided by an embodiment of the present application;

[0045] Figure 2 is a CBAM attention module schematic diagram provided by an embodiment of the present application;

[0046] Figure 3 is a ConvNeXt model structure schematic diagram based on a CBAM attention mechanism provided by an embodiment of the present application;

[0047] Figure 4 is a structure schematic diagram of an ear multi-modal data abnormal feature extraction device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0049] In the description of the present application, it should be understood that the term "first" is only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features limited by "first" can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise specified, the meaning of "several" is two or more.

[0050] The ear multi-modal data abnormal feature extraction method provided by the embodiments of the present application aims to solve the problem that the existing AI auxiliary analysis technology is mostly based on CT images in part of the image data, lacks deep fusion capability of multi-modal data, and the model generalization is insufficient, so it is difficult to efficiently extract abnormal features to meet the ear multi-modal data rapid processing requirement.

[0051] Embodiment one:

[0052] Please refer to Figure 1 The embodiments of the present application provide an ear multi-modal data abnormal feature extraction method, which includes S1-S2, and the specific implementation steps are as follows:

[0053] S1, obtaining ear multi-modal data of a user.

[0054] The step S1 of the embodiments of the present application is specifically:

[0055] Obtaining otological multi-modal data of a user, including but not limited to: patient CT images, endotoscopic images, audiological reports, and simultaneously obtaining otologically related text information in the user's clinical otological records, to provide a complete data source for subsequent multi-modal data fusion analysis.

[0056] S2, extracting abnormal features of the otological multi-modal data according to the comprehensive model, to obtain otological data abnormal features; wherein the comprehensive model is obtained by training an initial model in multiple stages according to positive and negative sample pairs and pseudo-label pairs, the positive and negative sample pairs are generated by pairing joint representation vectors of different users, the joint representation vectors are obtained by feature extraction on historical otological image data and otological text data; the pseudo-label is generated by generating a joint representation vector from historical otological multi-modal data without labeled lesion information, and filtering the prediction results with a confidence greater than a preset value by the trained initial model; the initial model is a multi-modal generative AI system based on a Transformer architecture.

[0057] The step S2 of the embodiment includes S2.1-S2.3, specifically:

[0058] S2.1, obtaining historical otological multi-modal data of a plurality of otological patients within a target period of time, including historical otological image data and historical otological text data; specifically including temporal bone CT images, endotoscopic images, audiological reports, and clinical otological record texts, and dividing them into a training set and a validation set; wherein the training set is used for model learning and training, and the validation set is used for subsequent evaluation of model performance.

[0059] Data preprocessing is performed on the training set and the validation set, including image data processing: dividing the image data such as temporal bone CT and endotoscopic images into 1024x1024 pixels to ensure consistent sample spatial resolution; eliminating brightness and contrast differences of different imaging devices through grayscale normalization; removing noise by using Gaussian filtering, with a focus on eliminating metal artifacts of CT images and lens reflections of endotoscopic images to ensure clear key areas such as the auditory ossicular chain, tympanum, and tympanic membrane. Text data processing: performing word segmentation on audiological reports and clinical otological record texts using a medical word segmentation dictionary to avoid splitting of professional terms such as "air conduction bone conduction difference"; after removing stop words such as "of" and "consultation" that have no diagnostic significance, extracting structured features such as "air conduction decreased by 30 dB", "tympanic membrane perforation", and "air conduction bone conduction difference of 20 dB" to provide standardized input for subsequent feature extraction.

[0060] Based on a preset ConvNeXt model, a ConvNeXt comprehensive model (also known as a ConvNeXt+CBAM model) is generated by integrating a ConvNeXt Block module, a Downsample module, and a CBAM attention module; wherein the ConvNeXt model is a convolutional neural network architecture for image feature extraction.

[0061] Wherein, the ConvNeXt Block module is established based on deep separable convolution and layer normalization mechanism, the Downsample module is established by down-sampling the feature map, and the CBAM attention module is established by dynamically adjusting the feature weight through channel attention and spatial attention. Specifically:

[0062] ①ConvNeXt Block module: taking deep separable convolution as the core operation unit, it is constructed with layer normalization mechanism (Layer Normalization), which can effectively extract local detail features in the image and lay a data foundation for subsequent high-level feature analysis;

[0063] ②Downsample module: by down-sampling the feature map output by the previous module, it reduces the resolution and computational dimension of the feature map while preserving key feature information, balancing model operation efficiency and feature extraction accuracy;

[0064] ③CBAM attention module: composed of channel attention (Channel Attention Module, CAM) and spatial attention (Spatial Attention Module, SAM), CAM can focus on features channels with diagnostic value in the image, and SAM can locate key anatomical regions in the image. By dynamically adjusting the weight of different features, the representation ability of core diagnostic information is strengthened;

[0065] And, in the module integration logic, considering that high-level feature maps can better reflect the association between lesions and anatomical structures, the model only adds an attention mechanism in the last one of the four repeated modules, after the last ConvNeXtBlock, to efficiently capture and transmit key information related to abnormal features of conductive hearing loss, avoiding interference from irrelevant low-order features.

[0066] The "four repeated modules" refer to the number of times the ConvNeXt Block module is stacked in the model, that is, the model uses the "ConvNeXt Block" feature extraction unit 4 times to gradually extract low, medium and high level features of the image. The first three ConvNeXt Blocks extract low / medium level features such as texture and basic anatomical structure, and the fourth one extracts high level features such as ossicular chain and tympanic cavity related to the lesion. "Adding attention mechanism only in the last module of the four repeated modules" means that a CBAM attention module is connected after the fourth ConvNeXt Block module: because high-level features can better reflect the key information related to the abnormal features of conductive hearing loss in the image, adding CBAM here can precisely adjust the weight of high-level features, focus on the core diagnostic area, avoid interference from low-level features, and improve the relevance of feature extraction.

[0067] Based on the training set, the anatomical structure features and lesion features in the historical otology image data are automatically extracted and quantitatively represented through the multi-module collaborative operation of the ConvNeXt comprehensive model, and finally the image feature vector is output. In addition, ResNet50 can also be used to extract image feature vectors from historical otology image data. ResNet50 is a basic model used for image feature extraction in this embodiment, which is a 50-layer classic version of the residual neural network (Residual Neural Network, ResNet). It can also be used as a teacher model for knowledge distillation to train a lightweight image feature extraction model.

[0068] The BERT model is used to encode the historical otology text data and extract structured features, including air-bone gap, tympanic membrane perforation description, etc., to obtain a text feature vector.

[0069] An image feature query matrix is established according to the image feature vector, and a key matrix and a value matrix of the text feature are established according to the text feature vector.

[0070] Based on the fusion formula of the Query-Key-Value attention mechanism, the attention weight is calculated according to the query matrix and the key matrix, and the weighted sum is calculated in combination with the value matrix. At the same time, the regional features in the image feature vector and the corresponding semantic descriptions in the text feature vector are semantically aligned, for example, the regional features of "abnormal morphology of auditory ossicles" in the CT image are associated and matched with the semantic description of "interruption of the ossicular chain" in the audiology report or otology record text, to ensure that the image spatial information and the text semantic information form a corresponding relationship. Finally, through the above series of operations, the joint representation vectors of a plurality of otology patients are calculated.

[0071] The fusion formula is:

[0072]

[0073] wherein Fusion output is the joint representation vector, Q image is the query matrix of image features, and V text are the key matrix and value matrix of text features, respectively, and d is the feature dimension.

[0074] The embodiment S2.1 extracts image features through the ConvNeXt comprehensive model to obtain an image feature vector, encodes text data to obtain a text feature vector, and uses an attention mechanism to fuse the two features to generate a joint representation vector, effectively solving the modal difference problem of otological images and text data. The introduction of the attention mechanism enables dynamic focusing on the relevance features of images and text, improving the relevance and effectiveness of multi-modal data fusion, providing more comprehensive and accurate basic feature support for subsequent abnormal feature extraction and diagnosis, and enhancing the recognition ability of the model for conductive hearing loss related features;

[0075] In addition, by establishing the calculation relationship between the image query matrix and the text key-value matrix, the model can adaptively allocate attention weights to highlight the key feature information for diagnosis. The semantic alignment operation further eliminates the semantic gap between the image region and the text description, ensuring the accurate correspondence between the image lesion region and the text diagnostic description. This mechanism enables the joint representation vector to retain the spatial details of the image and integrate the clinical semantics of the text, providing higher quality feature input for subsequent model training and abnormal recognition, and improving the accuracy and reliability of auxiliary diagnosis;

[0076] In addition, by integrating the ConvNeXt Block, Downsample, and CBAM attention modules to build a comprehensive model, the technical advantages of each module are fully utilized, and the efficiency and accuracy of otological image feature extraction are improved. The ConvNeXt Block is based on depth separable convolution and layer normalization, which reduces the computational complexity while enhancing the feature extraction capability; the Downsample module effectively reduces the feature map dimension through downsampling, which can improve the model operation efficiency; the CBAM attention module dynamically adjusts the feature weights of the channel and spatial dimensions, enabling the model to focus on the key regions related to conductive hearing loss in otological images. The synergistic effect of the three makes the model not only efficient in processing multi-modal image data, but also accurate in capturing lesion features, which can provide high-quality image feature input for subsequent multi-modal fusion and abnormal recognition, enhancing the clinical applicability of the model.

[0077] S2.2, based on the joint representation vector of several otology patients, the joint representation vector generated by the multi-modal data of the same patient is taken as the positive sample pair, and the two correspond to the conductive hearing loss lesion of the same patient, the features have strong correlation and similarity; the joint representation vector of different patients is randomly paired as a negative sample pair, and the two belong to unrelated patients and lesions, and the features have significant differences, so as to obtain positive and negative sample pairs;

[0078] According to InfoNCE Loss (Information Noise Contrastive Estimation Loss), a loss function is established

[0079] Based on the unlabeled abnormal technical features or lesion related information of the historical otology multi-modal data, the first joint representation vector is generated according to the ConvNeXt comprehensive model, and the first joint representation vector is input into the initial model pre-trained by the positive and negative sample pairs, and the abnormal prediction result set containing confidence is output by the model. Wherein, the initial model is a multi-modal generative AI system based on the Transformer architecture, and the architecture includes: ① feature extraction: image data is processed by ConvNeXt+CBAM, and text data is encoded by BERT; ② cross-modal fusion: generate joint representation vector through Transformer layer; ③ diagnosis reasoning: output abnormal features and corresponding confidence based on joint representation vector, and generate judgment logic.

[0080] The abnormal prediction result with confidence greater than the preset value in the abnormal prediction result set is defined as a pseudo label; wherein, the preset value is 0.9.

[0081] Wherein, the loss function is:

[0082]

[0083] Wherein, z i and z j are positive sample feature vectors, τ is a temperature coefficient, N represents the total number of samples involved in one contrast calculation; k is the index variable of summation operation, and the value range is 1 to N.

[0084] The embodiment S2.2 uses the pre-trained model to predict the joint representation vector of unlabeled data, and selects the result with confidence higher than the preset value as the pseudo label, which not only expands the training sample size, but also ensures the reliability of the pseudo label through the confidence threshold, avoiding the interference of low-quality labels on model training. This semi-supervised learning strategy can fully exploit the implicit abnormal information in unlabeled data, reduce the dependence on large-scale labeled data, and is especially suitable for scenarios with high clinical data labeling cost and scarce samples, which can enhance the generalization ability and practical clinical application value of the model.

[0085] S2.3, based on the loss function According to the positive and negative sample pairs, the initial model is trained by contrast learning to obtain a pre-trained initial model; wherein the loss function is used to optimize the initial model by narrowing the distance of the positive sample features and widening the distance of the negative sample features;

[0086] The pseudo-label and the historical otology multi-modal data without labeled lesion information are combined to form new samples, which are expanded to a training set containing historical otology image data and otology text data, to obtain an expanded training set, and the pre-trained initial model is trained again according to the expanded training set to obtain a comprehensive model.

[0087] Based on the validation set, the comprehensive model is verified and optimized again, including:

[0088] ① Precision optimization: hierarchical 5-fold cross-validation is used to ensure that the proportion of abnormal types such as tympanosclerosis and middle ear ossicular chain interruption in each fold of data is consistent with the overall data set, and the validation set contains 20% of complex cases such as anatomical variations and multiple coexisting lesions; AUC (target ≥ 0.95), sensitivity (> 90%), and specificity (> 88%) are used as main indicators, and F1-score and precision-recall curve (PR curve) are used as auxiliary indicators to evaluate and iteratively adjust the number of ConvNeXt repeating modules, CBAM attention module weights, and other hyperparameters to improve the diagnostic accuracy of the model. Among them, AUC is an index for measuring the ability of a binary classification model to distinguish between positive and negative samples, with a value range of 0-1, and the closer the value is to 1, the stronger the model's ability to distinguish between positive and negative samples; F1-score is the harmonic mean of precision and recall, which is used to balance the precision and recall of the model in the classification task, avoiding the limitations of a single indicator.

[0089] ② Real-time optimization: In terms of model lightweight, knowledge distillation is carried out with ResNet50 as the teacher model and MobileNetV3 as the student model; among them, MobileNetV3 only replaces the image feature extraction module of the initial model, i.e. replacing ResNet50 (basic image extraction scheme) or ConvNeXt+CBAM (improved image extraction scheme), while the cross-modal fusion and diagnostic reasoning part remains unchanged, reducing the parameter amount by 50%; At the same time, the model weight is quantized from FP32 to INT8, and the inference speed is improved by 2 times; In terms of hardware acceleration, the inference engine is optimized by TensorRT, and the single diagnosis inference time on NVIDIA T4 GPU is ≤0.1 second to meet the clinical rapid diagnosis demand. Among them, TensorRT is a deep learning inference optimization engine developed by NVIDIA, and NVIDIA T4 GPU is a special inference GPU (Graphics Processing Unit) developed by NVIDIA.

[0090] ③Interpretability optimization: Grad-CAM (Gradient-weighted Class Activation Mapping) is used to generate heat maps for attention visualization, clearly showing the abnormal areas in the temporal bone CT and ear endoscope images corresponding to the model's focus on abnormal features, such as tympanic membrane sclerosis and abnormal parts of the ossicular chain. The logic extraction generates generative AI capabilities, automatically outputting reasoning texts containing evidence and confidence, such as "CT shows the hammer-anvil gap is widened (attention weight 0.87), combined with the audiology report that the air conduction is decreased by 30 dB, it is speculated that the ossicular chain is abnormal and interrupted (confidence 92%)". This assists relevant personnel in understanding and judging the basis.

[0091] According to the verified and optimized comprehensive model, the ear multi-modal data of the user is processed for abnormal features, and the process is as follows: first, the image data is processed by MobileNetV3, and the text data is encoded by BERT model; then, the joint representation vector is generated through the cross-modal Transformer layer; finally, the abnormal features and the corresponding confidence scores are outputted, and the abnormal features and the corresponding confidence scores of the user's ear data are obtained, and the judgment logic is generated.

[0092] The multi-stage training strategy is adopted in this embodiment S2.3, first through contrast learning with positive and negative sample pairs, and then second training with the training set expanded by pseudo labels, which significantly improves the diagnostic performance of the comprehensive model. Contrast learning narrows the distance between positive samples and pushes away the distance between negative samples, enabling the model to more accurately distinguish between normal and abnormal features, laying a good foundation for feature discrimination. The second training with the expanded training set by pseudo labels further utilizes the effective information in unlabeled data, refining the model's learning of complex lesion features. The phased training enables the model to first master the basic discrimination ability and then optimize the detail recognition accuracy, ensuring the stability of the model while improving the ability to capture subtle abnormalities related to conductive hearing loss, ultimately improving the accuracy and robustness of auxiliary diagnosis.

[0093] For the application of the embodiments of the present application, please refer to Figures 2-3 , Figure 2is a schematic diagram of a CBAM attention module provided by an embodiment of the present application, which is a structural schematic diagram of CBAM (Convolutional Block Attention Module), and the core shows the composition and data processing flow of the CBAM attention module: the diagram contains two core components, a channel attention module (CAM) and a spatial attention module (SAM), and clearly marks the input (such as an HRCT image with a size of 224x224), the process of processing by the CAM and the SAM in turn, and the final output of the refined feature, which directly reflects the working principle of the CBAM module by dynamically adjusting the feature weight and focusing on the key regions of the image;

[0094] Figure 3 is a ConvNeXt model structure schematic diagram based on the CBAM attention mechanism provided by an embodiment of the present application, which is a complete architecture schematic diagram of the ConvNeXt+CBAM model (i.e., the ConvNeXt integrated model), and the core shows the hierarchical structure and data flow of the improved ConvNeXt+CBAM model: the diagram clearly marks the components of the whole process of the model, including an input layer (an HRCT image with a size of 224x224), an initial Conv2d layer, a Layer Normal, a DownSample (downsampling) module, multiple groups of ConvNeXt Block modules (annotating the repetition number n=3, n=9, etc. of different modules), and clearly embodies the design of "connecting the CBAM attention module after a specific ConvNeXt Block", and finally outputs the classification result (abnormal feature classification extraction result) through the Layer Normal and Linear (fully connected) layer, which completely presents the integration mode of the CBAM attention mechanism and the ConvNeXt model, and the hierarchical logic of the model for extracting image features.

[0095] Overall, the embodiment has the following beneficial effects:

[0096] According to the comprehensive model, the abnormal feature extraction of the ear multi-modal data is carried out, since the model has mastered the multi-modal correlation logic through the previous training, the abnormal feature can be quickly located and extracted, the intermediate redundant processing steps are reduced, the abnormal feature extraction efficiency can be improved, and the demand of clinical multi-modal data rapid processing is met. For the comprehensive model, the joint representation vector is generated by directly extracting the features of the historical ear image data and the text data, rather than processing the features of the two modalities separately. This way can actively capture the internal correlation between the image and the text, form a unified multi-modal feature carrier, avoid the integration inefficiency caused by the modality isolation, and provide higher fusion basis data for subsequent abnormal feature extraction. The positive and negative sample pairs are generated by pairing the joint representation vectors of different users, so that the initial model can accurately distinguish the multi-modal differences between normal and abnormal features in the contrast learning. In addition, the high-confidence pseudo-labels are generated by using the unlabeled data to expand the training set, which not only solves the problem of clinical annotation data scarcity, but also enables the model to learn the correlation rules from more diverse multi-modal data. In addition, the attention mechanism of the Transformer architecture can dynamically adapt to the importance of different modal features, which can greatly improve the integration efficiency and feature learning accuracy of multi-modal data compared with the shallow fusion of the prior art.

[0097] Embodiment two:

[0098] Please refer to Figure 4 The embodiment of the present application provides an ear multi-modal data abnormal feature extraction device, which comprises a data module 10 and an extraction module 20.

[0099] The data module 10 is used for acquiring ear multi-modal data of a user.

[0100] The extraction module 20 is used for extracting abnormal features of ear multi-modal data according to a comprehensive model, so as to obtain ear data abnormal features. The comprehensive model is obtained by multi-stage training of an initial model according to positive and negative sample pairs and pseudo-labels. The positive and negative sample pairs are generated by pairing joint representation vectors of different users. The joint representation vectors are obtained by extracting features of historical ear image data and ear text data. The pseudo-labels are generated by generating joint representation vectors from historical ear multi-modal data without labeled lesion information, and filtering the prediction results with a confidence greater than a preset value by the trained initial model. The initial model is a multi-modal generative AI system based on a Transformer architecture.

[0101] In one embodiment, the data module 10 is specifically:

[0102] Obtaining otological multi-modal data of a user, including but not limited to: patient CT images, endotoscopic images, audiological reports, and simultaneously obtaining otological related text information in the user's clinical otological records, to provide complete data sources for subsequent multi-modal data fusion analysis.

[0103] In one embodiment, the extraction module 20 includes a data unit, an image unit, a text unit, a matrix subunit, a vector subunit, a sample unit, a training unit, and an extraction unit.

[0104] The data unit is configured to obtain historical otological multi-modal data of a plurality of otological patients within a target period, including historical otological image data and historical otological text data; specifically including temporal bone CT images, endotoscopic images, audiological reports, and clinical otological record texts, and dividing them into a training set and a validation set; wherein the training set is used for model learning and training, and the validation set is used for subsequent evaluation of model performance.

[0105] The data unit is also configured to perform data preprocessing on the training set and the validation set, including image data processing: dividing temporal bone CT and endotoscopic images and other image data into 1024x1024 pixels to ensure consistent sample spatial resolution; eliminating brightness and contrast differences of different imaging devices through grayscale normalization; and removing noise using Gaussian filtering, with a focus on eliminating metal artifacts of CT images and lens reflections of endotoscopic images to ensure clear key areas such as the ossicular chain, tympanum, and tympanic membrane. Text data processing: performing word segmentation on audiological reports and clinical otological record texts using a medical word segmentation dictionary to avoid splitting of professional terms such as "air-bone gap"; after removing stop words such as "of" and "consultation" that have no diagnostic significance, extracting structured features such as "air conduction decreased by 30 dB", "tympanic membrane perforation", and "air-bone gap of 20 dB" to provide standardized input for subsequent feature extraction.

[0106] The image unit is configured to generate a ConvNeXt comprehensive model (also known as a ConvNeXt+CBAM model) based on a preset ConvNeXt model by integrating a ConvNeXt Block module, a Downsample module, and a CBAM attention module; wherein the ConvNeXt model is a convolutional neural network architecture for image feature extraction.

[0107] The ConvNeXt Block module is established based on depthwise separable convolution and layer normalization mechanism, the Downsample module is established by downsampling the feature map, and the CBAM attention module is established by dynamically adjusting the feature weight through channel attention and spatial attention. Specifically:

[0108] ①ConvNeXt Block module: taking depthwise separable convolution as the core operation unit, combined with layer normalization mechanism, it can effectively extract local detailed features in the image and lay a data foundation for subsequent high-level feature analysis;

[0109] ②Downsample module: by downsampling the feature map output by the previous module, the resolution and computational dimension of the feature map are reduced while the key feature information is preserved, balancing the model operation efficiency and feature extraction accuracy;

[0110] ③CBAM attention module: composed of channel attention (CAM) and spatial attention (SAM), CAM can focus on the feature channels with diagnostic value in the image, and SAM can locate the key anatomical regions in the image. By dynamically adjusting the weights of different features, the representation ability of core diagnostic information is enhanced.

[0111] In terms of module integration logic, considering that high-level feature maps can better reflect the relationship between lesions and anatomical structures, the model only adds an attention mechanism in the last of the four repeated modules, after the last ConvNeXtBlock, to efficiently capture and transmit key information related to abnormal features of conductive hearing loss, avoiding interference from irrelevant low-level features.

[0112] Among them, "four repeated modules" refers to the number of times the ConvNeXt Block module is stacked in the model, that is, the model will use the "ConvNeXt Block" feature extraction unit 4 times to gradually extract low, medium and high-level features of the image. The first three ConvNeXt Blocks extract low / middle-level features such as texture and basic anatomical structure, and the fourth extracts high-level features such as the ossicular chain and tympanic cavity related to lesions. "Only in the last module of the four repeated modules, add an attention mechanism" means that only after the fourth ConvNeXt Block module, the CBAM attention module is connected: because high-level features can better reflect the key information related to abnormal features of conductive hearing loss in the image, adding CBAM here can accurately adjust the weight of high-level features, focus on the core diagnostic area, avoid interference from low-level features, and improve the relevance of feature extraction.

[0113] The image unit is further configured to automatically extract and quantitatively represent anatomical structure features and lesion features in historical ear surgery image data based on the training set through multi-module collaborative operation of the ConvNeXt comprehensive model, and finally output an image feature vector. In addition, a ResNet50 can also be used to extract an image feature vector from the historical ear surgery image data. The ResNet50 is a basic model for image feature extraction in the embodiment, which is a 50-layer classic version of a residual neural network (ResNet) and can be used as a teacher model for training a lightweight image feature extraction model through knowledge distillation.

[0114] The text unit is configured to encode historical ear surgery text data using a BERT model and extract structured features, including air-bone gap, tympanic membrane perforation description, etc., to obtain a text feature vector.

[0115] The matrix sub-unit is configured to establish a query matrix of image features based on the image feature vector, and establish a key matrix and a value matrix of text features based on the text feature vector.

[0116] The vector sub-unit is configured to calculate attention weights based on a fusion formula of a Query-Key-Value attention mechanism according to the query matrix and the key matrix, and perform weighted summation combined with the value matrix, while performing semantic alignment between region features in the image feature vector and corresponding semantic descriptions in the text feature vector, for example, associating and matching the region features of "abnormal morphology of auditory ossicles" in a CT image with the semantic description of "auditory ossicular chain interruption" in a hearing report or ear surgery record text, to ensure that image spatial information and text semantic information form a corresponding relationship. Finally, a plurality of joint representation vectors of ear surgery patients are calculated through the above series of operations.

[0117] The fusion formula is:

[0118]

[0119] wherein Fusion output is the joint representation vector, Q image is the query matrix of image features, and V text are the key matrix and the value matrix of text features, respectively, and d is the feature dimension.

[0120] The data unit, image unit, text unit, matrix sub-unit and vector sub-unit of the embodiment extract image feature vectors through the ConvNeXt comprehensive model and encode text data to obtain text feature vectors, and generate joint representation vectors by fusing the two features by using an attention mechanism, effectively solving the modal difference problem of otological images and text data. The introduction of the attention mechanism enables the data processing to dynamically focus on the correlation features of the images and the text, improving the relevance and effectiveness of the multi-modal data fusion, providing more comprehensive and accurate basic feature support for subsequent abnormal feature extraction and diagnosis, and enhancing the recognition ability of the model for the features related to conductive hearing loss;

[0121] Moreover, by establishing the calculation relationship between the image query matrix and the text key-value matrix, the model can adaptively allocate attention weights to highlight the key feature information for diagnosis. The semantic alignment operation further eliminates the semantic gap between the image regions and the text descriptions, ensuring the accurate correspondence between the image lesion regions and the text diagnostic descriptions. This mechanism enables the joint representation vectors to retain the spatial details of the images and fuse the clinical semantics of the text, providing higher quality feature input for subsequent model training and abnormal recognition, and improving the accuracy and reliability of the auxiliary diagnosis;

[0122] In addition, by integrating the ConvNeXt Block, Downsample and CBAM attention modules to build a comprehensive model, the technical advantages of each module are fully utilized, and the efficiency and accuracy of otological image feature extraction are improved. The ConvNeXt Block is based on depth separable convolution and layer normalization, which reduces the computational complexity while enhancing the feature extraction capability; the Downsample module effectively reduces the feature map dimension through downsampling, which can improve the model operation efficiency; the CBAM attention module dynamically adjusts the feature weights of the channel and spatial dimensions, which can enable the model to focus on the key regions related to conductive hearing loss in otological images. The synergistic effect of the three makes the model not only efficient in processing multi-modal image data, but also accurate in capturing lesion features, which can provide high-quality image feature input for subsequent multi-modal fusion and abnormal recognition, and enhance the clinical applicability of the model.

[0123] The sample unit is configured to generate joint representation vectors based on the multi-modal data of a plurality of otological patients, and take the joint representation vectors generated by the multi-modal data of the same patient as positive sample pairs, which correspond to the conductive hearing loss lesions of the same patient and have strong correlation and similarity; take the joint representation vectors of different patients randomly paired as negative sample pairs, which belong to unrelated patients and lesions and have significant differences in features, so as to obtain positive and negative sample pairs;

[0124] The sample unit is further configured to establish a loss function according to an InfoNCE Loss (Information Noise Contrastive Estimation Loss)

[0125] The sample unit is further configured to generate a first joint representation vector according to a ConvNeXt comprehensive model based on historical otological multi-modal data without labeled abnormal technical features or lesion-related information, and input the first joint representation vector into an initial model pre-trained according to the positive and negative sample pairs, so as to output an abnormal prediction result set containing a confidence from the model. The initial model is a multi-modal generative AI system based on a Transformer architecture, and the architecture of the initial model includes: ① feature extraction: processing image data by ConvNeXt+CBAM and encoding text data by BERT; ② cross-modal fusion: generating a joint representation vector through a Transformer layer; and ③ diagnosis reasoning: outputting abnormal features and corresponding confidence based on the joint representation vector, and generating a judgment logic.

[0126] The sample unit is further configured to define an abnormal prediction result with a confidence greater than a preset value in the abnormal prediction result set as a pseudo label; and the preset value is 0.9.

[0127] The loss function is:

[0128]

[0129] wherein z i and z j are the positive sample feature vectors, τ is a temperature coefficient, N represents the total number of samples participating in one contrast calculation, and k is an index variable for summation operation, and the value range of k is 1 to N.

[0130] The sample unit of the embodiment utilizes a pre-trained model to predict the joint representation vector of unlabeled data, and selects a result with a confidence higher than a preset value as a pseudo label, which not only expands the training sample quantity, but also ensures the reliability of the pseudo label through the confidence threshold, thereby avoiding the interference of low-quality labels on model training. This semi-supervised learning strategy can fully exploit the implicit abnormal information in unlabeled data, reduce the dependence on large-scale labeled data, and is especially suitable for scenarios with high clinical data labeling cost and scarce samples, and can enhance the generalization ability and actual clinical application value of the model.

[0131] The training unit is configured to train the initial model based on the loss function The initial model is trained through contrast learning based on the positive and negative sample pairs, and a pre-trained initial model is obtained; wherein the loss function is used to optimize the initial model by shortening the distance of the positive sample features and lengthening the distance of the negative sample features;

[0132] The training unit is further configured to form new samples by combining the pseudo labels and the historical otology multi-modal data without labeled lesion information, expand the training set containing historical otology image data and otology text data to obtain an expanded training set, and retrain the pre-trained initial model according to the expanded training set to obtain a comprehensive model.

[0133] The training unit is further configured to verify and re-optimize the comprehensive model based on the verification set, including:

[0134] ① Precision optimization: hierarchical 5-fold cross-validation is adopted to ensure that the proportion of abnormal types such as tympanic membrane sclerosis and middle ear ossicular chain interruption in each fold of data is consistent with the overall data set, and the verification set contains 20% of complex cases such as anatomical variations and multiple coexisting lesions; AUC (target ≥ 0.95), sensitivity (> 90%), and specificity (> 88%) are used as main indicators, and F1-score and precision-recall curve (PR curve) are used as auxiliary indicators to evaluate and iteratively adjust the number of ConvNeXt repeating modules, CBAM attention module weights and other hyperparameters to improve the diagnostic accuracy of the model. Among them, AUC is an index for measuring the ability of a binary classification model to distinguish between positive and negative samples, with a value range of 0-1, and the closer the value is to 1, the stronger the model's ability to distinguish between positive and negative samples; F1-score is the harmonic mean of precision and recall, which is used to balance the precision and recall ability of the model in the classification task to avoid the limitations of a single indicator.

[0135] ② Real-time optimization: in terms of model lightweight, knowledge distillation is carried out with ResNet50 as the teacher model and MobileNetV3 as the student model; among them, MobileNetV3 only replaces the image feature extraction module of the initial model, i.e., replacing the ResNet50 (basic image extraction scheme) or ConvNeXt+CBAM (improved image extraction scheme) part, and the cross-modal fusion and diagnosis reasoning part remains unchanged, with a compression parameter amount of 50%; at the same time, the model weight is quantized from FP32 to INT8, and the inference speed is improved by 2 times; in terms of hardware acceleration, the inference engine is optimized through TensorRT, and the single diagnosis inference time is ≤0.1 seconds on NVIDIA T4 GPU to meet the clinical rapid diagnosis demand. Among them, TensorRT is a deep learning inference optimization engine developed by NVIDIA, and NVIDIA T4 GPU is a special inference GPU (graphics processing unit) developed by NVIDIA.

[0136] ③Interpretability optimization: Grad-CAM (Gradient-weighted Class Activation Mapping) is used to generate heat maps for attention visualization, clearly showing the abnormal areas of temporal bone CT and ear endoscope images corresponding to the model's focus on abnormal features, such as tympanic membrane sclerosis and abnormal parts of the auditory ossicular chain. The extraction logic generates generative AI capabilities, automatically outputting reasoning text containing evidence and confidence, such as "CT shows that the space between the malleus and the incus is widened (attention weight 0.87), combined with the audiology report that the air conduction is decreased by 30 dB, it is speculated that the auditory ossicular chain is abnormal and interrupted (confidence 92%)". This assists relevant personnel in understanding and judging the basis.

[0137] The extraction unit is used to perform abnormal feature processing on the user's otological multi-modal data based on the verified and optimized integrated model. The process is as follows: first, process the image data with MobileNetV3 and encode the text data with the BERT model; then generate joint representation vectors through the cross-modal Transformer layer; finally, output abnormal features and corresponding confidence scores, and ultimately obtain the abnormal features of the user's otological data and the confidence results of each feature, and generate judgment logic.

[0138] The training unit and extraction unit of this embodiment use a multi-stage training strategy. First, contrastive learning is performed on positive and negative sample pairs, and then the training set is expanded with pseudo-labels for secondary training, significantly improving the diagnostic performance of the integrated model. Contrastive learning narrows the distance between positive samples and widens the distance between negative samples, enabling the model to more accurately distinguish between normal and abnormal features, laying a good foundation for feature discrimination. The secondary training with pseudo-label expanded training set further utilizes the effective information in unlabeled data, refining the model's learning of complex lesion features. The phased training enables the model to first master basic discrimination ability and then optimize detail recognition accuracy, ensuring the stability of the model while improving the ability to capture subtle abnormalities related to conductive hearing loss, ultimately improving the accuracy and robustness of auxiliary diagnosis.

[0139] For the application of this embodiment, please refer to Figures 2-3 , Figure 2is a schematic diagram of a CBAM attention module provided by the embodiment of the present application, which is a structural schematic diagram of CBAM (Convolutional Block Attention Module), and the core shows the composition and data processing flow of the CBAM attention module: the diagram contains two core components, a channel attention module (Channel Attention Module, CAM) and a spatial attention module (Spatial Attention Module, SAM), and clearly marks the input (such as an HRCT image with a size of 224x224), the process of processing by the CAM and the SAM in turn, and the final output of the refined feature (Refined Feature) result, which directly reflects the working principle of the CBAM module by dynamically adjusting the feature weight and focusing on the key area of the image;

[0140] Figure 3 is a ConvNeXt model structure schematic diagram based on the CBAM attention mechanism provided by the embodiment of the present application, which is a complete architecture schematic diagram of the ConvNeXt+CBAM model (i.e., the ConvNeXt integrated model), and the core shows the hierarchical structure and data flow of the improved ConvNeXt+CBAM model: the diagram clearly marks the components of the whole process of the model, including the input layer (an HRCT image with a size of 224x224), the initial Conv2d layer, the Layer Normal, the DownSample (downsampling) module, multiple groups of ConvNeXt Block modules (annotating the repetition number n=3, n=9, etc. of different modules), and clearly embodies the design of "connecting the CBAM attention module after a specific ConvNeXt Block", and finally outputs the classification result (abnormal feature classification extraction result) through the Layer Normal, Linear (fully connected) layer, which fully presents the integration mode of the CBAM attention mechanism and the ConvNeXt model, and the hierarchical logic of the model for extracting image features.

[0141] Overall, the embodiment has the following beneficial effects:

[0142] The application extracts abnormal features from otology multi-modal data according to a comprehensive model. Since the model has mastered the multi-modal correlation logic through previous training, it can quickly locate and extract abnormal features, reduce intermediate redundant processing steps, improve the efficiency of abnormal feature extraction, and meet the clinical demand for rapid processing of multi-modal data. For the comprehensive model, the joint representation vector is generated by directly extracting features from historical otology image data and text data, rather than processing the features of the two modalities separately. This approach actively captures the internal correlation between images and text, forms a unified multi-modal feature carrier, avoids the integration inefficiency caused by modality isolation, and provides higher fusion data for subsequent abnormal feature extraction. By pairing different user joint representation vectors to generate positive and negative sample pairs, the initial model can accurately distinguish the multi-modal differences between normal and abnormal features in the contrast learning. Moreover, the use of unlabeled data to generate high-confidence pseudo-labels to expand the training set not only solves the problem of scarce clinical annotation data, but also enables the model to learn the correlation rules from a wider variety of multi-modal data. In addition, the attention mechanism of the Transformer architecture can dynamically adapt to the importance of different modal features, which can significantly improve the integration efficiency and feature learning accuracy of multi-modal data compared to the shallow fusion of existing technologies.

[0143] Embodiment three:

[0144] The embodiment of the application provides a computer readable storage medium, the computer readable storage medium comprises a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the abnormal feature extraction method of otology multi-modal data when the computer program runs.

[0145] The abnormal feature extraction method of otology multi-modal data can be stored in a computer readable storage medium if it is realized in the form of a software function unit and used as an independent product. Based on this understanding, the application can implement all or part of the processes in the above-mentioned embodiment methods, and can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium, and the computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0146] The above is the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can also make several improvements and refinements, these improvements and refinements are also considered to be within the scope of the present application.

Claims

1. A method for extracting abnormal features from otological multimodal data, characterized in that, include: Acquire the user's otological multimodal data; Anomaly features are extracted from the otology multimodal data using a comprehensive model to obtain otology data anomaly features. The comprehensive model is obtained by multi-stage training of an initial model based on positive and negative sample pairs and pseudo-labels. The positive and negative sample pairs are generated by pairing joint representation vectors of different users. The joint representation vectors are obtained by feature extraction from historical otology image data and otology text data. The pseudo-labels are generated by producing joint representation vectors from historical otology multimodal data without labeled lesion information, and by filtering prediction results with confidence scores greater than preset values ​​through a trained initial model; the initial model is a multimodal generative AI system based on the Transformer architecture.

2. The method for extracting abnormal features from otological multimodal data as described in claim 1, characterized in that, The joint representation vector is obtained by feature extraction from historical otology image data and otology text data, specifically as follows: Acquire historical otology image data and historical otology text data from several users; Feature extraction was performed on the historical otology image data using the ConvNeXt comprehensive model to obtain image feature vectors; The historical otology text data is encoded and structured features are extracted to obtain text feature vectors; The image feature vector and the text feature vector are fused according to a preset attention mechanism to obtain the joint representation vector.

3. The method for extracting abnormal features from otological multimodal data as described in claim 2, characterized in that, The joint representation vector is obtained by fusing the image feature vector and the text feature vector according to a preset attention mechanism, specifically as follows: A query matrix for image features is established based on the image feature vectors, and a key matrix and a value matrix for text features are established based on the text feature vectors. Based on the Query-Key-Value attention mechanism, attention weights are calculated according to the query matrix and the key matrix, and then weighted and summed in combination with the value matrix. At the same time, the regional features in the image feature vector and the corresponding semantic descriptions in the text feature vector are semantically aligned to calculate the joint representation vector.

4. The method for extracting abnormal features from otological multimodal data as described in claim 1, characterized in that, The pseudo-labels are generated by producing joint representation vectors from historical otological multimodal data without labeled lesion information, and by filtering prediction results with confidence scores greater than a preset value using a trained initial model. Specifically: Based on historical otology multimodal data without labeled abnormal technical features or lesion-related information, a first joint representation vector is generated according to the ConvNeXt comprehensive model. The first joint representation vector is input into the initial model pre-trained by the positive and negative sample pairs, and the model outputs an abnormal prediction result set containing confidence. The abnormal prediction results with a confidence level greater than a preset value are defined as the pseudo-labels.

5. The method for extracting abnormal features from otological multimodal data as described in claim 1, characterized in that, The integrated model is obtained by training the initial model in multiple stages based on positive and negative sample pairs and pseudo-labels, specifically as follows: Based on a preset loss function, the initial model is trained by comparative learning according to the positive and negative sample pairs to obtain a pre-trained initial model; wherein, the loss function optimizes the initial model by bringing the positive sample features closer and pushing the negative sample features further apart. The pseudo-labels and the historical otology multimodal data without labeled lesion information are combined to form a new sample, which is then expanded to the training set containing the historical otology image data and otology text data to obtain the expanded training set. The pre-trained initial model is then trained again based on the expanded training set to obtain the comprehensive model.

6. A method for extracting abnormal features from otological multimodal data as described in any one of claims 1-5, characterized in that, The specific method for establishing the ConvNeXt synthesis model is as follows: Based on the preset ConvNeXt model, the ConvNeXt comprehensive model is generated by integrating the ConvNeXt Block module, the Downsample module, and the CBAM attention module. The ConvNeXt Block module is established based on depthwise separable convolution and layer normalization mechanism, the Downsample module is established by downsampling the feature map, and the CBAM attention module is established by dynamically adjusting the feature weights through channel attention and spatial attention.

7. An anomalous feature extraction device for otological multimodal data, characterized in that, Includes a data module and an extraction module; The data module is used to acquire the user's otological multimodal data. The extraction module is used to extract abnormal features from the otological multimodal data according to the comprehensive model, thereby obtaining abnormal features of the otological data. The comprehensive model is obtained by multi-stage training of an initial model based on positive and negative sample pairs and pseudo-labels. The positive and negative sample pairs are generated by pairing joint representation vectors from different users, and the joint representation vectors are obtained by feature extraction from historical otological image data and otological text data. The pseudo-labels are generated from joint representation vectors generated from historical otological multimodal data without labeled lesion information, and the prediction results with confidence levels greater than a preset value are selected through the trained initial model. The initial model is a multimodal generative AI system based on the Transformer architecture.

8. The device for extracting abnormal features from otological multimodal data as described in claim 7, characterized in that, The extraction module includes a data unit, an image unit, a text unit, and a fusion unit; The data unit is used to acquire historical otology image data and historical otology text data of several users; The image unit is used to extract features from the historical otology image data using the ConvNeXt comprehensive model to obtain image feature vectors; The text unit is used to encode the historical otology text data and extract structured features to obtain a text feature vector. The fusion unit is used to fuse the image feature vector and the text feature vector according to a preset attention mechanism to obtain the joint representation vector.

9. The device for extracting abnormal features from otological multimodal data as described in claim 8, characterized in that, The fusion unit includes matrix sub-units and vector sub-units; The matrix sub-unit is used to establish a query matrix of image features based on the image feature vector, and to establish a key matrix and a value matrix of text features based on the text feature vector. The vector subunit is used to calculate attention weights based on the query matrix and the key matrix according to the Query-Key-Value attention mechanism, and to perform weighted summation in combination with the value matrix. At the same time, it performs semantic alignment between the regional features in the image feature vector and the corresponding semantic descriptions in the text feature vector to calculate the joint representation vector.

10. A storage medium, characterized in that, The storage medium stores a computer program, which is called and executed by a computer to implement an abnormal feature extraction method for otological multimodal data as described in any one of claims 1 to 6.

Images