Image processing system and method and storage medium

Abstract

The invention discloses an image processing system and method and a storage medium. The method comprises the following steps: acquiring an MRI image of an ear of a historical patient and a corresponding medical report; generating left ear and right ear image features according to the MRI image codes; constructing image reconstruction loss according to the left ear and right ear image features; constructing image comparison learning loss according to anchor point features extracted from the MRI image; a specificity severity score is determined according to the medical report, cross-modal fusion and alignment between the image and the text features extracted from the medical report are executed according to the specificity severity score, and cross-modal comparison loss is constructed when cross-modal alignment is executed; constructing a target function according to each loss so as to pre-train the diagnosis model through the target function; and inputting the MRI image of the ear of the current patient and the corresponding medical report into the pre-trained diagnosis model to output a Meniere disease diagnosis result. According to the invention, accurate and efficient diagnosis of Meniere disease can be realized through image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, and in particular to an image processing system, method and storage medium. BACKGROUND

[0002] Meniere's disease (MD) is a multifactorial inner ear disease characterized by recurrent episodes of vertigo and fluctuating sensorineural hearing loss, with endolymphatic hydrops (EH) widely recognized as its main pathological feature. Although gadolinium-enhanced MRI (magnetic resonance imaging) has become a key technology for visualizing inner ear lesions, it still faces some significant limitations in routine clinical practice, and there are still several challenges in clinical practice. First, the inner ear lesions in early or mild MD are usually small and localized, which makes image resolution and partial volume effects limiting factors, which may lead to insufficient diagnosis. Second, although inner ear lesions are a typical pathological feature of MD, similar imaging patterns can also appear in other inner ear diseases, which may affect the accuracy of diagnosis. In addition, traditional diagnostic methods often overlook subtle pathological clues in cochlear and vestibular structures, partly because these methods are difficult to integrate complementary information provided by MRI images and radiology reports. Therefore, relying solely on imaging indicators is often insufficient to accurately and efficiently diagnose MD. SUMMARY

[0003] The present application aims to at least partially solve the technical problems in the related art. To this end, the first object of the present application is to provide an image processing method that can achieve accurate and efficient diagnosis of Meniere's disease by integrating complementary information provided by MRI images and radiology reports.

[0004] The second object of the present application is to provide an image processing system.

[0005] The third object of the present application is to provide a computer-readable storage medium.

[0006] To achieve the above objects, the present application is implemented by the following technical solutions:

[0007] An image processing method, comprising:

[0008] obtaining MRI images of the ear of a historical patient and the corresponding medical report;

[0009] annotating the region of interest on the MRI image and extracting image blocks of the region of interest, encoding the extracted image blocks through an encoder to generate left ear image features and right ear image features;

[0010] constructing an image reconstruction loss according to the left ear image features and the right ear image features, and performing image reconstruction to ensure that there is no loss of lesion information in the encoded image features;

[0011] construct an image contrast learning loss according to the anchor point features extracted on the MRI images, and perform image contrast learning, so that the diagnostic model can identify lesions and normal individual variations;

[0012] determine the specific severity scores of the left ear and the right ear according to the medical reports, perform cross-modal fusion and alignment between the images and the text features extracted from the medical reports according to the specific severity scores of the left ear and the right ear, and construct a cross-modal contrast loss when performing the cross-modal alignment;

[0013] construct a target function according to the image reconstruction loss, the image contrast learning loss and the cross-modal contrast loss, so as to pre-train the diagnostic model through the target function;

[0014] input the MRI images of the current patient's ear and the corresponding medical reports into the pre-trained diagnostic model, and classify and diagnose the fused bilateral image features corresponding to the MRI images of the current patient's ear through the linear classifier in the diagnostic model, to output the Meniere's disease diagnosis result.

[0015] In a possible implementation, the image reconstruction loss is constructed according to the left ear image features and the right ear image features, and the image reconstruction includes:

[0016] reconstruct the corresponding local image blocks through the linear transformation function respectively on the left ear image features and the right ear image features;

[0017] construct a mean square error loss function to measure whether the difference between the reconstructed local image blocks and the corresponding original image blocks is minimal, so that there is no loss of lesion information in the encoded image features; wherein the mean square error loss function is expressed as the sum of squares of the difference between each pixel value of the reconstructed local image blocks and the corresponding original image blocks.

[0018] In a possible implementation, the image contrast learning loss is constructed according to the anchor point features extracted on the MRI images, and the image contrast learning includes:

[0019] determine the positive samples and the negative samples according to the anchor point features extracted on the MRI images; wherein the samples whose spatial positions are within a preset neighborhood range and whose semantic similarity degrees with the anchor point features are greater than a preset value are classified as positive samples, and the samples whose spatial positions are not within the preset neighborhood range and whose semantic similarity degrees with the anchor point features are less than or equal to the preset value are classified as negative samples;

[0020] construct the image contrast learning loss according to the anchor point features, the positive samples and the negative samples;

[0021] The encoder parameters are adjusted to minimize the image contrast learning loss, resulting in an encoder with the minimum image contrast learning loss. The left and right ear image features encoded by the encoder with the minimum image contrast learning loss can identify lesion features or normal individual variation features.

[0022] In one possible implementation, specific severity scores for the left and right ears are determined based on medical reports, including:

[0023] Identify entities in the medical report, the entities being key clinical entities, the entities including anatomical structures, pathological descriptions, and lateral indications, the lateral indications being left, right, and bilateral;

[0024] Calculate the similarity between an entity and the global context in a medical report, so as to determine the context weight of each entity based on the similarity and the presence of negative words in the medical report;

[0025] Determine the basic weight of each entity, determine whether there are severe, moderate, and mild keywords in the medical report, and determine the severity level adjustment coefficient based on the keyword occurrence results;

[0026] The final weight value for each entity is determined based on its base weight, context weight, and severity level adjustment factor, so as to determine the specific severity score for the left and right ears based on the base weight and final weight value of each entity.

[0027] In one possible implementation, after determining the specific severity scores for the left and right ears, the method further includes: determining whether there is at least one bilateral entity in the medical report; if so, increasing the specific severity scores for both the left and right ears by a preset percentage.

[0028] In one possible implementation, cross-modal fusion between images and text features extracted from medical reports is performed based on left and right ear-specific severity scores, including:

[0029] Text features are extracted from medical reports, and left ear image features, right ear image features, and text features are projected into the same shared semantic space.

[0030] The semantic correlation between local image patches and text in the medical report is calculated based on the projected features of the left ear image features, the right ear image features, and the text features. Then, the left ear multi-head attention output and the right ear multi-head attention output are obtained by weighted summation based on the semantic correlation and the corresponding text.

[0031] The multi-head attention outputs of the left ear and the right ear are modulated in severity based on the left ear and right ear specific severity scores respectively to focus on more severe areas of the lesion and obtain the final left ear multi-head attention output and the right ear multi-head attention output;

[0032] The final left ear multi-head attention output and the right ear multi-head attention output are fused to obtain the fused bilateral image features.

[0033] In a possible implementation, the final left ear multi-head attention output and the right ear multi-head attention output are fused to obtain the fused bilateral image features, including:

[0034] The final left ear multi-head attention output and the right ear multi-head attention output are weighted and summed based on the left ear and right ear specific severity scores to obtain preliminary fused features;

[0035] An asymmetry factor is determined based on the left ear and right ear specific severity scores;

[0036] The preliminary fused features are enhanced based on the asymmetry factor;

[0037] The enhanced preliminary fused features are subjected to residual connection and layer normalization processing to obtain the final fused bilateral image features.

[0038] In a possible implementation, performing cross-modal alignment includes:

[0039] The final fused bilateral image features and the text features are subjected to normalization processing, and after the normalization processing, a cosine similarity matrix of the two is determined;

[0040] A dynamic temperature coefficient is determined according to the cosine similarity matrix of the two, the left ear and right ear specific severity scores;

[0041] The positive similarity scores between the final fused bilateral image features and the corresponding text features and other text features are determined respectively;

[0042] A cross-modal contrast loss is constructed according to the two positive similarity scores and the dynamic temperature coefficient, so as to guide the cross-modal alignment of the image and the text through the cross-modal contrast loss, and ensure that the image and the text match, the image being the final fused bilateral image;

[0043] Wherein, a target function constructed based on the cross-modal contrast loss, the image reconstruction loss and the image contrast learning loss is represented as follows:

[0044]

[0045] Wherein, represents the target function, image reconstruction loss for left and right ears respectively, image contrastive learning loss for left and right ears respectively, denotes a cross-modal contrastive loss, λ cl denotes a coefficient for balancing the image reconstruction loss and the image contrastive learning loss, λ gl denotes a coefficient for controlling the global alignment loss.

[0046] To achieve the above object, the second aspect of the present application provides an image processing system, comprising:

[0047] an acquisition module, configured to acquire an MRI image of an ear of a historical patient and a corresponding medical report;

[0048] an encoding module, configured to label a region of interest on the MRI image and extract an image block of the region of interest, encode the extracted image block through an encoder, and generate left ear image features and right ear image features;

[0049] an image reconstruction module, configured to construct an image reconstruction loss according to the left ear image features and the right ear image features, and perform image reconstruction so that there is no loss of lesion information in the encoded image features;

[0050] a contrastive learning module, configured to construct an image contrastive learning loss according to anchor point features extracted on the MRI image, and perform image contrastive learning so that a diagnosis model can identify lesions and normal individual variations;

[0051] a scoring module, configured to determine a specificity severity score of the left ear and the right ear according to the medical report, perform cross-modal fusion and alignment between an image and a text feature extracted from the medical report according to the specificity severity score of the left ear and the right ear, and construct a cross-modal contrastive loss when performing the cross-modal alignment;

[0052] a pre-training module, configured to construct an objective function according to the image reconstruction loss, the image contrastive learning loss and the cross-modal contrastive loss, so as to pre-train the diagnosis model through the objective function;

[0053] a diagnosis module, configured to input an MRI image of an ear of a current patient and a corresponding medical report into the pre-trained diagnosis model, classify and diagnose a fused bilateral image feature corresponding to the MRI image of the ear of the current patient through a linear classifier in the diagnosis model, and output a Meniere's disease diagnosis result.

[0054] To achieve the above object, the third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described above.

[0055] The present application has at least the following technical effects:

[0056] The present application provides a Meniere's disease diagnosis method and a corresponding image processing system, which are realized by image processing. The method is first to encode each MRI pixel and its local neighborhood into a high-dimensional embedding space within a clinically defined region of interest, so as to capture subtle structural changes. Secondly, in the self-supervised feature refinement stage of the image, a dual mechanism of image reconstruction and image contrast learning is adopted. By reconstructing adjacent pixels, the spatial dependence of fine inner ear structures is retained. By comparing the distribution of lesions and normal tissues, the sensitivity to minor abnormalities is improved. Finally, the global lesion representation of both ears is aggregated and aligned with the overall textual report description in a shared semantic space, so as to ensure the semantic consistency at the patient level, maintain the details of local lesions, and enhance the clinical interpretability of multi-modal fusion. The present application can realize accurate and efficient diagnosis of Meniere's disease through the overall scheme.

[0057] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is a flowchart of the image processing method of the embodiment of the present application.

[0059] Figure 2 is a working principle diagram of the image processing method of the embodiment of the present application.

[0060] Figure 3 is an ROC curve and probability distribution diagram of the embodiment of the present application.

[0061] Figure 4 is a structural block diagram of the image processing system of the embodiment of the present application. DETAILED DESCRIPTION

[0062] The present embodiment is described in detail below, and examples of the embodiment are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0063] An image processing system, method and storage medium of the present embodiment are described below with reference to the accompanying drawings.

[0064] The image processing method provided by the embodiment is in the aspect of imaging, an expert radiologist assists in identifying a region of interest related to Meniere's disease, and each pixel and its local neighborhood are encoded into a high-dimensional embedding space to preserve the subtle features of the lesion. In the aspect of text, the scheme also involves a severity assessment method that can extract the lesion description of both ears from a standardized image report, i.e., a medical report or a radiology report, thereby quantifying the bilateral pathological difference and enhancing asymmetric modeling. Secondly, MA, a self-supervised learning method, does not rely on image-level augmentation in many contrast learning methods, but learns from regions within the image. This design focuses on learning local small-scale features rather than maintaining invariance to global transformations, which is particularly important for medical images that are globally uniform but locally non-uniform. In addition, the present application also introduces a severity-aware global cross-modal alignment mechanism for connecting local lesion features with patient-level clinical semantics. This mechanism builds a differentiable shared semantic space by integrating global information, ensures that local pathological details are preserved by providing bilateral lesion images containing complete descriptions, and realizes semantic alignment between image and text modalities, thereby improving the explainability and consistency of multi-modal diagnostic decisions.

[0065] Figure 1 A flowchart of the image processing method of the embodiment of the present application. As shown in Figure 1 , the method comprises:

[0066] Step S101: Obtain MRI images of the ears of historical patients and corresponding medical reports.

[0067] Specifically, 524 inpatients diagnosed with Meniere's disease in the dizziness and hearing center of a university-affiliated hospital from January 2021 to December 2024 were included in the present study. The cohort included 199 male and 325 female patients, with an age range of 12 to 78 years (mean age ± standard deviation: 50.1 ± 11.9 years), and an onset age range of 12 to 78 years (mean age ± standard deviation: 46.1 ± 16.3 years). All patients showed typical symptoms of Meniere's disease, including dizziness and ear symptoms, specifically low to mid-frequency sensorineural hearing loss, with or without tinnitus and / or ear fullness. The diagnosis of endolymphatic hydrops (EH) was confirmed by inner ear imaging. Patients with other diagnoses, such as vestibular migraine, sudden sensorineural hearing loss, central vertigo, or benign paroxysmal positional vertigo, were excluded. All participants underwent intravenous gadolinium-enhanced inner ear imaging and 3D-FLAIR MRI (three-dimensional fluid-attenuated inversion recovery magnetic resonance imaging) and provided written informed consent. The study has been approved by the medical ethics committee of the university-affiliated hospital.

[0068] All patients received 3D-FLAIR MRI of the inner ear approximately four hours after intravenous injection of gadopentetate dimeglumine. Image data were independently evaluated by two experienced radiologists blinded to clinical information, focusing on endolymphatic hydrops within the vestibular and cochlear membranous labyrinth. Most patients showed varying degrees of endolymphatic hydrops. Through the clinical diagnostic procedure, the radiologists manually annotated 32x32 pixel regions of interest in the anatomical regions most relevant to typical Meniere's lesions, such as cochlear hydrops and vestibular dilation. These annotated regions encompassed substructures clinically prone to lesions, such as the cochlear basal turn and the lateral portion of the vestibule, rather than strictly following precise anatomical boundaries to ensure that the captured regions had practical diagnostic value.

[0069] In this study, MRI slices of patients diagnosed with Meniere's disease and their corresponding standardized image reports, i.e., medical reports, were utilized to investigate the relationship between radiological features and clinical symptoms, and a self-supervised image-text pre-training framework, i.e., a diagnostic model, was developed to learn multi-modal features to improve the ability to automatically analyze and distinguish MD-related pathological features, thereby improving their clinical relevance and interpretability.

[0070] Step S102: Annotate the region of interest on the MRI image, extract the image block of the region of interest, encode the extracted image block through the encoder, and generate left ear image features and right ear image features.

[0071] In a possible implementation, annotating the region of interest on the MRI image and extracting the image block of the region of interest include: manually annotating the region of interest on the MRI image using a bounding box of a preset size of pixels, wherein the region of interest includes a cochlear basal turn and a vestibular lateral region; and extracting an image block of a preset pixel size with an ear center point as the center.

[0072] Specifically, a ROI (region of interest) detection and encoding process based on radiologist annotations is outlined to ensure that these annotations are only used to delineate local structural regions, rather than directly providing diagnostic information. Two experienced radiologists independently annotated the bounding boxes using anonymized patient data. Since the clinical diagnosis of Meniere's disease often emphasizes specific anatomical regions, a pre-set size of pixels, i.e., a fixed 32x32 pixel bounding box, is used to define the region of interest, such as the cochlear base turn and the lateral vestibular area. This method can capture fine-grained structural and textural details that are crucial for subsequent analysis. After determining the bounding box, the horizontal and vertical coordinates of the center coordinates of each ear can be calculated as xcenter = x+w / 2 and ycenter = y+h / 2, respectively, where (x, y, w, h) represent the horizontal coordinate, vertical coordinate, width, and height of the top-left corner of the bounding box, respectively. Then, a patch, i.e., an image block, of a pre-set size of pixels, i.e., 32x32 pixels, is extracted from the center position (xcenter, ycenter), with the top-left corner and the bottom-right corner located at (xcenter-16, ycenter-16) and (xcenter+16, ycenter+16), respectively. If the patch exceeds the image boundary, zero padding is applied to ensure consistency of all extracted patches.

[0073] After ROI detection, as shown in FIG. 2, a convolutional neural network (CNN encoder) is used as a patch encoder to map the 32x32 image block into a latent embedding space, generating left ear image features and right ear image features, denoted as zl and zr, respectively. In this embodiment, the encoder contains convolutional layers with batch normalization and nonlinear activation functions, but does not perform pooling operations to maintain spatial consistency between the input image and the final feature map. This design effectively captures local structural details, allowing subsequent models to analyze these features while maintaining spatial relationships. Figure 2

[0074] It should be noted that the encoder encodes the image block extracted from the MRI image of the left ear and the right ear, and then generates the left ear image features and the right ear image features.

[0075] Step S103: Construct an image reconstruction loss based on the left ear image features and the right ear image features, and perform image reconstruction to ensure that there is no loss of lesion information in the encoded image features.

[0076] ​In one possible implementation, an image reconstruction loss is constructed based on the features of the left ear image and the right ear image, and image reconstruction is performed, including: reconstructing corresponding local image patches based on the features of the left ear image and the right ear image respectively through a linear transformation function; constructing a mean squared error loss function to measure whether the difference between the reconstructed local image patch and the corresponding original image patch is minimized, so that the encoded image features do not lose lesion information; wherein, the mean squared error loss function is expressed as the sum of squares of the differences between each pixel value between the reconstructed local image patch and the corresponding original image patch.

[0077] This embodiment introduces a local patch, or local image patch reconstruction constraint, to enhance the representation capability of small-scale image features. Considering that spatial dependencies between adjacent elements within a 32×32 pixel region are easily lost, an image reconstruction loss centered on pixel-level patches is designed. This constraint ensures that the encoder preserves local structural features in the latent embedding, thereby better maintaining fine-grained contextual information.

[0078] To enhance the representation of small-scale medical image regions, a local patch reconstruction constraint mechanism is introduced. This mechanism addresses the loss of spatial dependencies between adjacent elements in a 32×32 pixel region. By reconstructing pixel-centered patches, this mechanism encourages the encoder to preserve the structural features of local regions in the latent embedding space. Specifically, for lesion regions of the left and right ears, a local patch is constructed around each central pixel. The corresponding embedding vector is represented as z. ij By using the recon function, z ij Map back to the original pixel space. The reconstruction loss is defined as: P ij f represents the pixel features corresponding to the original patch. recon (z ij ) represents a linear projection layer. Where, f recon (z ij )=Wz ij +b, W, and b represent the learnable weights and biases, respectively. This image reconstruction mechanism ensures that the embedding vector preserves the local features of each patch and its surrounding neighborhood by minimizing the reconstruction error. In this embodiment, the encoder adopts a fully convolutional architecture without pooling operations, which maintains the spatial alignment between the input and output feature maps and avoids the loss of structural information due to downsampling. Therefore, the local structural reconstruction losses for the left and right ears are denoted as follows: and

[0079] Step S104: Construct an image contrast learning loss based on the anchor point features extracted from the MRI images, and perform image contrast learning so that the diagnostic model can identify lesions and normal individual variations.

[0080] In a possible implementation, an image contrast learning loss is constructed according to anchor point features extracted on the MRI image, and image contrast learning is performed, including: determining positive samples and negative samples according to anchor point features extracted on the MRI image; wherein samples whose spatial positions are in a preset neighborhood range of the anchor point features and whose semantic similarity degrees with the anchor point features are greater than a preset value are classified as the positive samples, and samples whose spatial positions are not in the preset neighborhood range of the anchor point features and whose semantic similarity degrees with the anchor point features are less than or equal to the preset value are classified as the negative samples; constructing an image contrast learning loss according to the anchor point features, the positive samples and the negative samples; adjusting an encoder parameter to minimize the image contrast learning loss, and obtaining the encoder at which the image contrast learning loss is minimized; wherein left ear image features and right ear image features after encoding by the encoder at which the image contrast learning loss is minimized can recognize lesion features or normal individual variation features.

[0081] Specifically, the proposed contrast learning framework enhances the recognition ability of the diagnostic model for local pathological features of Meniere's disease by imposing dual constraints of spatial proximity and deep semantic consistency. For the anchor patch centered at coordinates (i1, j1) The embedding features are extracted by the encoder as When constructing the positive sample set, strict spatial and semantic conditions must be met: the geometric center of the positive sample must be located within the anchor neighborhood N(i1, j1) = {(i1', j1') | |i1-i1'| ≤ p, |j1-j1'| ≤ p}, and its pathological similarity with the anchor point needs to be verified by deep image structure similarity (DISTS) measurement, using a threshold of δ DISTS = 0.5, where i1 and j1 are the horizontal and vertical coordinates of the anchor patch, i1' and j1' are the horizontal and vertical coordinates of other samples, p is a preset neighborhood value, and δ DISTS is a preset value. DISTS is based on multi-scale feature alignment of a pre-trained network and is very sensitive to subtle pathological changes such as endolymphatic space expansion, and is superior to traditional metrics in modeling local deformations.

[0082] The construction of negative samples follows complementary constraints to positive samples: candidate patches must be located outside the anchor neighborhood When N(i, j) or DISTS similarity is lower than a preset value These negative samples usually contain anatomical variations, such as physiological narrowing or spatially deviated irrelevant regions of the cochlear aqueduct, where represents the candidate patch. By imposing symmetry constraints between positive and negative samples, the diagnostic model is forced to distinguish between local pathological features, i.e., lesions, and normal anatomical variations, i.e., normal individual variations, thereby improving the clinical interpretability of the embedding space. The image contrast learning loss is defined as:

[0083]

[0084] wherein, denotes the image contrastive learning loss, Z + denotes the positive sample, Z - denotes the negative sample, Ω + denotes the positive sample set, Ω - denotes the negative sample set, e is a natural constant, sim() denotes the cosine similarity, and τ denotes the temperature coefficient.

[0085] Step S105: determining the specific severity scores of the left ear and the right ear according to the medical report, and performing cross-modal fusion and alignment between the image and the text features extracted from the medical report according to the specific severity scores of the left ear and the right ear, wherein a cross-modal contrastive loss is constructed when the cross-modal alignment is performed.

[0086] In one possible implementation, determining the specific severity scores of the left ear and the right ear according to the medical report comprises: determining entities in the medical report, the entities being key clinical entities, the entities including anatomical structures, pathological descriptions, and lateral indications, the lateral indications being left, right, and bilateral; calculating the similarity of the entities and the global context in the medical report, so as to determine the context weight of each entity according to the similarity and whether there is a negation word in the medical report; determining the base weight of each entity, determining whether there is a severe, moderate, or mild keyword in the medical report, and determining a severity level adjustment coefficient according to the keyword appearance result; determining the maximum weight value of each entity according to the base weight, the context weight, and the severity level adjustment coefficient of each entity, so as to determine the specific severity scores of the left ear and the right ear according to the base weight and the maximum weight value of each entity.

[0087] Although the standardized radiology reports, i.e., medical reports, of Meniere's disease ensure consistency in description, they often fail to quantitatively capture individual differences in lesion severity and asymmetry between the left and right ears. To solve this problem, the present embodiment proposes an interpretable severity scoring mechanism that extracts different pathological burdens from homogeneous text descriptions. This helps to guide the fine-grained semantics of cross-modal alignment between images and text, and enhances the ability of the model to distinguish unilateral and bilateral abnormalities.

[0088] Specifically, a named BioBERT-CRF entity recognition model is used to accurately identify key clinical entities in the report, including anatomical structures (such as cochlea, vestibule), pathological descriptions (such as edema, dilation), and lateral indications (such as left, right, bilateral). According to the annotation guidelines jointly formulated by two experienced otolaryngologists, a severity lexicon K is assigned to each entity a base weight w b(e), which strictly follows the core pathologic indicators defined in the American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS) clinical grading standards.

[0089] To capture the dynamic influence of the diagnostic context on the pathological features, the embodiment designs a context-aware weighting module based on BERT (a pre-trained language model). For an entity e, its dynamic context weight w c (e) is calculated as follows:

[0090]

[0091] where sim(e, c) represents the cosine similarity between the entity embedding t e and the global context embedding t c . The ReLU function (activation function) suppresses negative correlation through ReLU(x) = max(0, x), where x represents the cosine similarity between the entity embedding and the global context embedding. For example, if "edema" co-occurs with the highly relevant term "significant cochlear dilation" (e.g., sim(e, c) = 0.9), its context weight will increase to w c (e) = 1.9.

[0092] Further, a severity level adjustment coefficient is introduced, defined as:

[0093] γ(l) = 1.2 × δ l (severe) + 1.0 × δ l (moderate) + 0.8 × δ l (mild) (3)

[0094] where γ(l) represents the severity level adjustment coefficient, δ l (severe), δ l (moderate), and δ l (mild) represent parameters indicating the presence or absence of severe, moderate, and mild keywords, respectively.

[0095] Therefore, the final weight value w t (e) of the entity is calculated as: w t (e) = w b (e) × w c (e) × γ(l). To simulate the spatial distribution of lesions in Meniere's disease, the embodiment adopts an anatomical co-occurrence rule. For example, when an anatomical region (such as the cochlea) co-occurs with a pathological description (such as edema) in the same semantic unit, the correlation coefficient is set to δ(e) = 1.0. The side-specific severity score is calculated as follows:

[0096]

[0097] wherein E side represents a set of entities related to the side, represents a set of anatomical site entities under the corresponding side.

[0098] For bilateral expression (e.g. bilateral vestibular dilatation), according to anatomical research, 70% of the weight can be evenly distributed to both ears according to the update rule of E side ← E side ∪(E B ×0.7), E B represents a set of bilateral anatomical entities appearing in the report.

[0099] Finally, the normalized left and right scores are adjusted by a bilateral enhancement factor I bilateral (1 if there is a bilateral entity, otherwise 0), i.e. to determine whether there is at least one bilateral entity in the medical report, if there is, then the specificity severity scores of the left ear and the right ear are both increased by a preset percentage such as 30%, as shown below:

[0100] S l = S left ×(1+0.3×I bilateral ), S r = S right ×(1+0.3×I bilateral ) (5)

[0101] wherein S l , S r are the final specificity severity scores of the left ear and the right ear, respectively, S left , S right are the initial specificity severity scores of the left ear and the right ear, respectively, determined according to the base weight and the final weight value of each entity.

[0102] In a possible implementation, the cross-modal fusion between the image and the text features extracted from the medical report is performed according to the specificity severity scores of the left ear and the right ear, including: extracting text features from the medical report, projecting the left ear image features, the right ear image features and the text features into the same shared semantic space; calculating the semantic correlation between each text in the medical report and the local image block based on the projected features of the left ear image features, the right ear image features and the text features, so as to obtain the left ear multi-head attention output and the right ear multi-head attention output by weighted summation based on the semantic correlation and the corresponding text; modulating the multi-head attention output of the left ear and the right ear based on the specificity severity scores of the left ear and the right ear respectively, so as to focus on the more serious area of the lesion, and obtain the final left ear multi-head attention output and the right ear multi-head attention output; and performing feature fusion on the final left ear multi-head attention output and the right ear multi-head attention output, so as to obtain the fused bilateral image features.

[0103] In a possible implementation, the feature fusion is performed on the final left ear multi-head attention output and the right ear multi-head attention output, so as to obtain the fused bilateral image features, including: performing weighted summation on the final left ear multi-head attention output and the right ear multi-head attention output based on the specificity severity scores of the left ear and the right ear respectively to obtain preliminary fused features; determining an asymmetry factor based on the specificity severity scores of the left ear and the right ear; enhancing the preliminary fused features based on the asymmetry factor; and performing residual connection and layer normalization processing on the enhanced preliminary fused features to obtain the final fused bilateral image features.

[0104] Specifically, the embodiment proposes a severity-aware cross-ear attention fusion mechanism to simulate the heterogeneity of unilateral and bilateral lesions of Meniere's disease. Through cross-modal attention, the mechanism dynamically integrates imaging features from the left ear and the right ear and their corresponding text descriptions, thereby extracting discriminative representations for subsequent semantic alignment. For single-ear imaging features (left ear image features zl or right ear image features zr) and text features T, the two modalities are first projected into a shared latent space, i.e., a shared semantic space. Cross-modal interaction is then calculated through a multi-head attention mechanism:

[0105] Q = z l / r W Q ,K = TW K ,V = TW V

[0106]

[0107] wherein Q, K, V represent the target to be retrieved, the identifier stored in the database or memory, and the information content actually stored, respectively. l / rdenote left ear image features or right ear image features, T denotes text features, W Q , W K , and W V denote the first to third learnable projection matrices respectively, d denotes hidden dimension, denotes semantic correlation, Softmax is a mathematical function that converts a set of arbitrary real numbers into a probability distribution between 0 and 1, and the sum of all outputs is 1, and the generated multi-head attention output Attention(Q, K, V) is simply denoted as A, which captures the semantic correlation between the local imaging area and the clinical description. In order to highlight the key pathological area, the specific severity scores s l and s r modulate the multi-head attention outputs of the left ear and the right ear:

[0108] A′ l / r = A l / r ☉log(1+s l / r ) (7)

[0109] wherein A l / r denotes the multi-head attention output of the left ear or the right ear, S l / r denotes the specific severity score of the left ear or the right ear, and A′ l / r denotes the final multi-head attention output of the left ear or the right ear.

[0110] Here, the log(1+s) formula smooths the contribution of severity, avoiding the dominance of extreme values while maintaining the enhancement effect on clinically meaningful areas, wherein s is the specific severity score. Subsequently, according to the respective severity ratios, the final multi-head attention outputs of the left ear and the right ear A l ' and A r ' are adaptively fused:

[0111]

[0112] wherein a l , a r denote the weighting factors of the left ear and the right ear respectively, ε denotes a very small positive number, which prevents the denominator from being zero and avoids numerical instability, and Z Fusion1 is the preliminary fused feature.

[0113] In order to quantify the difference between the two ears, an asymmetry factor Δs = |s l -s r | is introduced. The asymmetry factor is used to enhance the preliminary fused feature, and the adjusted fused representation is:

[0114] z Fusion = z☉(1+Δs) (9)

[0115] This adjustment improves sensitivity to predominantly unilateral lesions (e.g., severe right ear hydrops with normal left ear), aligning with clinical concerns for asymmetric manifestations. Finally, training stability is enhanced by applying residual connections and layer normalization.

[0116] z' Fusion = LayerNorm(z Fusion + Dropout(A' l / r )) (10)

[0117] where z' Fusion is the final fused bilateral image feature, Dropout(A' l / r ) denotes a regularization function that applies dropout operation on the final left or right ear multi-head attention output to obtain a randomly masked feature vector, and LayerNorm denotes a layer normalization function.

[0118] In this embodiment, the severity score guided attention modulation and anatomy information fusion strategy can extract structured pathological semantics from standardized textual reports, i.e., medical reports, and provide quantitative representation of spatial lesion distribution, thereby supporting interpretable cross-modality diagnosis modeling.

[0119] In one possible implementation, performing cross-modality alignment includes: performing normalization processing on the final fused bilateral image feature and the text feature, and determining a cosine similarity matrix of the two after the normalization processing; determining a dynamic temperature coefficient according to the cosine similarity matrix of the two, the severity score specific to the left ear, and the severity score specific to the right ear; determining a positive similarity score between the final fused bilateral image feature and the corresponding text feature and other text features, respectively; constructing a cross-modality contrast loss according to the two positive similarity scores and the dynamic temperature coefficient, so as to guide the cross-modality alignment of the image and the text through the cross-modality contrast loss, and ensure that the image and the text match, the image being the final fused bilateral image.

[0120] Although local feature modeling (such as intra-image patch reconstruction and image contrast learning) can effectively capture subtle pathological clues in regions such as the cochlea and vestibule, the final diagnosis of Meniere's disease still highly depends on the global semantic consistency between multiple modalities. To solve this problem, the embodiment proposes a cross-modality contrast alignment strategy based on severity score. This method realizes effective semantic alignment of imaging and text features in a shared latent space by dynamically adjusting the strength of contrast constraints.

[0121] To adjust the temperature parameter of the image contrast loss and enhance the similarity between positive pairs, the severity score of endolymphatic hydrops specific to the two ears of a particular patient (s l, right ear is s r ) and its inter-aural difference, i.e., asymmetry factor As. This design improves the model's sensitivity to severe unilateral or highly asymmetric cases.

[0122] First, the final fused bilateral image features z Fusion and text features T are L2-normalized to eliminate the scale difference between different modalities. Subsequently, the cosine similarity matrix S between them is calculated according to the following formula:

[0123]

[0124] To simulate the rigorous review of severe or asymmetric pathologies in clinical assessment, a dynamic temperature coefficient τ i2 is defined:

[0125]

[0126] where τ0 is the base temperature parameter, β is the weighting factor of the inter-aural difference, As i2 are the specific severity scores of the left ear, right ear under the i2-th final fused bilateral image features or corresponding samples, and the difference measure between the left and right ear-specific severity scores. For severe or asymmetric cases, a lower τ i2 value will result in steeper gradients in the image contrast loss, thus forcing the alignment in the feature space to be more stringent.

[0127] At the same time, the positive similarity score is further enhanced by a non-linear boosting strategy:

[0128]

[0129] where S i2,i2 represents the cross-modal similarity between the i2-th final fused bilateral image features and the corresponding text features, γ l and γ r are the weighting factors of the left ear and right ear, respectively, controlling the similarity enhancement according to the severity score. This mechanism ensures that the model focuses more on the alignment of clinically important regional features. Subsequently, the patient-specific InfoNCE contrast loss, i.e., cross-modal contrast loss, is defined as:

[0130]

[0131] where S i2,j represents the cross-modal similarity between the i2-th final fused bilateral image features and the j-th text features, S i2,jThe diagonal elements in the cosine similarity matrix S represent positive pairings, while the off-diagonal elements correspond to negative pairings of different patients. B is the total number of image samples.

[0132] This loss utilizes temperature modulation and similarity enhancement, guiding the diagnostic model to achieve precise semantic alignment, especially in challenging pathological cases.

[0133] Step S106: Construct a target function according to the image reconstruction loss, image contrast learning loss, and cross-modal contrast loss, so as to pre-train the diagnostic model through the target function.

[0134] Specifically, a unified target function can be constructed by jointly optimizing image reconstruction, intra-modal image contrast learning, and cross-modal alignment tasks:

[0135]

[0136] wherein, denotes the unified target function described above, denotes the image reconstruction loss of the left ear and the right ear, respectively, denotes the image contrast learning loss of the left ear and the right ear, respectively, denotes the cross-modal contrast loss, λ cl denotes the coefficient for balancing the image reconstruction loss and the image contrast learning loss, λ gl denotes the coefficient for controlling the global alignment loss. The selection of these hyperparameters will be discussed in the experimental section.

[0137] Step S107: Input the MRI image of the current patient's ear and the corresponding medical report into the pre-trained diagnostic model, and classify the fused bilateral image features corresponding to the MRI image of the current patient's ear through the linear classifier in the diagnostic model to output the Meniere's disease diagnosis result.

[0138] The following Meniere's disease diagnosis experiment is used to verify the diagnostic model.

[0139] To validate the diagnostic performance of the proposed model (SMAM-MD), it was compared with several baseline models: ResNet (a classical image classification model based on deep residual network, extracting global image features through hierarchical convolution), SimCLR (a self-supervised contrastive learning framework, constructing instance discrimination tasks with image augmentation techniques to learn general visual representations), MoCo (a self-supervised learning method based on momentum contrast, increasing the number of negative samples through dynamic queue mechanism to optimize feature discrimination), CLIP (a large-scale pre-trained visual-linguistic alignment model, achieving semantic mapping between images and texts through cross-modal contrastive learning); ALBEF (an attention-based alignment model, integrating visual and linguistic modalities to enhance fine-grained semantic understanding through cross-modal interaction), ConVIRT (a medical image-text contrastive learning framework, optimizing cross-modal feature alignment reports for radiology), and MGCA (a contrastive learning method combined with medical anatomical priors, enhancing the anatomical validity of image representations through graph structure constraints). To ensure fair comparison, all baseline models were fine-tuned with the same linear classifier under a unified data preprocessing pipeline.

[0140] Table 1 Classification results of MD classification tasks

[0141]

[0142]

[0143] As shown in Table 1, to comprehensively evaluate the diagnostic performance, five widely recognized indicators were adopted: accuracy or precision (ACC) to measure the overall classification accuracy; sensitivity (SEN) to reflect the model's ability to identify positive cases (e.g., Meniere's disease patients) and control the risk of missed diagnosis; specificity (SPE) to assess the reliability of excluding negative cases (non-patients) and reducing the risk of misdiagnosis; F1 score to balance precision and recall, considering the proportion of false positives and false negatives; and the area under the receiver operating characteristic curve (AUC) to reflect the classification stability at different decision thresholds. All indicators are in the range of [0, 1], and the higher the value, the better the model's discrimination ability, clinical safety, and robustness.

[0144] The effectiveness of this approach is validated in the diagnosis task of Meniere's disease. Specifically, the labeled data is divided into training and validation sets, with a proportion p (from 10% to 40% in steps of 10%) to simulate the challenge of limited labeling resources in the real world. The remaining data is used as an unlabeled set. To evaluate the robustness of the model, a five-fold cross-validation strategy is adopted, i.e., the patient data is randomly divided, and the experiment is repeated five times. The final performance indicators are reported as the average and standard deviation. All experiments are conducted in the same hardware environment, using a fixed random seed to ensure the reproducibility of the results.

[0145] Table 1 shows the classification performance of various methods for the diagnosis task of Meniere's disease under different labeling proportions, with the best results highlighted in bold. All methods show a consistent trend of performance improvement as the labeling proportion p increases from 10% to 40%. Among them, the accuracy of the diagnosis model increases from 86.35% to 94.36%, and the AUC value improves significantly.

[0146] From 85.67% to 97.08%, the sensitivity (SEN) and specificity (SPE) increased by 5.28% and 14.03%, respectively, indicating that more labeled data significantly enhanced the model's ability to capture cochlear fluid features and vestibular-topological abnormalities. In contrast, single-modal baseline methods such as ResNet, SimCLR, and MoCo perform poorly under low labeling settings (p = 10%), with accuracy less than 83%. Multimodal methods show moderate improvements due to cross-modal alignment: general visual language models such as CLIP and ALBEF achieve accuracies of 81.17% and 82.36%, while medical-specific contrast methods such as ConVIRT and MGCA achieve accuracies of 83.63% and 85.00%, respectively. However, these results are still significantly lower than 86%. The diagnosis model (SMAM-MD) achieves 35%, highlighting the importance of fine-grained, domain-aware modeling in medical diagnosis tasks.

[0147] The diagnostic model significantly improves the diagnostic performance of Meniere's disease by synergistically integrating local pathological reconstruction, regional contrast learning, and global cross-modal alignment mechanisms. Experimental results show that under low annotation rate (10%), local pathological reconstruction enhances the model's sensitivity to subtle lesions by encoding pixel-level cochlear hydrops features, increasing specificity by 4.3% (up to 80.26%) compared to the baseline method. Regional contrast learning further reduces the risk of false negatives by capturing heterogeneous patterns between unilateral inner ear lesions and normal regions, achieving a sensitivity of 90.14% at a 20% annotation level. On this basis, the global cross-modal alignment module integrates anatomical-semantic associations between image lesions and text descriptions, constructing more discriminative cross-modal classification boundaries. Therefore, the model achieves an AUC of 97.08% at a 40% annotation rate, reducing false positives by 58.3%.

[0148] Compared with the baseline, the model performs well in cross-class prediction overlap. As shown in Figure 3 , ROC (Receiver Operating Characteristic) curve and probability distribution analysis show that the model maintains stable performance under different annotation ratios. In addition, 95% of multi-modal case prediction results fall within a high confidence interval (probability greater than 0.8), which verifies that the multi-modal collaboration framework optimizes the explainability and generalization ability of diagnosis by effectively integrating hierarchical features.

[0149] Further, the present application also provides an image processing system.

[0150] As shown in Figure 4 , the system sequentially includes an acquisition module, an encoding module, an image reconstruction module, a contrast learning module, a scoring module, a pre-training module, and a diagnosis module.

[0151] The image processing system comprises an acquisition module, an encoding module, an image reconstruction module, a contrast learning module, a scoring module and a pre-training module.

[0152] It should be noted that the specific implementation of the image processing system of the embodiment of the present application can refer to the specific implementation of the image processing method described above, and to avoid redundancy, it will not be described here.

[0153] Further, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the image processing method described above.

[0154] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0155] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and the various embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment or some parts of the embodiment.

[0156] Finally, it should be noted that the above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement methods and are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, include: Obtain historical MRI images of patients' ears and corresponding medical reports; The region of interest (ROI) on the MRI image is labeled, and image patches of the ROI are extracted. The extracted image patches are then encoded using an encoder to generate left ear image features and right ear image features. An image reconstruction loss is constructed based on the features of the left and right ear images, and image reconstruction is performed to ensure that no lesion information is lost in the encoded image features; An image contrast learning loss is constructed based on anchor point features extracted from MRI images, and image contrast learning is performed to enable the diagnostic model to identify lesions and normal individual variations. The specific severity scores of the left and right ears are determined based on the medical report. Cross-modal fusion and alignment between the image and text features extracted from the medical report are performed based on the specific severity scores of the left and right ears. A cross-modal contrast loss is constructed when performing cross-modal alignment. An objective function is constructed based on image reconstruction loss, image contrast learning loss, and cross-modal contrast loss, so as to pre-train the diagnostic model using the objective function; The MRI image of the patient's ear and the corresponding medical report are input into the pre-trained diagnostic model. The linear classifier in the diagnostic model classifies and diagnoses the fused bilateral image features corresponding to the MRI image of the patient's ear, and outputs the diagnosis result of Meniere's disease.

2. The method as described in claim 1, characterized in that, An image reconstruction loss is constructed based on the features of the left and right ear images, and image reconstruction is performed, including: Local image patches were reconstructed from the features of the left and right ear images using a linear transformation function, respectively. A mean squared error loss function is constructed to measure whether the difference between the reconstructed local image patch and the corresponding original image patch is minimized, so that the encoded image features do not lose lesion information; wherein, the mean squared error loss function is expressed as the sum of squares of the differences between each pixel value between the reconstructed local image patch and the corresponding original image patch.

3. The method as described in claim 1, characterized in that, An image contrast learning loss is constructed based on anchor point features extracted from MRI images, and image contrast learning is performed, including: Positive and negative samples are determined based on anchor point features extracted from MRI images. Among them, samples whose spatial location from the anchor point feature is within a preset neighborhood and whose semantic similarity to the anchor point feature is greater than a preset value are classified as positive samples, while samples whose spatial location from the anchor point feature is not within the preset neighborhood and whose semantic similarity to the anchor point feature is less than or equal to a preset value are classified as negative samples. Construct an image contrast learning loss based on anchor point features, positive samples, and negative samples; The encoder parameters are adjusted to minimize the image contrast learning loss, resulting in an encoder with the minimum image contrast learning loss. The left and right ear image features encoded by the encoder with the minimum image contrast learning loss can identify lesion features or normal individual variation features.

4. The method as described in claim 1, characterized in that, Specific severity scores for the left and right ears were determined based on medical reports, including: Identify entities in the medical report, the entities being key clinical entities, the entities including anatomical structures, pathological descriptions, and lateral indications, the lateral indications being left, right, and bilateral; Calculate the similarity between an entity and the global context in a medical report, so as to determine the context weight of each entity based on the similarity and the presence of negative words in the medical report; Determine the basic weight of each entity, determine whether there are severe, moderate, and mild keywords in the medical report, and determine the severity level adjustment coefficient based on the keyword occurrence results; The final weight value for each entity is determined based on its base weight, context weight, and severity level adjustment factor, so as to determine the specific severity score for the left and right ears based on the base weight and final weight value of each entity.

5. The method as described in claim 4, characterized in that, After determining the specific severity scores for the left and right ears, the method further includes: determining whether there is at least one bilateral entity in the medical report; if so, increasing the specific severity scores for both the left and right ears by a preset percentage.

6. The method as described in claim 1, characterized in that, Cross-modal fusion between images and text features extracted from medical reports was performed based on specific severity scores for the left and right ears, including: Text features are extracted from medical reports, and left ear image features, right ear image features, and text features are projected into the same shared semantic space. The semantic correlation between local image patches and text in the medical report is calculated based on the projected features of the left ear image features, the right ear image features, and the text features. Then, the left ear multi-head attention output and the right ear multi-head attention output are obtained by weighted summation based on the semantic correlation and the corresponding text. Based on the specific severity scores of the left and right ears, the multi-head attention output of the left and right ears was modulated to focus on the more severe areas of the lesion, and the final multi-head attention output of the left and right ears was obtained. Feature fusion is performed on the final multi-head attention outputs of the left and right ears to obtain fused bilateral image features.

7. The method as described in claim 6, characterized in that, Feature fusion is performed on the final left-ear multi-head attention output and right-ear multi-head attention output to obtain fused bilateral image features, including: The preliminary fused features were obtained by weighting and summing the final multi-head attention outputs of the left and right ears based on the specific severity scores of the left and right ears, respectively. Asymmetry factors were determined based on specific severity scores for the left and right ears; Enhanced features after initial fusion based on asymmetric factors; The enhanced initial fused features are subjected to residual connection and layer normalization to obtain the final fused bilateral image features.

8. The method as described in claim 7, characterized in that, Performing cross-modal alignment includes: The final fused bilateral image features and text features are normalized, and the cosine similarity matrix between the two is determined after normalization. A dynamic temperature coefficient is determined based on the cosine similarity matrix of the two ears and the specific severity scores of the left and right ears. Determine the positive similarity scores between the final fused bilateral image features and their corresponding text features and other text features; A cross-modal contrast loss is constructed based on two positive similarity scores and a dynamic temperature coefficient to guide cross-modal alignment of images and text, ensuring image and text matching. The image is the final fused two-sided image. The objective function, constructed based on cross-modal contrast loss, image reconstruction loss, and image contrast learning loss, is expressed as follows: Among them, l total Describe the objective function. These represent the image reconstruction losses for the left and right ears, respectively. Let l represent the image contrast learning loss for the left and right ears, respectively. cross λ represents the cross-modal contrast loss. cl λ represents the coefficient used to balance the image reconstruction loss and the image contrast learning loss. gl This represents the coefficient used to control the global alignment loss.

9. An image processing system, characterized in that, include: The acquisition module is used to acquire historical MRI images of patients' ears and corresponding medical reports; The encoding module is used to annotate the region of interest on the MRI image and extract image patches of the region of interest. The encoder encodes the extracted image patches to generate left ear image features and right ear image features. The image reconstruction module is used to construct an image reconstruction loss based on the features of the left ear image and the right ear image, and to perform image reconstruction so that the encoded image features do not lose lesion information; The contrast learning module is used to construct an image contrast learning loss based on anchor point features extracted from MRI images and to perform image contrast learning so that the diagnostic model can identify lesions and normal individual variations. The scoring module is used to determine the specific severity scores of the left and right ears based on the medical report, and to perform cross-modal fusion and alignment between the image and text features extracted from the medical report based on the specific severity scores of the left and right ears, wherein a cross-modal contrast loss is constructed when performing cross-modal alignment; A pre-training module is used to construct an objective function based on image reconstruction loss, image contrast learning loss, and cross-modal contrast loss, so as to pre-train the diagnostic model using the objective function; The diagnostic module is used to input the current patient's ear MRI image and corresponding medical report into the pre-trained diagnostic model. The linear classifier in the diagnostic model classifies and diagnoses the fused bilateral image features corresponding to the current patient's ear MRI image to output the diagnosis result of Meniere's disease.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images