Highly myopia retina image classification method and system fused with multi-modal information

Abstract

The invention discloses a multi-modal information fused high-myopia retina image classification method and system. The method comprises the following steps: acquiring an OCT image of a to-be-classified high-myopia retina and structured numerical data corresponding to the image; the OCT image is preprocessed, the preprocessed image is input into an image encoder for high-dimensional semantic representation learning, and overall semantic vector representation of the image is obtained; converting the structured numerical data into a medical language description text, and inputting the medical language description text into a text encoder for deep semantic modeling to obtain overall semantic vector representation of the text; inputting to a multi-modal fusion module, carrying out feature interaction and fusion through a bidirectional cross attention mechanism, and generating a fused multi-modal feature; and outputting a retina splitting stage category corresponding to the OCT image through a classification module. According to the method, the OCT image, the structured numerical data and the split staging definition text are utilized to perform multi-modal feature fusion, so that the accuracy of image staging recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image analysis, and more particularly to a highly myopic retina image classification method and system fusing multi-modal information. BACKGROUND

[0002] Ophthalmic diseases have become an important public health problem worldwide, leading to vision loss and decreased quality of life. High myopia significantly increases the risk of patients developing retinal-related complications, such as myopic macular schisis and retinal detachment.

[0003] Optical coherence tomography (OCT) as a non-invasive imaging technique can visualize the layered structure of the retina with high resolution; some studies have attempted to use intelligent diagnostic systems to identify OCT images. A patent discloses a cloud-based retinal OCT recognition intelligent diagnostic system based on a hardware and software platform, which realizes lesion classification by using deep learning methods on OCT image data; another patent discloses an ophthalmic disease analysis method based on deep learning.

[0004] Although significant progress has been made in medical image analysis technology, existing methods still have significant shortcomings in multi-modal information fusion, mainly in the weak modeling ability of structured information. Most current methods still use images as the only input modality, making it difficult to fully utilize structured numerical information of patients, resulting in a lack of complete discriminant basis for the model when dealing with complex or ambiguous cases, thereby affecting the accuracy of staging.

[0005] In addition, in practical applications, medical records are often incomplete or missing, and the semantic expression of the report content is significantly heterogeneous due to being written by different people, making it difficult to form a unified feature representation, which seriously affects the generalization ability and robustness of the model in cross-center or complex scenarios.

[0006] Therefore, it is a problem that needs to be solved by those skilled in the art to provide a multi-modal diagnostic model that can efficiently integrate image and structured numerical information to improve the accuracy of automatic staging of highly myopic retinal schisis. SUMMARY

[0007] Therefore, the present application provides a highly myopic retina image classification method and system fusing multi-modal information, which fuses image and text information through a bidirectional cross-attention mechanism to obtain the staging category of retinal schisis.

[0008] To achieve the above purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a highly myopic retina image classification method fusing multi-modal information, comprising the following steps: Obtaining an OCT image of a high myopia retina to be classified and structured numerical data corresponding to the image; the structured numerical data includes demographic information and eye measurement indicators; Preprocessing the OCT image and inputting the preprocessed image into an image encoder for high-dimensional semantic representation learning to obtain an overall semantic vector representation of the image; Converting the structured numerical data into a medical language description text and inputting the text into a text encoder for deep semantic modeling to obtain an overall semantic vector representation of the text; Inputting the overall semantic vector representations of the image and the text into a multi-modal fusion module, interacting and fusing the features to generate fused multi-modal features; Classifying the multi-modal features by a classification module to output a retinoschisis staging category corresponding to the OCT image.

[0009] Further, it also includes: Building a staging definition memory matrix to provide semantic prior constraints for calculating a staging memory guided contrast loss; Based on the staging definition memory matrix, a contrast learning mechanism is used to calculate the similarity scores between the overall semantic vector of the OCT image and the semantic vectors of each stage, and the similarity scores of the same stage samples are maximized and the similarity scores of different stage samples are minimized as constraints to realize the alignment of the image semantic vector and the stage semantic vector; Building a combined loss function to fuse the FocalLoss classification loss and the staging memory guided contrast loss, and optimizing the model parameters in the process of training the entire multi-modal retinoschisis staging model.

[0010] Further, the preprocessing of the OCT image includes: Performing size normalization, image enhancement, pixel value normalization and format conversion on the OCT image; Wherein, the image enhancement includes random horizontal flipping, random rotation within a preset angle range, brightness and contrast adjustment, Gaussian blur and region occlusion operation.

[0011] Further, the image processing process of the image encoder includes: Dividing the input image into image blocks, embedding each image block into a feature space through linear mapping, adding position encoding information to each image block, and inserting a classification mark at the beginning of the sequence to summarize the overall semantic information of the image; Inputting the image block vector sequence containing the classification mark into a multi-layer BEiT encoder for deep semantic feature extraction; wherein each layer of the BEiT encoder includes a multi-head self-attention mechanism, a residual connection, a layer normalization and a feedforward neural network; The vector representation containing the classification label is extracted from the output of the last BEiT encoder as the overall semantic vector representation of the image.

[0012] Further, the process of converting the structured numerical data into medical language description text specifically includes: Field standardization and cleaning of the structured numerical data are performed. Natural language concatenation, completion, embedding, and tokenization are performed through a natural language template to output a description text with medical language.

[0013] Further, the process of text processing performed by the text encoder specifically includes: The input text is segmented into word or subword sequences, and each word or subword is mapped to a unique identifier. Each identifier is concatenated with positional encoding information to form an input sequence, and a CLS identifier is added at the beginning of the sequence to input the BERT encoder. The BERT encoder encodes the input sequence to obtain a hidden feature vector for each identifier, and the feature vector corresponding to the CLS identifier is taken as the overall semantic vector representation of the text.

[0014] Further, the multi-modal fusion module is composed of multiple layers of bidirectional cross-attention layers; each bidirectional cross-attention layer is composed of two single-direction cross-attention blocks with opposite order and independent parameters. The single-direction cross-attention block takes either image or text as Query and the other as Key / Value; cross-modal response is calculated to obtain cross-modal enhanced features, which are adaptively fused with the original features through gated residual to obtain enhanced image or text semantic features. The enhanced image or text semantic features are respectively subjected to average pooling operation and concatenated to obtain a global multi-modal feature.

[0015] Further, the multi-modal feature is classified by the classification module to output the retinoschisis staging category corresponding to the OCT image; specifically including: The multi-modal feature is classified by a linear layer and a Softmax activation function to output S0-S4 results corresponding to normal retina and retinoschisis staging, respectively. Wherein, S0 represents the no macular retinoschisis stage; S1 represents the extrafoveal focal schisis stage; S2 represents the schisis stage extending to the fovea from the fovea; S3 represents the full-layer schisis stage involving the entire macula; and S4 represents the stage of macular detachment from full macular schisis.

[0016] Further, the construction process of the staging definition memory matrix includes: The five types of retinal split staging definition texts are input into the text encoder respectively to extract semantic vectors corresponding to the five staging definitions respectively; and the five semantic vectors are spliced to form a staging definition memory matrix; The five types of retinal split staging definition texts comprise: The stage of no macular retinal split indicates that the high myopia eye has not yet formed a split, which is used as a baseline stage to distinguish between eyes with complete structure and eyes with early split changes; The stage of focal split outside the fovea indicates that the split is limited to the area outside the fovea, and the fovea itself is not involved, which is the earliest detectable stage in high myopia; The stage of split involving the fovea and extending outside the fovea involves both the fovea and the peripheral area, but has not yet involved the entire macula; The stage of full-layer split involving the entire macula indicates that the split spans the entire macular area and involves multiple layers of the retina; The stage of progression from full-macular split to macular detachment also includes foveal split with layered or full-layered macular holes, even if full-macular split does not occur.

[0017] In a second aspect, the present application provides a high myopia retinal image classification system that fuses multi-modal information, comprising the following units: An acquisition unit is configured to acquire an OCT image of a high myopia retina to be classified and structured numerical data corresponding to the image; the structured numerical data comprises demographic information and eye measurement indicators; An image processing unit is configured to pre-process the OCT image and input the pre-processed image into an image encoder for high-dimensional semantic representation learning to obtain an overall semantic vector representation of the image; A text processing unit is configured to convert the structured numerical data into medical language description text and input the text into a text encoder for deep semantic modeling to obtain an overall semantic vector representation of the text; A fusion unit is configured to input the overall semantic vector representations of the image and the text into a multi-modal fusion module to perform feature interaction and fusion through a bidirectional cross-attention mechanism to generate fused multi-modal features; A classification unit is configured to classify the multi-modal features through a classification module to output a retinal split staging category corresponding to the OCT image.

[0018] Further, it further comprises: A memory matrix construction unit is configured to construct a staging definition memory matrix to provide semantic prior constraints for calculating a staging memory-guided contrast loss; Loss building unit: used to construct a combined loss function, which integrates FocalLoss classification loss and the staged memory-guided contrastive loss to optimize model parameters during the training of the entire multimodal retinal schisis staged model.

[0019] Furthermore, the image processing unit includes: an image preprocessing subunit and an image encoder processing subunit; The image preprocessing subunit is used to perform size normalization, image enhancement, pixel value normalization, and format conversion on the OCT image; the image enhancement includes random horizontal flipping, random rotation within a preset angle range, brightness and contrast adjustment, Gaussian blur, and region occlusion operations. The image encoder processing subunit is used to segment the image processed by the image preprocessing subunit into image blocks, embed each image block into a 3D feature space through linear mapping, add position encoding information to each image block, insert a classification label at the beginning of the sequence, and summarize the semantic information of the whole image. The sequence of image patch vectors containing classification labels is input into a multi-layer BEiT encoder for deep semantic feature extraction; each layer of the BEiT encoder contains a multi-head self-attention mechanism, residual connections, layer normalization, and a feedforward neural network. Extract the vector representation containing the classification label from the output of the last BEiT encoder layer as the overall semantic vector representation of the image.

[0020] Furthermore, the text processing unit includes: a text conversion processing subunit and a text encoder processing subunit; The text conversion processing unit performs field standardization and cleaning on the structured numerical data; it is also used to perform natural language concatenation, completion, embedding, and tokenization using natural language templates to output descriptive text with medical terminology. The text encoder processing subunit is used to segment the text converted by the text conversion processing subunit into word or subword sequences, and map each word or subword to a unique identifier; Each identifier is concatenated with the location encoding information to form an input sequence, and a CLS identifier is added at the beginning of the sequence before inputting it into the BERT encoder. The BERT encoder encodes the input sequence to obtain the hidden feature vector of each identifier, and uses the feature vector corresponding to the CLS identifier as the overall semantic vector representation of the text.

[0021] Furthermore, the multimodal fusion module used in the fusion unit is composed of multiple layers of bidirectional cross-attention layers stacked together; each bidirectional cross-attention layer is composed of two unidirectional cross-attention blocks with opposite order and independent parameters; The unidirectional cross attention block takes image or text modal features as Query, and the other modal features as Key / Value; cross-modal response is calculated to obtain cross-modal enhanced features, and the original features are adaptively fused with the gated residual to obtain enhanced image or text semantic features; The enhanced image or text semantic features are respectively subjected to average pooling operation and spliced and fused into a global representation of multi-modal features.

[0022] Further, the classification unit is specifically configured to: classify the multi-modal features through a linear layer and a Softmax activation function, and output S0-S4 results corresponding to normal retinas and retinal splitting stages, respectively; wherein, S0 represents a no macular retinal splitting stage; S1 represents a focal splitting stage outside the fovea; S2 represents a splitting stage involving the fovea and extending outside the fovea; S3 represents a full-layer splitting stage involving the entire macula; and S4 represents a stage of macular detachment progressing from full macular splitting.

[0023] Further, the memory matrix construction unit is specifically configured to: input five types of retinal splitting stage definition texts into the text encoder, respectively extract semantic vectors corresponding to the five stages of definition, and splice the five semantic vectors to form a stage definition memory matrix; The five types of retinal splitting stage definition texts include: a no macular retinal splitting stage, which indicates that a high myopia eye has not yet formed splitting, as a baseline stage, for distinguishing between eyes with complete structure and eyes with early splitting changes; a focal splitting stage outside the fovea, which is limited to the area outside the fovea, and the fovea itself is not involved, which is the earliest detectable stage in high myopia; a splitting stage involving the fovea and extending outside the fovea, which involves both the fovea and the peripheral area, but has not yet involved the entire macula; a full-layer splitting stage involving the entire macula, which involves multiple layers of retina; a stage of macular detachment progressing from full macular splitting, which also includes foveal splitting with layered or full-layer macular holes, even if there is no full macular splitting.

[0024] According to the above technical solution, compared with the prior art, the application provides a multi-modal information fusion high myopia retinal image classification method and system, which has the following beneficial effects: The application adopts BEiT as an image encoder, adopts BERT as a text encoder, has deep semantic modeling capability based on a Transformer architecture and self-supervised pre-training, thereby obtaining local semantic features and context semantic information, and a double-branch feature extraction architecture is constructed, so that the image features and the text features are in an alignable semantic space, laying a foundation for subsequent multi-modal fusion.

[0025] The application introduces a contrast learning mechanism to enhance the cross-modal representation capability, introduces a multi-modal contrast loss, learns the modality consistency through image-text positive-negative contrast samples in the training stage, improves the semantic alignment capability of cross-domain staging decision, and provides a more general modeling basis for complex multi-modal medical tasks. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only a part of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0027] Figure 1 The fusion visual and text multi-modal high myopia retina image classification method flow chart provided by the embodiment of the present application.

[0028] Figure 2 The fusion visual and text multi-modal high myopia retina image classification model structure diagram provided by the embodiment of the present application.

[0029] Figure 3 The fusion visual and text multi-modal high myopia retina image classification system block diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0031] Embodiment 1 The embodiment of the present application discloses a high myopia retina image classification method fusing multi-modal information, referring to Figure 1 As shown in the figure, the method comprises the following steps: obtain an OCT image of a high myopia retina to be classified and structured numerical data corresponding to the image; the structured numerical data includes demographic information and eye measurement indicators; preprocess the OCT image, and input the preprocessed image into an image encoder to perform high-dimensional semantic representation learning, to obtain an overall semantic vector representation of the image; convert the structured numerical data into a medical language description text, and input the text into a text encoder to perform deep semantic modeling, to obtain an overall semantic vector representation of the text; input the overall semantic vector representations of the image and the text into a multi-modal fusion module, perform feature interaction and fusion through a bidirectional cross-attention mechanism, and generate a fused multi-modal feature; classify the multi-modal feature through a classification module, and output a retinoschisis staging category corresponding to the OCT image.

[0032] The embodiment is applied to an auxiliary diagnosis system of a medical device, and realizes a full-process closed loop from "image acquisition" to "intelligent diagnosis suggestion". After the OCT image, patient data and medical knowledge are obtained in the embodiment, the encoder is used for processing, and the bidirectional cross-attention mechanism is used for feature interaction and fusion. Finally, not only the staging result is given, but also an explainable basis is provided.

[0033] The implementation process of the embodiment will be described in detail below.

[0034] In step S1, an OCT image of a high myopia retina to be classified and structured numerical data corresponding to the image are obtained.

[0035] The embodiment receives a retinal tomographic image obtained by an OCT device, adjusts the image size to be 256 high x 512 wide, and maintains the consistency of the tomographic spatial structure.

[0036] The structured numerical data of the patient generated by the template conversion mode is also received. The structured numerical data includes demographic information and eye measurement indicators. Referring to Figure 2 As shown in the figure, the clinical assessment of this middle-aged (more than 40) male (age: 47) shows that the axial length is all in the severe long range, which is 29.4 mm, and the visual acuity level is good (VA = 0.3).

[0037] In step S2, the OCT image is preprocessed, and the preprocessed image is input into an image encoder to perform high-dimensional semantic representation learning, to obtain an overall semantic vector representation of the image.

[0038] To improve the robustness and generalization ability of the model to the retinal OCT image, uniform preprocessing and enhancement operations are performed on the image after size adjustment in the data input stage, that is, OCT image preprocessing, which includes the following steps: 1. Size normalization and resolution standardization All input OCT images are adjusted to a fixed size of 256x512 by scaling or padding to ensure consistent model input dimensions and preserve the original image aspect ratio. This embodiment uses equal scaling followed by center cropping or edge padding.

[0039] 2. Image augmentation strategy A series of random augmentation operations are introduced during the training phase to improve the model's robustness to image quality variations, slight structural misalignments, and other real-world scenarios. The augmentation strategies include but are not limited to: 1) Random horizontal flip: This embodiment flips the image with a certain probability to simulate the symmetry of left and right eye structures.

[0040] 2) Random rotation within a preset angle range: This embodiment randomly rotates within ±10° to simulate slight shifts in imaging angles.

[0041] 3) Brightness and contrast adjustment: Used to enhance the ability to adapt to images under different lighting conditions.

[0042] 4) Gaussian blur or noise disturbance: Used to improve fault tolerance to imaging blur or artifact interference.

[0043] 5) Regional occlusion operation: Used to enhance learning ability for local missing regions and suppress overfitting.

[0044] 3. Pixel value normalization The enhanced images are processed by mean-standard deviation normalization per channel, converting them to zero-mean, unit-variance distribution to accelerate convergence and improve numerical stability.

[0045] Specifically, each pixel value is normalized as follows: where, represents the normalized pixel value, represents the original pixel value before normalization, represents the pixel mean value of the corresponding channel, represents the pixel standard deviation of the corresponding channel.

[0046] 4. Format conversion Convert the image from PIL format or OpenCV format to PyTorch Tensor and adjust the dimension order to (channel number C, height H, width W) to adapt to the image block embedding module of the Transformer visual encoder.

[0047] The embodiment adopts a BEiT model as an image encoder to perform high-dimensional semantic representation learning on an input OCT image. The BEiT is a visual encoder based on a Transformer architecture, which draws on the pre-training ideas of BERT in natural language processing, and combines an image block-level mask token prediction task, and has good image context modeling capability. Compared with a traditional convolutional neural network, the BEiT can capture long-distance spatial dependency, and is suitable for fine-grained feature extraction requirements in medical images.

[0048] The input image in the embodiment is adjusted to a size of in the preprocessing stage, where 3 represents the number of RGB channels C, and 256 and 512 are the height H and width W of the image respectively. Then the image is cut into patches of a fixed size, and the patch side length is taken as (that is, 16x16), and a total of image blocks are obtained.

[0049] The first image block is flattened as ( ) where P represents the image block side length, and N represents the total number of image blocks.

[0050] The encoding process of the BEiT encoder includes the following three stages: 1. Patch embedding process.

[0051] Each image block (patch) is embedded into a feature space with a dimension of = 1024 through linear mapping. Let be a linear embedding matrix, be the position embedding of the image block.

[0052] is placed at the beginning of the sequence, and its initial vector is denoted as . The initial representation of the image block is

[0053] and an initial sequence containing the first position aggregation mark is obtained

[0054] where represents the pixel vector of the image block after flattening, represents the semantic vector of the image block in dimensions, which is obtained by adding the content projection and the position embedding.

[0055] 2. Transformer encoder encoding process.

[0056] All vectors are sent into the encoder stacked by layers. Each layer adopts a two-step structure as follows: first, layer normalization is performed on the output of the previous layer, and then global dependency modeling is performed using multi-head self-attention to obtain the context-enhanced intermediate representation and add the input residual; then layer normalization is performed on , and the token-by-token nonlinear transformation is performed by the feedforward network, and the residual is added to obtain the output of the current layer. The multi-head self-attention adopts attention heads, and the single-head dimension is . In the entire encoding process, the sequence length remains consistent with the input, and the spatial correspondence is not destroyed, thereby forming a high-dimensional context fusion representation in the channel dimension.

[0057] 3. Global feature vector extraction.

[0058] The output of the first layer of the BEiT encoder is denoted as , which is also used as the global feature vector and the overall semantic vector representation of the image.

[0059] Finally, the image encoder outputs the global feature vector , which is used as the input of the subsequent multi-modal fusion module. BEiT has significant advantages over traditional CNNs in encoding long-range spatial dependencies and structural integrity, and is suitable for modeling complex hierarchical structures and pathological features in retinal OCT images.

[0060] Step S3: Convert the structured numerical data into medical language description text, and input the text into the text encoder for deep semantic modeling to obtain the overall semantic vector representation of the text.

[0061] This embodiment constructs a structured information natural language transcription module, which converts structured data including patient basic information (such as age, gender) and eye measurement indicators (such as axial length, visual acuity, etc.) into medical language description text in a unified style to adapt to the input requirements of the pre-trained language model. The data processing process of this module has the following core steps: 1. Field standardization and cleaning.

[0062] The original structured table data is subjected to field screening and validity verification, and missing value abnormal items are removed. The fields include: age, gender, axial length, and best corrected visual acuity.

[0063] 2. Semantic template construction.

[0064] According to domain knowledge and language habits, a natural language template is constructed to embed structured fields into a clinical-style description sentence. For example: "The clinical assessment of the {age group} {gender} patient shows that the axial length is {axial length group} mm, and the visual acuity level is {visual acuity group}." 3. Natural language concatenation and completion.

[0065] This embodiment supports multi-field splicing and redundant field elimination, and dynamically generates complete sentences according to the existence of fields. If some information is missing, the template sentence corresponding to the field will be automatically skipped to ensure the coherence of the sentence.

[0066] 4. Embedding and Tokenization. In this embodiment, the templated text is sent to a large language model. The model splits and encodes the text, and aggregates the global semantic representation of the entire text to be used for subsequent multi-modal alignment.

[0067] The following is an example output: Input structured data: age: "47 years old", gender: "male", axial length: "29.4 mm", visual acuity: "0.3", generate natural language text: "The clinical assessment of the middle-aged male (age: 47 years old) patient shows that the axial length is 29.4 mm, and the visual acuity level is 0.3." This embodiment uses BERT as a text encoder to model the deep semantics of the descriptive text after the structured information is transformed. BERT is a bidirectional Transformer language model that can model the dependency between any two words in the context, making it suitable for encoding tasks in the medical field with long and semantically dense description information.

[0068] The text processing process of the text encoder includes: 1. Tokenization and embedding.

[0069] The input text is first processed by the tokenizer to obtain a subword-level sequence . Each token is embedded into a dimensional vector space and added to the position encoding to form the input sequence.

[0070] Its initial representation is:

[0071] where represents the vector found from the word embedding matrix, is the position encoding.

[0072] 2. Encoding and output.

[0073] Subsequently input The layer BERT encoder obtains an output sequence.

[0074] The semantic representation of the entire text is:

[0075] The BERT encoder ensures that the text features can accurately express semantic content such as age, visual acuity level, and axial length in structured information, providing a language foundation for cross-modal alignment.

[0076] Step S4, input the overall semantic vector representation of the image and the text into the multi-modal fusion module, and perform feature interaction and fusion through a bidirectional cross-attention mechanism to generate a fused multi-modal feature.

[0077] To achieve deep alignment of image features and text semantics, the present embodiment constructs a bidirectional cross-attention module that uses the multi-head attention mechanism in Transformer to explicitly build a graph between the image and text modalities. The module is stacked with several layers of symmetric "bidirectional fusion layers"; each layer contains two single-direction cross-attention blocks with opposite directions and independent parameters, with one end modality as the query end (Q) and the other end modality as the key / value end (K / V) to calculate the cross-modal response, and use a gated residual to adaptively fuse the "original features" and "cross-modal enhanced features".

[0078] The data processing process of the single-direction cross-attention block is as follows: The image and text sequence features from the encoder are first linearly projected to a unified dimension . Let the query end feature in this direction be , and the key / value end feature be . Among them, is the query end sequence length (such as the number of image blocks or the number of text tokens ), is the key / value end sequence length; Cross-modal attention: assuming that the query end needs to be supplemented by the other modality, multi-head attention is used to retrieve key information from the other modality:

[0079] The attention output is denoted as . Among them, is the number of heads of multi-head attention, .

[0080] Gated residual: the query end original feature is fused with the cross-modal attention output After performing global statistics and splicing, the fusion coefficients are obtained by passing the data through two layers of multilayer perceptrons connected to a Sigmoid gated network. Then use In "reservation" "and "introduction" The two signals are adaptively weighted and then normalized at the end to obtain the output in that direction. This allows for dynamic adjustment of the weights of the two signals based on data quality, suppressing noise and modal bias, and improving the stability and robustness of the fusion process.

[0081] The data processing procedure for the bidirectional fusion structure is as follows: A bidirectional fusion layer consists of two unidirectional cross-attention blocks in opposite order, with normalization / residual processing performed between the two blocks. The image block representation output by BEiT is projected from 1024 dimensions to 768 dimensions to obtain... After aligning the dimensions of image and text features, iterative bidirectional fusion is performed: Image → Text (image as query, text as key / value): Calculate cross-modal attention, then adjust the weights using gated residuals to obtain the image features supplemented with text. .

[0082] Text → Image (text as query, image as key / value): The interaction is performed similarly, followed by fusion and normalization to obtain the text features supplemented by the image. .

[0083] The aforementioned two-way interactive modules can be stacked multiple times to achieve deeper levels of text and image alignment. The final output... and These are used as the enhanced semantic features of the image and text, respectively, and then input into the subsequent fusion prediction module.

[0084] Finally, feature fusion and output are performed.

[0085] Image and text features after cross-enhancement and Average pooling was used respectively Merge into a global representation vector:

[0086] The combination of the two is used as a unified representation. It is used for downstream multi-class prediction.

[0087]

[0088] Step S5: Classify the multimodal features through the classification module and output the retinal schisis stage category corresponding to the OCT image.

[0089] The joint feature vector in this embodiment After a linear layer and a Softmax activation function, the prediction stage is obtained , is the number of categories, corresponding to normal S0 and retinal split stages S1-S4 respectively. Among them, S0 represents the stage of no macular retinal split; S1 represents the stage of local split outside the fovea; S2 represents the stage of split involving the fovea and extending outside the fovea; S3 represents the stage of full-layer split involving the entire macula; S4 represents the stage of macular detachment progressing from full macular split.

[0090] The Softmax calculation formula is as follows:

[0091] Among them, represents the score corresponding to the category, and Softmax normalizes the original scores of all categories into a probability distribution.

[0092] The embodiment also constructs a stage definition memory embedding module, introduces a stage definition embedding memory matrix, and provides semantic prior constraints in contrastive learning. The memory matrix is constructed from standard stage definitions by a text encoder, and the specific process is as follows: Referring to Figure 2 , the descriptive medical definition text of the five stages is encoded by using the text encoder BERT.

[0093] The stage definition description text is as follows: S0 stage definition: the stage of no macular retinal split; it is prompted that the highly myopic eye has not yet formed a split, and it is used as a baseline stage to distinguish between eyes with complete structure and eyes with early split changes.

[0094] S1 stage definition: the stage of local split outside the fovea; the split is limited to the area outside the fovea, and the fovea itself is not involved, which is the earliest detectable stage in high myopia.

[0095] S2 stage definition: the stage of split involving the fovea and extending outside the fovea; both the fovea and the peripheral area are involved, but the entire macula has not yet been involved.

[0096] S3 stage definition: the stage of full-layer split involving the entire macula; the split spans the entire macular area and involves multiple layers of retina.

[0097] S4 stage definition: the stage of macular detachment progressing from full macular split; it also includes central foveal split with layered or full-layer macular holes, even if full macular split has not occurred.

[0098] This definition is defined by experts according to clinical experience, and after these definitions are composed into a text list, input into the word segmentation tool for word segmentation, completion and truncation operations, and formatted into the input tensor required by the model. Let the input be:

[0099] After passing through the text encoder, the feature output of the definition is obtained , where is the maximum text length.

[0100] In order to obtain the unified embedding representation of each stage definition, take the global feature vector corresponding to each piece of text as the output: 5 5) That is, the final representation of each definition is the feature in its first position.

[0101] The above five vectors are spliced to form a stage definition memory matrix:

[0102] The stage memory matrix does not participate in gradient update during training, and the semantic similarity between the image and each stage definition is calculated as a contrast loss, which improves the model's ability to distinguish the meaning of the medical stage.

[0103] In order to enhance the robustness of the model under the condition of class imbalance, and improve the discrimination ability between images and semantic memory, the embodiment constructs a combined loss function, which combines the FocalLoss classification loss and the contrast loss guided by the staging memory to optimize the discrimination boundary and modal alignment effect of the model.

[0104] Among them, the classification main loss part adopts FocalLoss, which is used to alleviate the sample imbalance problem and focus the model's learning ability on difficult samples. The FocalLoss classification loss is expressed by the formula:

[0105] Among them, represents the prediction confidence of the th sample, is the class imbalance adjustment factor ( =0.25), is the focal modulation factor (γ=2), which is used to enhance the loss weight of difficult classification samples.

[0106] represents the standard cross-entropy loss, which is expressed by the formula:

[0107] Meanwhile, to enhance the separation and convergence between image features and semantics at each stage, a multimodal contrastive loss based on memory embedding vectors is introduced. The process is as follows: first, image features... Through projection network Map and normalize to Then, with each type of fixed stage memory vector (Also normalized) Perform inner product to form a similarity matrix:

[0108] The contrastive loss is defined using the cross-entropy form as follows:

[0109] Meanwhile, to strengthen the gap between the positive class and the most difficult negative class, a boundary loss is introduced. The final total loss is:

[0110] in and These are learnable weight coefficients, initially set to 0.3 and 0.1 respectively, used to adjust the contribution of contrastive constraints to the overall optimization. This mechanism significantly enhances the model's ability to distinguish semantic proximity during the segmentation phase, especially under multimodal input, effectively improving the clarity of the model's discrimination boundaries and its ability to recognize nested class structures.

[0111] Finally, this embodiment was validated using an experimental dataset from an ophthalmology hospital. This dataset included patients diagnosed with high myopia (axial length > 26 mm), for whom corresponding OCT images and metadata (age, axial length, best corrected visual acuity, and gender) were available for analysis. The dataset consisted of 2251 retinal OCT images from 688 eyes of 497 patients. The dataset categories included normal and four stages of retinal schisis, with 557 normal images, 401 images detected in stage 1 schisis, 316 images in stage 2, 448 images in stage 3, and 529 images in stage 4. The training and test sets were divided by eyes in a 4:1 ratio, with the training set containing 1777 images and the test set containing 474 images. The dataset distribution information is shown in Table 1. Table 1 Dataset Distribution Information

[0112] The four indicators of accuracy, specificity, sensitivity and F1-score are used for evaluation.

[0113] Accuracy represents the proportion of correctly predicted samples in the total number of samples, and is a basic indicator for measuring the overall classification performance. The calculation formula is as follows:

[0114] Among them, TP is the number of true cases, TN is the number of true negative cases, FP is the number of false positive cases, and FN is the number of false negative cases.

[0115] Specificity represents the ability of the model to correctly identify negative classes, i.e., the proportion of true negative cases in the predicted negative classes. This indicator reflects the ability of the model to misreport, and the higher the specificity, the fewer the misreports.

[0116]

[0117] Sensitivity (also known as recall) represents the ability of the model to identify positive classes, i.e., the proportion of true positive cases in the predicted positive classes. This indicator reflects the ability of the model to miss reports, and the higher the sensitivity, the fewer the missed reports.

[0118]

[0119] F1 score is the harmonic mean of precision and recall, suitable for class imbalance situations, and can comprehensively reflect the overall performance of the model:

[0120]

[0121] The model MCTS of the present embodiment is compared with several mainstream networks (ResNet-50, EfficientNet-b0, ViT, VGG, Swin, DINO) with excellent classification performance at present. As shown in Tables 2 and 3, Tables 2 and 3 are the average performance of all models running ten times on the dataset.

[0122] Table 2 Comparison of image-level classification performance experiments

[0123] Table 3 Comparison of eye-level classification performance experiments

[0124] Table 2 is a comparison of the classification performance of all models on a single image as a sample, and Table 3 is a comparison of the classification performance at the eye level. Since one eye in the dataset contains multiple retinal OCT images, the most severe true label in all images is taken as the true label of each eye, and each index is calculated in this way.

[0125] In the image-level experiment, the performance of various mainstream visual classification models and the proposed MCTS multi-modal model was compared. The results show that MCTS achieves the best performance in accuracy, specificity, sensitivity, and F1-score, with an accuracy of 97.43% and an F1-score of 96.90%. Compared with the optimal single-modal baseline DINO, the accuracy and F1-score are improved by about 2.89% and 3.29%, respectively. Compared with ResNet-50, the accuracy is improved by 35.49%, and the F1-score is improved by 37.68%. These results fully verify the importance of introducing structured information to supplement image features and effectively enhance the model's ability to distinguish complex clinical features.

[0126] In the eye-level experiment, the overall performance of each model decreased, but MCTS still led in the four indicators. The accuracy, specificity, sensitivity, and F1-score of the MCTS model were 93.69%, 98.42%, 92.63%, and 92.79%, respectively, significantly better than all comparison models. This result shows that at a more granular evaluation level, the multi-modal fusion strategy brings more significant performance gains, further demonstrating the stronger robustness and practicality of the MCTS model in real clinical scenarios.

[0127] In the patient-level experiment, the overall performance of MCTS improved compared to the eye-level evaluation. Among traditional single-modal image models, DINO and Swin performed best, with an accuracy of 92.39% and an F1-score of 91.54%, and an accuracy of 92.47% and an F1-score of 91.45%, showing strong feature extraction and discrimination ability. However, compared with these models, the MCTS model proposed in this embodiment significantly improves the four indicators, with an accuracy of 98.03%, a specificity of 99.03%, a sensitivity of 98.17%, and an F1-score of 98.07%, performing the best among all models.

[0128] Specifically, compared with the best single-modal model DINO, the accuracy, specificity, sensitivity and F1-score of MCTS are improved by 5.64%, 2.83%, 5.26% and 5.60%, respectively; compared with the Swin model, the accuracy and F1-score are improved by 6.49% and 6.62%, respectively. These results fully demonstrate that the introduction of the multi-modal fusion strategy (image + text) can effectively enhance the comprehensive understanding ability of the model to the lesion pattern, make MCTS show higher robustness and clinical usability in the patient-level triage task, and verify its strong application potential in real diagnosis and treatment scenarios. The experimental comparison results are shown in Table 4: Table 4 Performance comparison of patient-level triage

[0129] To evaluate the robustness of the present application under different OCT scanning devices and imaging parameters, an independent internal dataset (61 patients, 100 eyes, 323 images) was constructed on the Gupai device in this embodiment. This dataset follows the same inclusion criteria as the above data. All OCT scans were collected according to the standard protocol of the device manufacturer, covering different imaging ranges such as standard mode (10.00 mm x 3.00 mm) and wide-field mode (12.00 mm x 3.00 mm, 12.00 mm x 4.50 mm, 12.00 mm x 6.00 mm). Thus, the dataset covers a variety of scanning depths and field widths, and is more close to the real diversity in clinical applications. Some wide-field images contain peripheral text overlay information. In order to maintain the authenticity and complexity of the data, we retained these features in the experiment. In the model migration experiment, we performed 60 rounds of lightweight fine-tuning on the Gupai data. The dataset is divided into training set and validation set according to the ratio of 8:2, ensuring that the same eye does not appear in different divisions at the same time, and maintaining the same staging distribution ratio when stratified sampling. All cross-device performance evaluation results are based on the Gupai validation set after fine-tuning.

[0130] In the cross-device experiment, the performance of different models on the Ophidi dataset is shown in Table 5. It can be observed that all traditional single-modal image models show different degrees of performance degradation after migrating to new devices, especially in image-level and eye-level evaluation, the F1-score of ResNet-50 and EfficientNet-b0 is only 20.28% and 18.39% respectively, indicating that they are sensitive to device domain shift. With the increase of model structure complexity, the performance of Transformer series models (such as ViT, Swin and DINO) gradually improves, among which DINO achieves 91.00% and 82.22% in accuracy and F1-score at the patient level, respectively, which is better than all single-modal control models. In contrast, the MCTS model proposed in this study is significantly better than other models in three evaluation levels, with accuracy of 96.73%, 92.50% and 98.00% at image level, eye level and patient level, and F1-score of 88.06%, 83.33% and 96.44% respectively. Compared with the best single-modal model DINO, MCTS improves the accuracy of the patient level by 7.00% and the F1-score by 14.22%. These results fully prove that the multi-modal information fusion strategy can effectively alleviate the influence of cross-device domain difference on model performance, so that MCTS can still maintain stable diagnostic ability under different OCT platforms, and show excellent cross-domain generalization performance and clinical robustness.

[0131] Table 5 Comparison of cross-device classification performance experiments

[0132] To further analyze the model's discriminant basis and its attention ability to key lesion areas, this embodiment uses the Grad-CAM (Gradient-weighted Class Activation Mapping) method to visualize the attention area of the image classification model. Grad-CAM calculates the gradient of the target class to the feature map of a specific convolutional layer, generating a weight-weighted heat map, which reflects the spatial position that the model focuses on when making a classification decision. This embodiment selects the last layer of the main feature map as the activation region, calculates the attention map combined with the gradient information, and superimposes it on the original image to intuitively show the model's attention area. MCTS can more accurately focus on the lesion area and eliminate redundant background information, indicating that it has stronger spatial perception ability and discriminability after fusing structured information. This visualization result helps to improve the explainability of the model and provides intuitive support for its application in clinical auxiliary diagnosis.

[0133] Enhance the generalization ability and clinical applicability of the model: by fusing structured information, the application improves the adaptability of the model to different populations and individual differences, effectively reduces the overfitting risk of the model to image pseudo-features, and improves its stability and generalization ability in real clinical environment.

[0134] Support real-time intelligent diagnosis process closed loop: the application can meet the real-time image stream processing requirements of OCT, combined with the local deployment mechanism, can be directly embedded into the OCT image acquisition terminal, realize the full-process automatic staging diagnosis system of "scanning-identification-staging-prompt", and improve the clinical diagnosis and treatment efficiency.

[0135] In summary, the application not only improves the accuracy and intelligent level of retinal splitting staging diagnosis, but also takes into account the deployment convenience and running efficiency of the system, and has a wide clinical application prospect in the field of intelligent auxiliary diagnosis of high myopia fundus lesions.

[0136] Embodiment 2 The embodiment of the application discloses a high myopia retinal image classification system fusing multi-modal information, referring to Figure 3 As shown in the figure, it includes the following units: Acquisition unit: for acquiring the OCT image of the high myopia retina to be classified and the structured numerical data corresponding to the image; the structured numerical data includes demographic information and eye measurement indicators; Image processing unit: for pre-processing the OCT image, and inputting the pre-processed image into an image encoder for high-dimensional semantic representation learning to obtain the overall semantic vector representation of the image; Text processing unit: for converting the structured numerical data into medical language description text and inputting the text into a text encoder for deep semantic modeling to obtain the overall semantic vector representation of the text; Fusion unit: for inputting the overall semantic vector representation of the image and the text into a multi-modal fusion module, and performing feature interaction and fusion through a bidirectional cross-attention mechanism to generate fused multi-modal features; Classification unit: for classifying the multi-modal features through a classification module, and outputting the retinal splitting staging category corresponding to the OCT image.

[0137] The embodiment is applied to an auxiliary diagnosis system of a medical device, and realizes a full-process closed loop from "image acquisition" to "intelligent diagnosis suggestion". After the OCT image, patient data and medical knowledge are acquired by the acquisition unit, they are processed by the image processing unit and the text processing unit, and the feature interaction and fusion are performed through the fusion unit combined with the bidirectional cross-attention mechanism; finally, the classification unit gives the staging result.

[0138] The various embodiments described in this specification are implemented in a progressive manner, each embodiment focusing on the differences from other embodiments, and the same or similar parts between embodiments can be mutually referred to. For the apparatus disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0139] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those of ordinary skill in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of classifying high myopia retinal images fusing multi-modal information, characterized in that, The method comprises the following steps: obtaining an OCT image of a high myopia retina to be classified and structured numerical data corresponding to the image; the structured numerical data comprises demographic information and eye measurement indicators; preprocessing the OCT image and inputting the preprocessed image into an image encoder for high-dimensional semantic representation learning to obtain an overall semantic vector representation of the image; the structured numerical data is converted into a medical language description text and input into a text encoder for deep semantic modeling to obtain an overall semantic vector representation of the text; input the overall semantic vector representations of the image and the text into a multi-modal fusion module, interact and fuse the features through a bidirectional cross-attention mechanism to generate a fused multi-modal feature; classify the multi-modal feature through a classification module to output a retinoschisis staging category corresponding to the OCT image. 2.The highly myopic retina image classification method of fusing multi-modal information according to claim 1, wherein, It also includes: constructing a staging definition memory matrix to provide semantic prior constraints for calculating a staging memory-guided contrast loss; based on the staging definition memory matrix, a contrast learning mechanism is used to calculate the similarity scores between the overall semantic vector of the OCT image and the semantic vectors of each stage, and the similarity scores of the same stage samples are maximized and the similarity scores of different stage samples are minimized as constraints to align the image semantic vector with the stage semantic vector; construct a combined loss function to fuse the FocalLoss classification loss and the staging memory-guided contrast loss, and optimize the model parameters in the process of training the entire multi-modal retinoschisis staging model. 3.The highly myopic retina image classification method of fusing multi-modal information according to claim 1, wherein, The preprocessing of the OCT image specifically includes: size normalization, image enhancement, pixel value normalization and format conversion of the OCT image; wherein the image enhancement includes random horizontal flipping, random rotation within a preset angle range, brightness and contrast adjustment, Gaussian blur and region occlusion operation. 4.The highly myopic retina image classification method of fusing multi-modal information according to claim 1, wherein, The process of image processing by the image encoder specifically includes: image block segmentation of the input image, embedding each image block into a feature space through linear mapping, adding position encoding information to each image block, and inserting a classification mark at the beginning of the sequence to summarize the overall semantic information of the image; input the image block vector sequence containing the classification mark into a multi-layer BEiT encoder for deep semantic feature extraction; wherein each layer of the BEiT encoder includes a multi-head self-attention mechanism, a residual connection, a layer normalization and a feedforward neural network; extract the vector representation containing the classification mark from the output of the last layer of the BEiT encoder as the overall semantic vector representation of the image. 5.The highly myopic retina image classification method of fusing multi-modal information according to claim 1, wherein, The process of converting the structured numerical data into a medical language description text specifically includes: field standardization and cleaning of the structured numerical data; natural language concatenation, completion, embedding and Tokenization through a natural language template to output a description text with medical terminology. 6.The highly myopic retina image classification method of fusing multi-modal information according to claim 1, wherein, The process of text processing by the text encoder specifically includes: segmenting the input text into word or subword sequences and mapping each word or subword to a unique identifier; Splicing each token with position encoding information to form an input sequence, and adding a CLS token at the beginning of the sequence to input a BERT encoder; The BERT encoder encodes the input sequence to obtain a hidden feature vector of each token, and takes the feature vector corresponding to the CLS token as the overall semantic vector representation of the text.

7. The highly myopic retina image classification method fusing multi-modal information according to claim 1, wherein, The multi-modal fusion module is composed of multiple layers of bidirectional cross-attention layers; each bidirectional cross-attention layer is composed of two single-direction cross-attention blocks with opposite order and independent parameters; The single-direction cross-attention block takes the image or text as Query and the other modality feature as Key / Value; the cross-modal response is calculated to obtain the cross-modal enhanced feature, and the original feature is adaptively fused with the gated residual to obtain the enhanced image or text semantic feature; The enhanced image or text semantic feature is respectively subjected to average pooling operation and splicing fusion to obtain the multi-modal feature of the global representation.

8. The highly myopic retina image classification method fusing multi-modal information according to claim 1, wherein, The classification module classifies the multi-modal feature to output the retinal splitting stage category corresponding to the OCT image; specifically including: The multi-modal feature is classified by a linear layer and a Softmax activation function to output S0-S4 results corresponding to normal retina and retinal splitting stage respectively; Wherein, S0 represents the stage of no macular retinal splitting; S1 represents the stage of focal splitting outside the fovea; S2 represents the stage of splitting involving the fovea and extending outside the fovea; S3 represents the stage of full-layer splitting involving the entire macula; S4 represents the stage of macular detachment from full macular splitting.

9. The highly myopic retina image classification method fusing multi-modal information according to claim 2, wherein, The construction process of the stage definition memory matrix includes: Inputting five types of retinal splitting stage definition texts into the text encoder respectively to extract semantic vectors corresponding to the five stages of definition; and splicing the five semantic vectors to form a stage definition memory matrix; The five types of retinal splitting stage definition texts include: The stage of no macular retinal splitting, which indicates that the highly myopic eye has not yet formed splitting and is used as a baseline stage to distinguish between eyes with complete structure and eyes with early splitting changes; The stage of focal splitting outside the fovea, which is limited to the area outside the fovea and the fovea itself is not involved, which is the earliest detectable stage in high myopia; The stage of splitting involving the fovea and extending outside the fovea, which involves both the fovea and the peripheral area, but has not yet involved the entire macula; The stage of full-layer splitting involving the entire macula, which involves multiple layers of retina; The stage of macular detachment from full macular splitting, which also includes central foveal splitting with layered or full-layer macular holes, even if there is no full macular splitting.

10. A highly myopic retina image classification system fusing multi-modal information, characterized in that, The following units are included: An acquisition unit is configured to acquire an OCT image of a highly myopic retina to be classified and structured numerical data corresponding to the image; The structured numerical data includes demographic information and eye measurement indicators; An image processing unit is configured to pre-process the OCT image and input the pre-processed image into an image encoder for high-dimensional semantic representation learning to obtain an overall semantic vector representation of the image; The text processing unit is configured to convert the structured numerical data into medical language description text, and input the text into a text encoder for deep semantic modeling to obtain an overall semantic vector representation of the text. The fusion unit is configured to input the overall semantic vector representations of the image and the text into a multi-modal fusion module, interact and fuse features through a bidirectional cross-attention mechanism, and generate fused multi-modal features. The classification unit is configured to classify the multi-modal features through a classification module, and output a retinoschisis staging category corresponding to the OCT image.

Images