A clinical text-guided anti-vegf response level classification method

Abstract

The application discloses a clinical text guided anti-VEGF response level classification method, which comprises the following steps: pre-processing medical images and clinical texts, and constructing a multi-modal input data set; extracting features from OCT images, dividing regional features based on spatial positions, taking clinical text tokens as queries, weighting and aggregating regional features through a CTRA module, and extracting guided regional visual features; constructing an IR to OCT cross-regional collaborative enhancement module, adaptively injecting IR information into OCT structural regional enhancement representation, and feeding back to the CTRA module for secondary convergence; fusing multi-source tokens, performing level classification through a hierarchical boundary classification module, and optimizing parameters through joint supervision. The application realizes deep joint use of OCT images and clinical text information, enhances the discrimination ability of the model to local pathological regions and category boundaries, and improves the accuracy and stability of anti-VEGF response level classification.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a clinical text-guided hierarchical classification method for anti-VEGF response. Background Technology

[0002] The macula is a crucial structure for maintaining central vision, and its lesions often lead to irreversible visual impairment. Age-related macular degeneration, diabetic macular edema, and retinal vein occlusion can all affect the macula, and the burden of these diseases continues to increase with population aging. Anti-vascular endothelial growth factor therapy, by inhibiting abnormal angiogenesis and leakage, has become an important intervention for these diseases. However, treatment responses vary significantly among patients; therefore, accurate classification of treatment response levels before intervention is helpful in developing individualized treatment plans.

[0003] Optical coherence tomography (OCT) can non-invasively display structural information such as macular central thickness and intraretinal / subretinal fluid, making it a core tool for the diagnosis and treatment of fundus diseases. However, visual recovery is affected by multiple factors, including baseline visual acuity, age, and disease duration, making it difficult to accurately predict subsequent treatment responses based solely on a single OCT feature. Intelligent classification methods based on OCT images have been extensively explored. Early studies used convolutional networks to automatically extract pathological features from OCT images to predict postoperative visual acuity, visual improvement after anti-VEGF treatment, and recurrence risk. However, simple convolutional networks are insufficient for modeling cross-regional structural relationships and individualized clinical differences. Subsequently, researchers introduced multimodal fusion and attention mechanisms, such as fusing images with clinical text, using Transformers for multimodal fusion, and introducing group convolutions and multi-scale convolutional kernels to enhance lesion feature extraction. However, these methods still primarily focus on global feature fusion, with clinical information merely used for simple splicing, failing to actively guide the model to focus on key regions.

[0004] Recent studies have utilized pre- and post-treatment image pairs and multi-task learning, employing Siamese networks for efficacy classification, multi-task prediction of best-corrected visual acuity and OCT structure, and joint multimodal mutual learning. While these methods have improved predictive capabilities, they still lack sufficient focus on actively guiding local region modeling based on clinical text and determining efficacy level boundaries.

[0005] In summary, the existing technologies have the following shortcomings: global feature extraction is difficult to accurately locate local pathological areas and is easily affected by background interference; clinical information is simply spliced ​​with image features and fails to actively guide the selection of visual features; the directional auxiliary role of IR regions on OCT structural regions is not effectively modeled; and ordinary three-class classification is easily dominated by the majority class under class imbalance, compressing the discrimination space of stable and poor classes. Summary of the Invention

[0006] The purpose of this invention is to provide a clinical text-guided hierarchical classification method for anti-VEGF response, which solves the problems in existing technologies such as the susceptibility of global feature extraction to background interference, the simple concatenation of clinical information with image features, insufficient utilization of the complementary relationship between IR regions and OCT structural regions, and the compression of boundary class discrimination space in ordinary three-class classification under class imbalance. This invention achieves deep joint utilization of OCT images and clinical text information, enhances the model's ability to discriminate local pathological regions and class boundaries, and improves the accuracy and stability of anti-VEGF response hierarchical classification.

[0007] The technical solution adopted in this invention is a clinical text-guided hierarchical classification method for anti-VEGF responses, comprising the following steps: Step S1: Perform unified preprocessing on historical medical images and structured clinical texts to construct a multimodal input dataset for anti-VEGF response prediction; Step S2: Extract features from the preprocessed OCT image, flatten the feature map and divide it into different regions based on spatial location; use the clinical text token as the query vector, and use the CTRA module to perform weighted aggregation of the above regional features to extract the regional visual features guided by the clinical text. Step S3: Construct an IR to OCT cross-regional collaborative enhancement module, using the OCT structural region as the query and the IR region as the context to perform adaptive attention aggregation, realize the residual enhancement of IR information to the OCT structure, and feed the enhanced features back to the CTRA module for secondary aggregation; Step S4: Fuse and process the features of the multi-source tokens, and perform hierarchical classification of the processed features through the hierarchical boundary classification module. Calculate the overall loss using a joint supervision strategy to optimize the model parameters.

[0008] Further, step S1 specifically includes: Baseline OCT images and structured clinical text information corresponding to historical medical samples are obtained; the historical medical samples are those that have previously received anti-VEGF intervention; the OCT images include horizontal and vertical views; Based on the actual observation records of the samples during the follow-up process and the preset clinical interpretation criteria, response level labels are labeled for each sample to construct a labeled training sample set; the response level labels include three categories: good response, stable response, and poor response; The input images undergo uniform preprocessing, including resizing to a preset size and normalizing. Structured clinical text information is then processed, including encoding categorical variables and standardizing numerical variables to form clinical text feature vectors. ; via text encoder Convert to text query token: in, This represents the output text query token; Presentation layer normalization operation; Indicates the activation function; , These are learnable parameters in the text encoder; The preprocessed multimodal data is input into the TCH-Net network for subsequent feature extraction and classification.

[0009] Furthermore, in step S2, the feature extraction and spatial location division specifically include: The horizontal and vertical views of the OCT image are input into two independent ConvNeXt-Tiny branches for feature extraction. Both branches are initialized with ImageNet pre-trained weights, and their feature extraction parts are used as visual encoders. The feature maps are flattened and converted into spatial token sequences. Combining the spatial location information of the OCT image, the spatial token sequence of each view is divided into two types of region tokens: IR region tokens and OCT structure region tokens.

[0010] Further, in step S2, the extraction of region-level visual features guided by clinical text specifically includes: Clinical text token sequence For query vector, with Indicates the view type; where, This represents a horizontal view. Represents a vertical view; for any view Let its IR region token and OCT structure region token be respectively denoted as , Let the OCT structure region token and IR region token of the horizontal view be respectively... 、 The OCT structure region token and IR region token of the vertical view are respectively 、 ; Introducing symbols As a general placeholder, it is used to refer to any region of the token sequence in the current input module, i.e. ; Through linear transformation , These are respectively mapped to the query vectors required for attention calculation. Key vector Sum value vector Employing a multi-head attention mechanism, , , The feature dimension is uniformly divided into multiple sub-vectors. The weights of each attention head to the region token are calculated by scaling dot product attention, and the sub-vectors of the corresponding value vector are weighted and summed to obtain the region aggregation features of each attention head. The region aggregation features of all attention heads are concatenated along the feature dimension, and the final region convergence features are obtained by linear mapping, layer normalization and Dropout regularization operations. The weights of all attention heads are averaged to generate the region attention map.

[0011] Furthermore, in step S3, the specific implementation process of the IR-to-OCT cross-regional collaborative enhancement module is as follows: For views token with OCT structure area As a query vector, with IR region token As the key vector and value vector, the query vector is obtained through linear transformation. Key vector Sum value vector The scaling dot product attention mechanism is used to calculate the attention weights of the OCT structural region to the IR context, and the IR context supplementary features output by a single attention head are obtained by weighted summation. A multi-head cross-region attention structure is adopted, in which each attention head performs the above calculation in parallel. The context supplementary features output by each attention head are concatenated along the channel dimension, and after processing by the output projection layer, layer normalization and Dropout, the final OCT context supplementary features are obtained. ; Set learnable scale parameters Adaptive injection coefficients are generated using the Sigmoid function. The IR context features are injected as residuals into the OCT structural region token to obtain the enhanced OCT structural region token. ;in, This represents the Sigmoid activation function; Represents a view The original OCT structure region token sequence; This represents the token sequence of the enhanced OCT structure region.

[0012] Furthermore, step S3 also includes: The enhanced OCT structure area token With the original IR region token The inputs are combined and then aggregated in the CTRA module to obtain a regional token guided by clinical text semantics. in, This represents the IR region token of the horizontal view after processing by the CTRA module; Represents a clinical text token; This represents the original IR region token in the horizontal view; This represents the IR region token of the vertical view after processing by the CTRA module; This represents the original IR region token in the vertical view; This indicates the enhanced horizontal view OCT region token after processing by the CTRA module; This represents the OCT region token in the horizontal view after context enhancement; This represents the enhanced vertical view OCT region token after processing by the CTRA module; This represents the OCT region token sequence in the vertical view after context enhancement.

[0013] Furthermore, in step S4, the fusion and feature processing of the multi-source tokens specifically includes: The clinical text token, the horizontal view IR region token, the vertical view IR region token, the enhanced vertical view OCT region token, and the enhanced horizontal view OCT region token are sequentially concatenated into a multi-source fusion sequence. This fusion sequence is then input into the Transformer Encoder Layer, where a self-attention mechanism is used to model the global dependencies between tokens, outputting the fused token sequence. ;right After performing the flattening operation, the shared feature mapping function is used. A global representation is obtained.

[0014] Furthermore, in step S4, the specific implementation process of the hierarchical boundary classification module includes: The three-category task is broken down into two hierarchical sub-tasks: The first layer is a coarse boundary discrimination, distinguishing whether a sample belongs to a good response or a bad response; global representation. via branch function After transformation, the binary classification head is input, and two class probabilities are generated using Softmax: in, , These represent the probabilities of a sample being good or non-good, respectively. Representing global representation After the first level of branch function The intermediate feature representation obtained after transformation; This represents the weight matrix in the first-layer binary classification head; This represents the bias vector in the first-level binary classification head; The second layer is fine boundary discrimination, which further distinguishes between stable and poor responses within the non-good response subspace; global characterization. via branch function After transformation, the input binary classifier is used to generate conditional probabilities using Softmax: in, , These represent the conditional probabilities of being considered stable and poor under non-good conditions, respectively. Representing global representation After the second branch function The intermediate feature representation obtained after transformation; This represents the weight matrix in the second-layer binary classification head; This represents the bias vector in the second-layer binary classification head; Combining the results of the two-layer discrimination, the final three-class classification probability is defined as: in, The predicted probability that the sample ultimately belongs to the good category; This represents the predicted probability that a sample will eventually belong to a stable category. This represents the predicted probability that the sample will ultimately belong to the poorer category.

[0015] Furthermore, the total loss is composed of three weighted components: in, This represents the overall joint loss function of the hierarchical boundary classification module. Used to constrain the final three-class classification result Used to monitor the distinction between good and bad responses in the first phase. Used to monitor response stability and response difference differentiation in the second stage. , , These are the weighting coefficients used to balance the various losses.

[0016] The beneficial effects of this invention are: 1. This invention proposes a clinical text-guided regional attention mechanism, which enables the model to actively focus on local pathological regions related to the response level by combining individualized clinical information of patients. This overcomes the limitation of global feature extraction in the prior art being easily interfered with by background regions, and significantly improves the ability to identify key structural features and the accuracy of response level classification.

[0017] 2. This invention transforms structured clinical text into guiding vectors and participates in region attention calculation, thereby enabling clinical text to actively guide the OCT image feature extraction process. This addresses the shortcomings of existing technologies where clinical information is simply concatenated with image features and fails to truly participate in visual feature selection, thus enhancing the ability to deeply and jointly utilize multimodal information.

[0018] 3. This invention designs a cross-regional collaborative enhancement mechanism from IR to OCT, which directionally supplements the scan localization, fundus background and lesion context information in the IR region into the OCT structural region, making up for the deficiency of existing technology in not fully utilizing the complementary relationship between different functional regions and enhancing the expressive ability of response-related structural features.

[0019] 4. This invention constructs a hierarchical boundary classification mechanism, first distinguishing between good and non-good responses, and then further distinguishing between stable and poor responses. This effectively alleviates the problem of majority class dominance and compressed boundary class discrimination space in ordinary three-class classification models under class imbalance conditions, and improves the prediction stability of minority and boundary classes. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a diagram of the TCH-Net framework of the present invention.

[0022] Figure 2 This is a schematic diagram of the CTRA module of the present invention.

[0023] Figure 3This is a schematic diagram of the IOC module of the present invention.

[0024] Figure 4 This is a schematic diagram of the HBC module of the present invention.

[0025] Figure 5 The diagram shows the confusion matrices for other comparison methods, where (a), (b), (c), and (d) are the confusion matrices for HDF-Net, ConvNeXt-Tiny, RwithT-Net, and MDF-Net, respectively.

[0026] Figure 6 This is a confusion matrix diagram of the method of the present invention and other comparative methods, where (a), (b), (c), (d), and (e) are the confusion matrices of Siamese, AGC-SPP, KongMing, M2Net, and the TCH-Net of the present invention, respectively.

[0027] Figure 7 The images are visualizations comparing the results before and after using the CTRA module. (a), (d), (g), and (j) are the original input images, respectively; (b), (e), (h), and (k) are the heatmaps generated by the baseline model, respectively; and (c), (f), (i), and (l) are the heatmaps after using the CTRA module. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] like Figure 1 As shown in the example, a clinical text-guided hierarchical classification method for anti-VEGF responses includes the following steps: Step S1: Construct a multimodal input dataset for VEGF-resistant response prediction: This step aims to provide standardized input data for subsequent multimodal feature extraction. By using computers to perform unified preprocessing on historical medical images and structured clinical texts, the feature representation capability of multimodal data can be improved.

[0030] First, baseline OCT images and structured clinical text information corresponding to historical medical samples are obtained. Historical medical samples are a collection of samples that have previously received anti-VEGF intervention. The OCT images include horizontal and vertical OCT images. The structured clinical text information includes relevant variables such as the individual's gender, age, baseline visual acuity, diagnosis type, central macular thickness, and past medical history. Based on the actual observation records of the samples during the follow-up process, response level labels are assigned to each sample. The response level labels include three categories: good response, stable response, and poor response. This is used to construct a labeled training sample set.

[0031] During the annotation process, the clinical interpretation standards of physicians with experience in the diagnosis and treatment of fundus diseases were referenced, and their empirical logic was transformed into objective quantitative rules that can be executed by machines. The interpretation standards are mainly based on the following trends in clinical manifestations: changes in visual function, changes in macular central thickness, changes in intraretinal or subretinal fluid, degree of macular edema, and lesion activity. If all of the above indicators show significant improvement after treatment, the corresponding sample is labeled as having a good response. If there is no significant improvement in any of the indicators or if they worsen, the sample is labeled as having a stable response. If the lesions persist, worsen, or recur after treatment, and visual function does not improve well, the sample is labeled as having a poor response.

[0032] Then, the input images undergo uniform preprocessing, adjusting the horizontal and vertical OCT images to 224×224 pixels and normalizing them according to common settings for ImageNet pre-trained models; the mean values ​​of the RGB three channels are 0.485, 0.456, and 0.406, respectively, and the standard deviations are 0.229, 0.224, and 0.225, respectively. The structured clinical text information is then processed, including encoding categorical variables and standardizing numerical variables, ultimately forming a clinical text feature vector that can be input into the neural network.

[0033] Furthermore, in order to enable clinical text feature vectors to interact with image features in a unified semantic space, it is necessary to map the text feature vectors into embedding vectors that can participate in cross-modal computation; specifically, this is achieved by using a text encoder to map the clinical text feature vectors... Convert to text query token: in, This represents the output text query token; Indicates batch size. Indicates feature dimension; This represents the layer normalization operation. , These are the learnable parameters in the text encoder; the text encoder consists of a linear mapping layer, a GELU activation function, and Layer Normalization.

[0034] At this point, the standardization and alignment of the multimodal input data have been completed; that is, the flattened visual feature sequence and the corresponding text query token have been obtained. Subsequently, the preprocessed multimodal data (visual features and text token) is input into the TCH-Net network (clinical text-guided hierarchical classification network) constructed in this invention. This network mainly includes a clinical text-guided region attention (CTRA) module, an IR→OCT cross-regional collaborative enhancement (IOC) module, and a hierarchical boundary classification (HBC) module, which are used to realize multimodal feature fusion and response level classification.

[0035] Step S2: Extracting region-level visual features using clinical text-guided region attention: This step aims to address the problem that existing methods rely solely on global features of OCT images and struggle to actively focus on key pathological regions relevant to treatment efficacy. Specifically, the model achieves accurate extraction of region-level visual features through the following steps: The horizontal and vertical OCT images were input into two independent ConvNeXt-Tiny branches for feature extraction. Both branches were initialized with ImageNet pre-trained weights, but instead of using the classification head from the original network, their feature extraction parts were extracted and used as the visual encoder. Subsequently, the feature maps were flattened and converted into spatial token sequences. Combining the spatial location information of the OCT images, the spatial token sequences of each view were divided into two types of region tokens: IR region tokens, which mainly contain scan localization, fundus background, and lesion context information; and OCT structural region tokens, which mainly reflect the retinal tomographic structure and abnormal representations related to treatment efficacy (which are key regions used to distinguish response levels during model training).

[0036] To integrate clinical text features and regional visual features, this invention constructs a Clinical Text-Guided Regional Attention (CTRA) module, the structure of which is as follows: Figure 2 As shown; the query vector for this module is a sequence of clinical text tokens. ;in, The text features output in step 1; simultaneously, the present invention uses Indicates the view type; where, This represents a horizontal view. Represents a vertical view; for any view Let its IR region token and OCT structure region token be denoted as: , ;in, For batch size, For feature dimensions; , Representing the number of tokens in the OCT region and IR region respectively, the horizontal view corresponds to... 、 Vertical view corresponds 、 In the formula derivation of the CTRA module, symbols are introduced to simplify the repetitive writing for different views and regions. As a general placeholder, it is used to refer to any region of the token sequence in the current input CTRA module; specifically... It does not correspond to any newly generated features; its values ​​are only those mentioned above. , one of the.

[0037] Through linear transformation , These are respectively mapped to the query vectors required for attention calculation. Key vector Sum value vector The calculation formula is: in, , , These are learnable parameters; to enhance the diversity of feature representation, the following is adopted: Perform computation using parallel attention heads; , , Uniformly segmented along the feature dimension The nth subvector, the nth The subvectors of each attention head are represented as follows: ;in, ; The length of the sequence of tokens in the input region.

[0038] No. The region attention weights for each attention head are calculated by scaling the dot product attention: The corresponding region aggregation features are: in, This indicates the distribution of attention given to each visual feature within the text query area. Represents the normalization function; express transpose; The feature dimension of a single attention head; This represents the region aggregation feature output by the attention head.

[0039] All attention head output Concatenate along the feature dimension, and then perform linear mapping, layer normalization (LN), and Dropout regularization to obtain the final region convergence features. : in, , For learnable parameters, The region convergence feature output by the CTRA module represents specific region-level visual features after convergence via the text-guided attention mechanism. In this embodiment of the invention, the region convergence token sequence Z corresponds to different specific representations depending on the input source: when the input source is a horizontal view IR region, Z is denoted as a horizontal view IR region-level token; when the input source is a vertical view IR region, Z is denoted as a vertical view IR region-level token; similarly, when the input source is a horizontal view OCT structure region, Z is denoted as a horizontal view OCT structure region-level token; when the input source is a vertical view OCT structure region, Z is denoted as a vertical view OCT structure region-level token.

[0040] To visualize the region attention patterns of the model, the weights of all attention heads are averaged to generate a region attention map. : in, This represents the attention map corresponding to the IR / OCT structural region. Its value reflects the degree of attention the model pays to different regions under given clinical conditions, providing a basis for subsequent model interpretability analysis. This represents the total number of attention heads used in parallel computation during the multi-head attention mechanism.

[0041] It should be noted that the CTRA module of this invention adopts a lightweight text-guided region attention structure. This structure uses clinical text tokens as query vectors and focuses on converging region-level visual features related to efficacy among pre-divided IR region tokens and OCT structure region tokens, rather than performing deep sequence encoding on all image tokens. Compared with the standard Transformer Encoder structure, this invention can effectively preserve the interpretability of region division and reduce irrelevant interactions between redundant tokens by limiting the scope of attention interactions, thereby reducing the number of model parameters and mitigating the risk of overfitting.

[0042] Step S3: Construct a cross-regional collaborative enhancement mechanism from IR to OCT: This step integrates the scan localization, fundus background, and lesion context information contained in the IR region into the OCT structural region representation through a cross-regional attention mechanism, thereby achieving information complementarity between different functional regions.

[0043] like Figure 3 As shown, this invention designs an IR-to-OCT cross-regional collaborative enhancement (IOC) module. This module uses the OCT structural region token as the target representation and the IR region token as the context source. Specifically, the OCT structural region token is mapped to a query vector, and the IR region token is mapped to a key and value. The selective response of the OCT structural region to the IR context information is calculated through cross-regional attention, thereby obtaining the OCT context supplementary features guided by the IR region. In order to avoid excessive interference of IR region information to the OCT main structure representation, this invention further introduces a learnable scale parameter to adaptively control the injection intensity of IR context information and adds it to the OCT structural region token in the form of residual connections to obtain the enhanced OCT structural region representation.

[0044] The horizontal and vertical views respectively execute the above-mentioned IR to OCT cross-regional collaborative enhancement process, so that the OCT structural regions in both directions can obtain the auxiliary information of the corresponding IR regions. Through this design, the model can introduce scan localization and regional background information while preserving retinal tomographic structural information, thereby improving the model's ability to represent complex structures.

[0045] Specifically, for views This module uses an OCT structure area token. As input to the query, use the IR region token. As inputs to the Key and Value, a linear transformation is performed to obtain: in, Represents the view The query vector (Query) is obtained by linearly transforming the token in the OCT structure region. Represents a view The key vector obtained by linearly transforming the token in the IR region. Represents a view The value vector obtained by linearly transforming the token in the IR region; , , These are learnable parameters.

[0046] Subsequently, a scaled dot product attention mechanism is employed to calculate the attention weight of the OCT structural region to the IR context. Then, this weight is used to perform a weighted summation of the IR context features to achieve adaptive aggregation of IR context information to the OCT structural region. in, Represents a view Cross-regional attention weight matrix from the middle IR region to the OCT structural region; The dimension representing a single attention head; Represents the normalization function; express transpose; Indicates the first Each attention head outputs IR context supplementary features; due to the query vector The attention output originates from the OCT structure region token. In terms of spatial location, it still corresponds one-to-one with the token in the OCT structure area.

[0047] In a multi-head cross-region attention structure, each attention head calculates its corresponding IR context supplementary features according to the aforementioned process. ;in, This indicates the number of attention heads. Subsequently, the contextual supplementation features output by each attention head are concatenated along the channel dimension, and then processed through an output projection layer, layer normalization (LN), and Dropout to obtain the final OCT contextual supplementation features. in, These are the output projection parameters of the IOC module; Indicates the first The contextual supplementary features output by each attention head; in this embodiment, the number of attention heads... Each attention head has a feature dimension of 64; after the above concatenation and output projection, The dimension and the original OCT structure area token To maintain consistency, OCT tokens can continue to be injected via residuals.

[0048] To avoid excessive interference from the IR context on the OCT subject representation, this invention sets learnable scale parameters for different views in the IOC module. And the adaptive injection coefficients are obtained through the Sigmoid function: in, Represents a view Adaptive adjustment coefficients in the IR context; This represents the Sigmoid activation function; Represents the view index variable; Represents a horizontal view; This represents a vertical view.

[0049] Finally, IR contextual features are injected into the OCT structural region tokens in residual form to enhance their representation: in, Represents a view The original OCT structure region token sequence; This represents the token sequence of the enhanced OCT structural region; the residual structure design enables the model to adaptively control the injection intensity of the IR context, introducing auxiliary background information while maintaining the stability of the OCT representation; the horizontal view (H branch) and the vertical view (V branch) each use independent learnable scale parameters. , This corresponds to the adaptive adjustment coefficient of the horizontal view. Adaptive adjustment coefficient of vertical view This allows for IR-to-OCT directional enhancement in both views.

[0050] Furthermore, the enhanced OCT structure region token With the original IR region token The inputs are combined and then aggregated in the CTRA module to obtain a regional token guided by clinical text semantics. in, This represents the IR region token of the horizontal view after processing by the CTRA module; Represents a clinical text token; This represents the original IR region token in the horizontal view; This represents the IR region token of the vertical view after processing by the CTRA module; This represents the original IR region token in the vertical view; This indicates the enhanced horizontal view OCT region token after processing by the CTRA module; This represents the OCT region token in the horizontal view after context enhancement; This represents the enhanced vertical view OCT region token after processing by the CTRA module; This represents the OCT region token sequence in the vertical view after context enhancement.

[0051] Step S4: Perform multi-source token fusion and hierarchical boundary classification: After the aforementioned steps, the model has obtained the clinical text token ( ), Horizontal view IR area token ( ), Vertical view IR area token ( ), Enhanced vertical view OCT area token ( ) and the enhanced horizontal view OCT area token ( To model the global dependencies of multi-source information, this invention concatenates the aforementioned tokens in sequence into a multi-source fusion sequence. : in, 5 represents the batch size; 5 represents the total number of tokens, with the 5 tokens being clinical text features, IR region features of horizontal and vertical views, and enhanced OCT region features of horizontal and vertical views, respectively. For the token dimension.

[0052] fusion sequence Inputting a Transformer Encoder Layer, the system models the global dependencies between tokens using a self-attention mechanism, and outputs a fused token sequence. To extract a unified case-level characterization, After performing the flatten operation, the feature mapping function is shared. Obtain global representation : in, This represents a shared feature mapping function composed of linear mapping, normalization, nonlinear activation, and Dropout; in this embodiment, Features after flattening Mapped to dimension global representation To ensure consistency with subsequent first-level branch functions and the second-level branch function Input dimension matching.

[0053] To achieve hierarchical discrimination of response levels, this invention constructs a hierarchical boundary classification (HBC) module, the structure of which is as follows: Figure 4 As shown, the three-category task is decomposed into two hierarchical subtasks: The first layer is a coarse boundary discrimination, distinguishing whether a sample belongs to "good response" or "non-good response"; global representation. via branch function After transformation, the binary classification head is input, and two class probabilities are generated using Softmax: in, , These represent the probabilities of a sample being good or non-good, respectively. Representing global representation After the first level of branch function The intermediate feature representation obtained after transformation; This represents the weight matrix in the first-layer binary classification head; This represents the bias vector in the first-level binary classification head.

[0054] The second layer is a fine boundary discrimination, which further distinguishes between "stable" and "poor" responses within the "non-good" subspace; global representation. via branch function After transformation, the input binary classifier is used to generate conditional probabilities using Softmax: in, , These represent the conditional probabilities of being considered stable and poor under non-good conditions, respectively. Representing global representation After the second branch function The intermediate feature representation obtained after transformation; This represents the weight matrix in the second-layer binary classification head; This represents the bias vector in the second-level binary classification head.

[0055] Combining the results of the two-layer discrimination, the final three-class classification probability is defined as: in, The predicted probability that the sample ultimately belongs to the good category; This represents the predicted probability that a sample will eventually belong to a stable category. This represents the predicted probability that the sample will ultimately belong to the poorer category.

[0056] Through the above decomposition, the model first establishes coarse boundaries between "good" and "non-good", and then delineates fine boundaries between "stable" and "poor" in the "non-good" subspace. This avoids direct competition among the three classes on the same plane, alleviates the boundary squeezing problem caused by the dominance of the majority class (such as "good"), and improves the prediction stability of the minority class ("stable", "poor") and the boundary class.

[0057] To achieve end-to-end optimization of the hierarchical structure, this invention adopts a joint supervision strategy consistent with the hierarchical structure: the overall loss is composed of three weighted parts: in, This represents the overall joint loss function of the Hierarchical Boundary Classification (HBC) module. Used to constrain the final three-class classification result Used to supervise the good / non-good discrimination in the first stage. Used to supervise the stable and poor sub-discrimination in the second stage. , , These are the weighting coefficients used to balance the various losses.

[0058] Through the HBC module, the model simultaneously learns global category prediction, coarse boundaries, and fine boundaries, making the response level prediction results more consistent with the category structure of the task.

[0059] In this embodiment, the PyTorch platform is used for implementation, and five-fold cross-validation is employed for training and evaluation. During training, the Adam optimizer is used with an initial learning rate of 0.0001, a weight decay coefficient of 0.05, a batch size of 16, and 80 training epochs. The overall loss consists of a final three-class classification loss, a good / non-good response binary classification loss, and a stable / poor response binary classification loss. After training, the horizontal OCT image, vertical OCT image, and clinical text information of the sample to be predicted are input into the model, which then outputs three probabilities: good response, stable response, and poor response, providing auxiliary reference for subsequent medical data analysis. In this embodiment, the above parameters are optimal parameters obtained through five-fold cross-validation; they can be fine-tuned on other similar datasets according to actual conditions.

[0060] To verify the effectiveness of the above method, this invention conducted extensive comparative experiments and ablation studies on publicly available datasets. To ensure the fairness of the comparative experiments, this invention adapted and reproduced all participating models within a unified PyTorch framework, employing the same data partitioning, input preprocessing, five-fold cross-validation strategy, and evaluation metrics. For models without original code, this invention preserved their core network ideas and main structures as much as possible during reproduction. For modules in the original paper that relied on post-treatment image generation, BCVA regression, or additional fundus image input, this invention retained only the image encoding and feature fusion parts directly related to response level classification during reproduction, and added a post-fusion branch with clinical text features completely consistent with this model when necessary, to ensure that different models completed the same three-class classification prediction task under the data conditions of this invention. To verify the effectiveness of the method of this invention, the model of this invention was compared with existing methods such as HDF-Net, MDF-Net, RwithT-Net, AGC-SPP, Siamese, KongMing, M2Net, and ConvNeXt-Tiny. The experimental results are shown in Table 1: Table 1. Comparison of algorithms on clinical datasets (%) The comparison results shown in Table 1 indicate that the shallow convolutional model HDF-Net has limited capabilities in modeling complex lesion structures; ConvNeXt-T, which uses convolutional feature extraction alone, has strong local representation capabilities but lacks explicit multimodal region-level alignment; RwithT-Net and MDF-Net have improved in multimodal fusion, but still mainly focus on global-level modeling and do not fully consider regional-level clinical text guidance and category boundary reconstruction; Siamese ResNet-18 models the differences between paired OCT images using a weighted bi-branch network for three-class classification prediction of treatment response. However, its core still relies on global image-level feature stitching and lacks explicit modeling of the relationship between lesion regions and clinical text. The AGC-SPP model introduces grouped convolution, attention mechanisms, and spatial pyramid pooling on the basis of ResNet50 to enhance the multi-scale lesion representation ability in OCT images. Its performance is improved compared to traditional convolutional models. However, this method is mainly focused on visual feature enhancement and does not further utilize clinical text to actively guide key regions. The KongMing model uses a multi-task framework for lesion perception to jointly model post-treatment OCT generation, BCVA change classification, and BCVA numerical prediction. It can enhance prognostic representation by utilizing image structural changes. However, this method mainly relies on the lesion features of the OCT images themselves and lacks active guidance from clinical text for key regions. M2Net further adopts a multimodal, multi-task mutual learning strategy, jointly optimizing OCT with fundus image generation and efficacy classification tasks. It can improve prediction performance by utilizing cross-modal information, but it mainly focuses on mutual learning between image modalities and has not fully considered the guiding role of clinical text information on local visual regions.

[0061] The comparison results show that the method of this invention achieves the best results in all evaluation metrics. Compared with the best-performing existing comparative method AGC-SPP, the method of this invention improves accuracy, precision, sensitivity, F1 score, and specificity by 2.66%, 2.67%, 1.59%, 2.28%, and 1.41%, respectively. Compared with the recent multi-task method M2Net, the method of this invention improves ACC and F1 by 4.53% and 4.70%, respectively. This indicates that simply relying on a stronger convolutional backbone, multi-scale pooling, or image-level multimodal mutual learning is still insufficient to fully distinguish the three types of boundaries. However, the method of this invention, through a clinical text-guided region attention mechanism, actively introduces prior information from clinical text into the visual region localization process, enabling the model to pay more attention to IR and OCT structural regions related to changes in efficacy. The IR-to-OCT cross-regional collaborative enhancement further strengthens the discriminative features of structural lesion regions, while the hierarchical boundary classification module alleviates the problem of the stable and poor categories being easily compressed under the dominance of the good class.

[0062] Figure 5(a)-(d) Figure 6 The confusion matrices (a)-(e) visually demonstrate the classification performance of different contrastive networks in the three-class classification task and the discrimination effect of the proposed method. It can be seen that, due to the imbalanced sample distribution and similar class boundaries, various contrastive methods are generally more prone to misclassifying the poor class as the good or stable class. Furthermore, there is also some confusion between the stable and good classes, indicating that existing methods still lack sufficient ability to distinguish boundary classes and minority class samples. In contrast, the TCH-Net proposed in this invention exhibits more stable classification results in all three classification tasks.

[0063] To verify the contribution of each component module of this invention to the model performance, a module ablation experiment was conducted under the same experimental settings. The model of this invention consists of three parts: the CTRA module, the IOC module, and the HBC module. To verify the contribution of these modules to classification performance, an ablation experiment was performed, and the results are shown in Table 2. Using only the CTRA module, the accuracy was 89.89%. After introducing the IOC module, the accuracy improved by 1.86%, reaching 91.75%. Finally, after adding the HBC module, the accuracy further improved by 0.80%, ultimately reaching 92.55%. These results indicate that by gradually introducing the above modules, the model of this invention achieved a stable improvement in both feature representation and classification performance, and can more effectively predict the efficacy of anti-VEGF treatment.

[0064] Table 2 Results of ablation experiments (%) To further observe the impact of the CTRA module on visual region selection, this invention performs a visualization analysis of the attention response of typical samples, and the results are as follows: Figure 7 As shown in (a)-(l), the left column of the figures represents the original images, while the middle and right columns show the region response results with and without clinical text guidance, respectively. The visualization results show that without clinical text guidance, the model's response regions are relatively dispersed, with some attention focused on background areas less relevant to efficacy assessment. However, with clinical text guidance, the model's high-response regions are more concentrated near OCT structural abnormalities and their corresponding IR localization regions. This result indicates that clinical text information not only serves as an auxiliary feature in classification but can also guide the model to selectively aggregate visual regions in the form of queries, enabling the model to pay more attention to local structural changes related to patient disease status and efficacy prediction.

[0065] To further analyze the auxiliary value of the IR region in predicting treatment efficacy, this invention compared the model performance of using only the OCT structural region with the addition of CTRA and of extracting and fusing IR and OCT separately as two branches. The experimental results are shown in Table 3. When using only the OCT structural region, the model's ACC and F1 were 81.90% and 76.62%, respectively; after introducing the IR region, the ACC and F1 improved to 87.49% and 84.80%, respectively. This indicates that although the IR region is not the main region for final discrimination, the background scanning and fundus structural information it contains can effectively supplement OCT characterization. Therefore, in subsequent model design, this invention further adopts a targeted synergistic enhancement approach of IR→OCT, rather than simply fusing IR and OCT in parallel.

[0066] Table 3. Validation Results of the Auxiliary Value of the IOC Module (%) The HBC module proposed in this invention adopts a joint supervision strategy consistent with the hierarchical classification structure. The overall loss consists of the final three-class classification loss, the good / non-good binary classification loss, and the stable / poor binary classification loss, with their corresponding weights denoted as follows: , , Three parameters.

[0067] in, These are the weights of the main supervisory signal during model training. To ensure that the final efficacy classification task maintains its dominant position during the optimization process, this invention fixes... The good / non-good branch is mainly used to provide coarse boundary constraints. Preliminary experiments revealed that when the weight of this branch is too large, the model tends to overemphasize the coarse classification results of "good" and "non-good"; when the weight is too small, its effect on boundary constraints is not significant. Therefore, this invention fixes... This is to avoid auxiliary branches from interfering too much with the main task.

[0068] In classification tasks, due to the limited number of stable and poor samples, they are more susceptible to class imbalance and the dominance of the "good" class. Therefore, this invention focuses on... Sensitivity analysis was performed to examine the impact of different supervision intensities on hierarchical classification performance. The results are shown in Table 4 below. Experimental results show that when... At that time, the model lacked constraints on the internal structure of "stable" and "poor," resulting in insufficient ability to distinguish between the internal categories of "non-good." With... As the value increases from 0.2 to 0.8, the model's discrimination constraints on "stable" and "poor" gradually strengthen, and both ACC and F1 scores show an overall upward trend. When further increased to version 1.0, the auxiliary branches may weaken the dominant role of the final three-class main task, thus the overall performance will no longer improve. At that time, the model achieved the highest F1 score of 90.93%, while the ACC reached 92.55%, and the stable class recall also reached the highest at 89.19%. Considering the combined performance of ACC, F1, and recall for each class, this invention ultimately selected... As weights for the stable and poor branches in the hierarchical loss.

[0069] Table 4 Parameters Sensitivity analysis results (%) The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A clinical text-guided hierarchical classification method for anti-VEGF responses, characterized in that, Includes the following steps: Step S1: Perform unified preprocessing on historical medical images and structured clinical texts to construct a multimodal input dataset for anti-VEGF response prediction; Step S2: Extract features from the preprocessed OCT image, flatten the feature map and divide it into different regions based on spatial location; use the clinical text token as the query vector, and use the CTRA module to perform weighted aggregation of the above regional features to extract regional visual features guided by the clinical text. Step S3: Construct an IR to OCT cross-regional collaborative enhancement module, using the OCT structure region as the query and the IR region as the context to perform adaptive attention aggregation, realize the residual enhancement of IR information to the OCT structure, and feed the enhanced features back to the CTRA module for secondary aggregation; Step S4: Fuse and process the features of the multi-source tokens, and perform hierarchical classification of the processed features through the hierarchical boundary classification module. Calculate the overall loss using a joint supervision strategy to optimize the model parameters.

2. The clinical text-guided hierarchical classification method for anti-VEGF responses according to claim 1, characterized in that, Step S1 specifically includes: Baseline OCT images and structured clinical text information corresponding to historical medical samples are obtained; the historical medical samples are those that have previously received anti-VEGF intervention; the OCT images include horizontal and vertical views; Based on the actual observation records of the samples during the follow-up process and the preset clinical interpretation criteria, response level labels are labeled for each sample to construct a labeled training sample set; the response level labels include three categories: good response, stable response, and poor response; The input images undergo uniform preprocessing, including resizing to a preset size and normalizing. Structured clinical text information is then processed, including encoding categorical variables and standardizing numerical variables to form clinical text feature vectors. ; via text encoder Convert to text query token: in, This represents the output text query token; Presentation layer normalization operation; Indicates the activation function; , These are learnable parameters in the text encoder; The preprocessed multimodal data is input into the TCH-Net network for subsequent feature extraction and classification.

3. The clinical text-guided hierarchical classification method for anti-VEGF responses according to claim 1, characterized in that, In step S2, the feature extraction and spatial location division specifically include: The horizontal and vertical views of the OCT image are input into two independent ConvNeXt-Tiny branches for feature extraction. Both branches are initialized with ImageNet pre-trained weights, and their feature extraction parts are used as visual encoders. The feature maps are flattened and converted into spatial token sequences. Combining the spatial location information of the OCT image, the spatial token sequence of each view is divided into two types of region tokens: IR region tokens and OCT structure region tokens.

4. The clinical text-guided hierarchical classification method for anti-VEGF responses according to claim 3, characterized in that, In step S2, the extraction of region-level visual features guided by clinical text specifically includes: Clinical text token sequence For query vector, with Indicates the view type; where, This represents a horizontal view. Represents a vertical view; for any view Let its IR region token and OCT structure region token be respectively denoted as , Let the OCT structure region token and IR region token of the horizontal view be respectively... 、 The OCT structure region token and IR region token of the vertical view are respectively 、 ; Introducing symbols As a general placeholder, it is used to refer to any region of the token sequence in the current input module, i.e. ; Through linear transformation , These are respectively mapped to the query vectors required for attention calculation. Key vector Sum value vector Employing a multi-head attention mechanism, , , The feature dimension is uniformly divided into multiple sub-vectors. The weights of each attention head to the region token are calculated by scaling dot product attention, and the sub-vectors of the corresponding value vector are weighted and summed to obtain the region aggregation features of each attention head. The region aggregation features of all attention heads are concatenated along the feature dimension, and the final region convergence features are obtained by linear mapping, layer normalization and Dropout regularization operations. The weights of all attention heads are averaged to generate the region attention map.

5. The clinical text-guided hierarchical classification method for anti-VEGF responses according to claim 1, characterized in that, In step S3, the specific implementation process of the IR to OCT cross-regional collaborative enhancement module is as follows: For views token with OCT structure area As a query vector, with IR region token As the key vector and value vector, the query vector is obtained through linear transformation. Key vector Sum value vector ; The scaling dot product attention mechanism is used to calculate the attention weights of the OCT structural region to the IR context, and the IR context supplementary features of the single attention head output are obtained by weighted summation. A multi-head cross-regional attention structure is adopted, with each attention head performing the above calculations in parallel. The context supplementation features output by each attention head are concatenated along the channel dimension, and then processed by an output projection layer, layer normalization, and Dropout to obtain the final OCT context supplementation features. ; Set learnable scale parameters Adaptive injection coefficients are generated using the Sigmoid function. The IR context features are injected as residuals into the OCT structural region token to obtain the enhanced OCT structural region token. ;in, This represents the Sigmoid activation function; Represents a view The original OCT structure region token sequence; This represents the token sequence of the enhanced OCT structure region.

6. The clinical text-guided hierarchical classification method for anti-VEGF responses according to claim 5, characterized in that, Step S3 further includes: The enhanced OCT structure area token With the original IR region token The inputs are combined and then aggregated in the CTRA module to obtain a regional token guided by clinical text semantics. in, This represents the IR region token of the horizontal view after processing by the CTRA module; Represents a clinical text token; This represents the original IR region token in the horizontal view; This represents the IR region token of the vertical view after processing by the CTRA module; This represents the original IR region token in the vertical view; This indicates the enhanced horizontal view OCT region token after processing by the CTRA module; This represents the OCT region token in the horizontal view after context enhancement; This represents the enhanced vertical view OCT region token after processing by the CTRA module; This represents the OCT region token sequence in the vertical view after context enhancement.

7. The clinical text-guided hierarchical classification method for anti-VEGF responses according to claim 1, characterized in that, In step S4, the fusion and feature processing of the multi-source tokens specifically includes: The clinical text token, the horizontal view IR region token, the vertical view IR region token, the enhanced vertical view OCT region token, and the enhanced horizontal view OCT region token are sequentially concatenated into a multi-source fusion sequence. This fusion sequence is then input into the Transformer Encoder Layer, where a self-attention mechanism is used to model the global dependencies between tokens, outputting the fused token sequence. ;right After performing the flattening operation, the shared feature mapping function is used. A global representation is obtained.

8. The clinical text-guided hierarchical classification method for anti-VEGF responses according to claim 7, characterized in that, In step S4, the specific implementation process of the hierarchical boundary classification module includes: The three-category task is broken down into two hierarchical sub-tasks: The first layer is a coarse boundary discrimination, distinguishing whether a sample belongs to a good response or a bad response; global representation. via branch function After transformation, the binary classification head is input, and two class probabilities are generated using Softmax: in, , These represent the probabilities of a sample being good or non-good, respectively. Representing global representation After the first level of branch function The intermediate feature representation obtained after transformation; This represents the weight matrix in the first-layer binary classification head; This represents the bias vector in the first-level binary classification head; The second layer is fine boundary discrimination, which further distinguishes between stable and poor responses within the non-good response subspace; global characterization. via branch function After transformation, the input binary classifier is used to generate conditional probabilities using Softmax: in, , These represent the conditional probabilities of being considered stable and poor under non-good conditions, respectively. Representing global representation After the second branch function The intermediate feature representation obtained after transformation; This represents the weight matrix in the second-layer binary classification head; This represents the bias vector in the second-layer binary classification head; Combining the results of the two-layer discrimination, the final three-class classification probability is defined as: in, The predicted probability that the sample ultimately belongs to the good category; This represents the predicted probability that a sample will eventually belong to a stable category. This represents the predicted probability that the sample will ultimately belong to the poorer category.

9. The clinical text-guided hierarchical classification method for anti-VEGF responses according to claim 8, characterized in that, The total loss is composed of three weighted parts: in, This represents the overall joint loss function of the hierarchical boundary classification module. Used to constrain the final three-class classification result Used to monitor the distinction between good and bad responses in the first phase. Used to monitor response stability and response difference differentiation in the second stage. , , These are the weighting coefficients used to balance the various losses.

Images