A refined strabismus diagnosis method, system, device, and medium based on a hierarchical decoupled attention mechanism.

Abstract

This invention discloses a refined strabismus diagnosis method, system, device, and medium based on a hierarchical decoupled attention mechanism. The method includes: acquiring nine facial photographs of a patient and performing image preprocessing on each photograph to construct a nine-eye bitmap; dividing the nine-eye bitmap into multiple blocks of the same size, mapping these blocks to corresponding block embeddings, and performing positional encoding on these embeddings to form a nine-eye representation map; sequentially extracting intraocular detail features and interocular variation features from the nine-eye representation map using local multi-head attention, and rearranging these features to form a refined representation map; globally pooling the refined representation map to form a global descriptor, and using a combined loss function to train and optimize the global descriptor input to a hierarchical classifier to output a refined diagnostic result. This invention guides the model to simultaneously capture detailed features within each eye region and simulate subtle changes between different eye regions through a hierarchical decoupled attention mechanism, thereby achieving refined strabismus diagnosis.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a refined strabismus diagnosis method, system, device, and medium based on a hierarchical decoupled attention mechanism. Background Technology

[0002] Strabismus is a common eye misalignment characterized by incorrect eye positioning when focusing on objects. This condition is particularly prevalent in children and, if left untreated, can develop into severe binocular vision impairment. In addition to its direct impact on vision, strabismus can also have profound psychological effects on those affected. Fortunately, surgical intervention can effectively treat strabismus. However, the symptoms of strabismus are diverse, and the type of surgery required depends on an accurate diagnosis. Currently, the diagnosis of strabismus relies heavily on specialist eye examinations, such as the Cover test and the Hirschberg test. These examinations require specialized techniques and equipment, typically only available in larger hospitals or specialist clinics.

[0003] To assist doctors in diagnosis and reduce their workload, the development of automated strabismus diagnostic systems is increasingly being promoted. Diagnostic results can be automatically generated after taking facial images. One common method is based on the Hirschberg test, which relies on manually designed pathological features. Alternatively, deep neural networks offer a solution because they can automatically learn features from raw data to detect strabismus in patients.

[0004] Because strabismus encompasses various types, including esotropia, exotropia, and vertical strabismus, further subdivided into A-type strabismus, V-type strabismus, and non-AV-type strabismus, it is difficult for models to distinguish the fine-grained features of different strabismus types. Existing strabismus detection models cannot provide refined diagnoses, offering only coarse-grained diagnoses, such as simply identifying whether a patient has strabismus, which is insufficient for precise clinical treatment. Furthermore, existing strabismus detection models are trained and tested on small datasets, typically collected from volunteers, making it difficult to obtain sufficient samples covering all strabismus types and thus hindering good generalization performance. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a refined strabismus diagnosis method, system, device and medium based on a hierarchical decoupled attention mechanism. This invention guides the model to simultaneously capture detailed features within each eye region through a hierarchical decoupled attention mechanism and simulates subtle changes between different eye regions, thereby achieving refined strabismus diagnosis.

[0006] In a first aspect, the present invention provides a refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism, comprising the following steps: Step S11: Obtain nine facial photos of the patient and perform image preprocessing on the nine facial photos to construct a nine-eye bitmap. The nine facial photos are photos of the patient gazing in a specific direction, including the upper left, directly above, upper right, directly left, directly in front, directly right, lower left, directly below, and lower right. Step S12: After dividing the nine-eye bitmap into multiple blocks of the same size, the multiple blocks are mapped into multiple corresponding block embeddings, and the multiple block embeddings are positionally encoded to form a nine-eye bitmap representation. Step S13: Using local multi-head attention, extract the intraocular detail features and interocular variation features from the nine-eye representation map sequentially, and rearrange the intraocular detail features and interocular variation features to form a refined representation map. Step S14: The refined representation map is globally pooled to form a global descriptor, and a combined loss function is used to train and optimize the global descriptor input to the hierarchical classifier to output a refined diagnostic result. The combined loss function... Includes a direction loss function for directional strabismus recognition. Degree loss function for strabismus degree change recognition They are expressed as follows: ; ; ; In the formula, Indicates the category of directional strabismus. Indicate category The true label, Indicate category The predicted probability of 1 Categories representing changes in strabismus degree, Indicate category The true label, Indicate category The predicted probability of 2.

[0007] Preferably, in step S11, the step of preprocessing the nine facial photographs includes: Step S111: Use the dlib library to detect facial landmarks in each facial image, and calculate the horizontal distance between the two eyes based on the facial landmarks. and vertical distance ; Step S112, define the horizontal distance centered on the midpoint between the two eyes. and vertical distance A rectangle with side length is used as the main eye region for each facial image, and a nine-eye bitmap is constructed using nine main eye regions. The length of each main eye region is . × Width is × relaxation coefficient =1.2, relaxation coefficient =1.4.

[0008] Preferably, in step S11, the nine facial photos are acquired by taking photos of the patient gazing in a specific direction.

[0009] Preferably, in step S11, the acquisition of the nine facial photos includes the following steps: Video recording equipment was used to capture video of the patient. During the video capture process, the patient's eyes turned sequentially according to the direction of the arrows displayed on the recording equipment. Nine facial photos were extracted from the captured video based on the video frames.

[0010] Preferably, step S13 specifically includes: Step S131: Extract the intraocular features of each eye in the nine-eye representation map according to the three independent projections in the Transformer encoder to generate intraocular Query feature value, intraocular Value feature value and intraocular Key feature value for each eye position; Step S132: Multi-head attention is used to extract and interactively compare the intraocular Query feature value, intraocular Value feature value and intraocular Key feature value of each eye position in order to learn the intraocular detail features of each eye position. Step S133: Extract the features of the same corresponding positions of different eye positions in the nine-eye representation map according to the three independent projections in the Transformer encoder, so as to generate inter-eye Query feature value, inter-eye Value feature value and inter-eye Key feature value for different eye positions. Step S134: Multi-head attention is used to extract and combine interocular query feature values, interocular value feature values ​​and interocular key feature values ​​for different eye positions in order to capture the interocular variation features in different gaze directions. Step S135: The intraocular detail features and the interocular variation features are rearranged to form a refined characterization map.

[0011] Preferably, in step S14, the construction of the strabismus dataset for training and optimization includes the following steps: Nine facial photos of multiple patients were obtained, and image preprocessing was performed on the nine facial photos of each patient to construct the corresponding nine-eye bitmap; By having senior ophthalmologists perform fine-grained diagnosis of each patient's nine-eye bitmap, and using each patient's fine-grained diagnosis as a true annotation of the corresponding patient's nine-eye bitmap, a strabismus disease dataset is constructed. The strabismus dataset was divided into training, validation, and test subsets with a split ratio of 6:3:1.

[0012] Secondly, the present invention provides a refined strabismus diagnosis system based on a hierarchical decoupled attention mechanism, comprising: The image preprocessing module is used to acquire nine facial photos of the patient and perform image preprocessing on the nine facial photos to construct a nine-eye bitmap. The nine facial photos are photos of the patient gazing in a specific direction, which includes the upper left, directly above, upper right, directly left, directly in front, directly right, lower left, directly below, and lower right. The position encoding module is used to map the multiple blocks into corresponding multiple block embeddings after dividing the nine-eye bitmap into multiple blocks of the same size, and to perform position encoding on the multiple block embeddings to form a nine-eye bitmap representation map. The feature extraction and arrangement module is used to sequentially extract intraocular detail features and interocular variation features from the nine-eye representation map using local multi-head attention, and rearrange the intraocular detail features and interocular variation features to form a refined representation map. The pooling and optimization module is used to perform global pooling on the refined representation map to form a global descriptor, and to train and optimize the global descriptor input to the hierarchical classifier using a combined loss function to output a refined diagnostic result. The combined loss function... Includes a direction loss function for directional strabismus recognition. Degree loss function for strabismus degree change recognition They are expressed as follows: ; ; ; In the formula, Indicates the category of directional strabismus. Indicate category The true label, Indicate category The predicted probability of 1 Categories representing changes in strabismus degree, Indicate category The true label, Indicate category The predicted probability of 2.

[0013] Preferably, the feature extraction and arrangement module includes: The first feature extraction unit is used to extract each intraocular feature from the nine-eye representation map based on three independent projections in the Transformer encoder, so as to generate intraocular Query feature value, intraocular Value feature value and intraocular Key feature value for each eye position; The feature comparison learning unit uses multi-head attention to extract and interactively compare the intraocular query feature value, intraocular value value, and intraocular key feature value for each eye position in order to learn the intraocular detailed features of each eye position. The second feature extraction unit is used to extract features from the same corresponding positions of different eye positions in the nine-eye representation map based on the three independent projections in the Transformer encoder, so as to generate inter-eye Query feature values, inter-eye Value feature values ​​and inter-eye Key feature values ​​for different eye positions. The feature combination unit is used to extract and combine interocular query feature values, interocular value feature values ​​and interocular key feature values ​​from different eye positions using multi-head attention, so as to capture interocular variation features in different gaze directions; The feature arrangement unit is used to rearrange the intraocular detail features and the interocular variation features to form a refined characterization map.

[0014] Thirdly, the present invention provides a computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism as described in the first aspect.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism as described in the first aspect.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: First, by removing irrelevant details from the entire facial image, the main eye region is extracted to construct a nine-eye bitmap, thereby reducing computational costs. Second, by using a hierarchical decoupling attention mechanism, the model is guided to simultaneously capture detailed features within each eye region and simulate subtle changes between different eye regions, thereby achieving refined strabismus diagnosis. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism in this invention. Figure 2 for Figure 1A simplified flowchart illustrating the refined strabismus diagnosis method. Figure 3 for Figure 2 A simplified flowchart illustrating the hierarchical decoupling process in the middle layer; Figure 4 for Figure 2 A flowchart illustrating the decoupling attention mechanism in the middle; Figure 5 This is a typical nine-eye bitmap of several types in this invention and a distribution map of different fine-grained strabismus categories in the dataset; Figure 6 This is a schematic diagram of the composition of the refined strabismus diagnosis system based on the hierarchical decoupling attention mechanism in this invention; Figure 7 A heatmap for visual comparison between the original global attention mechanism and the hierarchical decoupled attention mechanism proposed in this invention.

[0018] Figure 8 This invention extends visualization to heatmaps of several patients exhibiting different forms of fine-grained strabismus. Detailed Implementation

[0019] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0020] Please see Figures 1 to 3 An embodiment of the present invention provides a refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism, comprising the following steps: Step S11: Obtain nine facial photos of the patient and perform image preprocessing on the nine facial photos to construct a nine-eye bitmap. The nine facial photos are photos of the patient gazing in a specific direction, including the upper left, directly above, upper right, directly left, directly in front, directly right, lower left, directly below, and lower right. Step S12: After dividing the nine-eye bitmap into multiple blocks of the same size, the multiple blocks are mapped into multiple corresponding block embeddings, and the multiple block embeddings are positionally encoded to form a nine-eye bitmap representation. Step S13: Using local multi-head attention, extract the intraocular detail features and interocular variation features from the nine-eye representation map sequentially, and rearrange the intraocular detail features and interocular variation features to form a refined representation map. Step S14: The refined representation map is globally pooled to form a global descriptor, and a combined loss function is used to train and optimize the global descriptor input to the hierarchical classifier to output a refined diagnostic result. The combined loss function... Includes a direction loss function for directional strabismus recognition. Degree loss function for strabismus degree change recognition They are expressed as follows: ; ; ; In the formula, Indicates the category of directional strabismus. Indicate category The true label, Indicate category The predicted probability of 1 Categories representing changes in strabismus degree, Indicate category The true label, Indicate category The predicted probability of 2.

[0021] Please see Figure 2 and Figure 3 It should be noted that in this invention, after image preprocessing, the nine-eye bitmap is processed in a hierarchical decoupled attention mechanism for comprehensive strabismus diagnosis. Based on the original architecture of the attention mechanism, the method of this invention enhances two key aspects: the decoupled attention mechanism and the hierarchical classifier, such as... Figure 3 As shown. Specifically, the nine-eye bitmap is divided into blocks of the same size and then mapped to block embeddings. To preserve spatial relationships, positional encodings are incorporated into these embeddings. This step ensures that the Transformer encoder effectively captures intraocular details and interocular variations. Throughout the process, the decoupled attention mechanism of this invention accurately extracts relevant information within each eye region while discerning subtle differences between different regions of the eye. The resulting refined representations are incorporated into the global embeddings and then processed by a hierarchical classifier to predict refined diagnostic results.

[0022] Please see Figure 2 Achieving fine-grained diagnostic results requires identifying directional strabismus (i.e., esotropia, exotropia, and vertical strabismus) and strabismus degree changes (i.e., A-type strabismus, V-type strabismus, and non-AV-type strabismus). However, since some categories contain only a small number of samples, treating each combination as a specific category leads to a scarcity of data required for effective model training. To address this challenge, this invention develops a hierarchical classifier consisting of two independent task heads: one for directional strabismus identification and the other for strabismus degree change identification, such as... Figure 3As shown in the diagram, in this method, the fine representation obtained from the transformer encoder is aggregated into a comprehensive global descriptor to capture all the necessary information from the nine-eye bitmap. This descriptor is then fed into a hierarchical classifier that simultaneously generates two recognition results: one for directional strabismus and the other for degree variation. These individual results are combined to produce the final output.

[0023] Furthermore, since the hierarchical classifier uses two independent task heads to process multiple recognition tasks simultaneously, this invention employs a cross-entropy loss function for each classification task. That is, two different loss functions and corresponding supervision signals are applied. Specifically, the categories of directional strabismus include esotropia, exotropia, and vertical strabismus, while the categories of strabismus degree variation include A-type strabismus, V-type strabismus, and non-AV-type strabismus.

[0024] In a preferred embodiment of the present invention, step S11, which involves preprocessing the nine facial photographs, includes: Step S111: Use the dlib library to detect facial landmarks in each facial image, and calculate the horizontal distance between the two eyes based on the facial landmarks. and vertical distance ; Step S112, define the horizontal distance centered on the midpoint between the two eyes. and vertical distance A rectangle with side length is used as the main eye region for each facial image, and a nine-eye bitmap is constructed using nine main eye regions. The length of each main eye region is . × Width is × relaxation coefficient =1.2, relaxation coefficient =1.4.

[0025] It should be noted that the eyes occupy a relatively small area in facial photographs. Applying a recognition model directly to the entire facial image may introduce irrelevant details, such as hairstyle, nose, and mouth, which could confuse the diagnostic process and unnecessarily increase computational costs. To address this issue, image preprocessing is performed to extract the main eye region. Furthermore, a relaxation coefficient is set... and relaxation coefficient To adjust the corresponding horizontal distance and vertical distance .

[0026] In a preferred embodiment of the present invention, in step S11, the nine facial photos are acquired by taking photos of the patient gazing in a specific direction.

[0027] Specifically, enter the diagnostic software (such as a mini-program on your phone), click the diagnostic button, and a shooting interface will pop up. Place your face within the portrait frame, and then follow the arrow prompts (upper left, upper right, upper right, left, right, lower left, lower right, and lower right) to click the red button in sequence to take nine facial photos.

[0028] In another preferred embodiment of the present invention, in step S11, the acquisition of the nine facial photographs includes the following steps: Video recording equipment was used to capture video of the patient. During the video capture process, the patient's eyes turned sequentially according to the direction of the arrows displayed on the recording equipment. Nine facial photos were extracted from the captured video based on the video frames.

[0029] Specifically, in the diagnostic software (such as a mini-program on your phone), click the diagnostic button to bring up the recording interface. Place your face within the image frame and click the red button to record in sequence. During recording, eight arrows will appear to indicate directions (upper left, directly above, upper right, directly left, directly right, lower left, directly below, and lower right). Follow the direction of the arrows with your eyes, and then the recording will be complete. After capturing the video, the system will use the MediaPipe face recognition library to extract the eye area from the images captured in the video frames, resulting in nine facial photos.

[0030] Please see Figure 3 and Figure 4 In a preferred embodiment of the present invention, step S13 specifically includes: Step S131: Extract the intraocular features of each eye in the nine-eye representation map according to the three independent projections in the Transformer encoder to generate intraocular Query feature value, intraocular Value feature value and intraocular Key feature value for each eye position; Step S132: Multi-head attention is used to extract and interactively compare the intraocular Query feature value, intraocular Value feature value and intraocular Key feature value of each eye position in order to learn the intraocular detail features of each eye position. Step S133: Extract the features of the same corresponding positions of different eye positions in the nine-eye representation map according to the three independent projections in the Transformer encoder, so as to generate inter-eye Query feature value, inter-eye Value feature value and inter-eye Key feature value for different eye positions. Step S134: Multi-head attention is used to extract and combine interocular query feature values, interocular value feature values ​​and interocular key feature values ​​for different eye positions in order to capture the interocular variation features in different gaze directions. Step S135: The intraocular detail features and the interocular variation features are rearranged to form a refined characterization map.

[0031] It should be noted that the method of this invention draws insights from the methods used by ophthalmologists when analyzing nine primary fixation positions, which involves two main stages: first, examining each individual eye region to assess its morphology; and second, comparing eye behavior across different regions. This method has proven crucial for discerning subtle differences between fine-grained strabismus types, such as A-type and V-type strabismus. To integrate this critical expert knowledge, this invention optimizes the global attention mechanism of the original attention mechanism and designs a decoupled attention mechanism with two key components: intraocular attention and interocular attention, such as… Figure 4 As shown, this invention simulates how doctors observe and diagnose strabismus patients, comparing the results when the patient is looking in one direction and when the patient is looking in different directions.

[0032] Specifically, this invention uses a Transformer encoder to extract image features. After uniformly dividing the nine-eye representation image into small blocks, three independent projections are used to generate Query, Value, and Key feature values ​​to represent the information in the nine-eye representation image. Unlike traditional multi-head attention in attention mechanisms, the eye position images in this invention are composite images obtained after the child observes nine different directions. Therefore, by uniformly dividing the image into small blocks, each block contains image information from only one eye position direction. This allows for comparison and learning between different blocks in the same direction when observing the patient's gaze in one direction; similarly, when observing the patient's gaze in different directions, comparison and learning can be performed between corresponding blocks in different directions.

[0033] Specifically, the arrangement embedded in the nine-eye structure divides them into nine eye-position directions. When observing information about a patient's gaze in one direction, a local window multi-head attention operation is applied to facilitate feature extraction and interactive comparison within the current eye position, in order to learn as much information as possible about the current eye position. Subsequently, information from corresponding positions across different eye positions is extracted and combined through an arrangement operation, and then another round of local window multi-head attention is performed to capture changes in the eyes across different gaze directions. Finally, the intraocular detail features and interocular variation features acquired through the arrangement are used to form a refined representation map.

[0034] In a preferred embodiment of the present invention, in step S14, the construction of the strabismus dataset for training optimization includes the following steps: Nine facial photos of multiple patients were obtained, and image preprocessing was performed on the nine facial photos of each patient to construct the corresponding nine-eye bitmap; By having senior ophthalmologists perform fine-grained diagnosis of each patient's nine-eye bitmap, and using each patient's fine-grained diagnosis as a true annotation of the corresponding patient's nine-eye bitmap, a strabismus disease dataset is constructed. The strabismus dataset was divided into training, validation, and test subsets with a split ratio of 6:3:1.

[0035] It should be noted that existing strabismus recognition datasets are typically limited by sample size and coarse-grained annotation. In this invention, the applicant addresses the need for fine-grained strabismus recognition by compiling a large-scale dataset containing 1075 strabismus patients from Xinhua Hospital affiliated with Shanghai Jiao Tong University School of Medicine. Each patient contributed nine facial photographs taken while gazing in a specific direction: upper left, upper, upper right, left, front, right, lower left, lower, and lower right. Each photograph had a resolution of 2592×1944 pixels. Subsequently, these images were processed in the image preprocessing stage to construct a nine-eye bitmap (e.g., ...). Figure 2 (As shown). Each chart was resized to 2400×900 pixels. Furthermore, each patient received a fine-grained diagnosis from a senior ophthalmologist, serving as a true annotation to the dataset. For clarity and illustration, Figure 5 Several typical nine-eye diagrams are shown, along with the distribution of different fine-grained strabismus categories in our dataset. We divided the dataset into training, validation, and test subsets using a 6:3:1 split ratio.

[0036] Please see Figure 6 This invention provides a refined strabismus diagnosis system based on a hierarchical decoupled attention mechanism, comprising: The image preprocessing module is used to acquire nine facial photos of the patient and perform image preprocessing on the nine facial photos to construct a nine-eye bitmap. The nine facial photos are photos of the patient gazing in a specific direction, which includes the upper left, directly above, upper right, directly left, directly in front, directly right, lower left, directly below, and lower right. The position encoding module is used to map the multiple blocks into corresponding multiple block embeddings after dividing the nine-eye bitmap into multiple blocks of the same size, and to perform position encoding on the multiple block embeddings to form a nine-eye bitmap representation map. The feature extraction and arrangement module is used to sequentially extract intraocular detail features and interocular variation features from the nine-eye representation map using local multi-head attention, and rearrange the intraocular detail features and interocular variation features to form a refined representation map. The pooling and optimization module is used to perform global pooling on the refined representation map to form a global descriptor, and to train and optimize the global descriptor input to the hierarchical classifier using a combined loss function to output a refined diagnostic result. The combined loss function... Includes a direction loss function for directional strabismus recognition. Degree loss function for strabismus degree change recognition They are expressed as follows: ; ; ; In the formula, Indicates the category of directional strabismus. Indicate category The true label, Indicate category The predicted probability of 1 Categories representing changes in strabismus degree, Indicate category The true label, Indicate category The predicted probability of 2.

[0037] Preferably, the feature extraction and arrangement module includes: The first feature extraction unit is used to extract each intraocular feature from the nine-eye representation map based on three independent projections in the Transformer encoder, so as to generate intraocular Query feature value, intraocular Value feature value and intraocular Key feature value for each eye position; The feature comparison learning unit uses multi-head attention to extract and interactively compare the intraocular query feature value, intraocular value value, and intraocular key feature value for each eye position in order to learn the intraocular detailed features of each eye position. The second feature extraction unit is used to extract features from the same corresponding positions of different eye positions in the nine-eye representation map based on the three independent projections in the Transformer encoder, so as to generate inter-eye Query feature values, inter-eye Value feature values ​​and inter-eye Key feature values ​​for different eye positions. The feature combination unit is used to extract and combine interocular query feature values, interocular value feature values ​​and interocular key feature values ​​from different eye positions using multi-head attention, so as to capture interocular variation features in different gaze directions; The feature arrangement unit is used to rearrange the intraocular detail features and the interocular variation features to form a refined characterization map.

[0038] The content of the above method embodiments is applicable to this system embodiment. The functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are the same as those in the above method embodiments. Therefore, they will not be repeated here.

[0039] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements a refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism as described in the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium on which a device (e.g., a computer, mobile phone, etc.) stores or transmits information in a readable form, and can be a read-only memory, a disk, or an optical disk, etc.

[0040] As one embodiment, the computer device includes a processor, a memory, and a computer program, wherein the computer program is stored in the memory and configured to be executed by the processor, and the computer program is configured to perform a refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism as described in the above embodiment.

[0041] The present invention will now be described in detail with reference to specific embodiments: In this invention, the proposed strabismus diagnosis method is implemented using the PyTorch library, a popular choice for deep learning research. The experimental computing platform consisted of an Intel Xeon Gold 5115 CPU running at 2.40 GHz, coupled with an NVIDIA GeForce RTX 2080Ti GPU and 32GB of RAM. During the training phase, we used the Adaptive Moment Estimation (Adam) optimizer with a learning rate of 0.01 and a weight decay of 0.001. Each iteration processed 32 samples to effectively update the model parameters.

[0042] Consistent with standard practices for classification tasks, the evaluation metrics in this invention include overall accuracy, sensitivity, specificity, and F1 score. Furthermore, to comprehensively evaluate the performance of the model across different recognition targets (i.e., directional strabismus recognition and angle change recognition), the corresponding accuracy for each task was also calculated. After evaluating a subset of tests, TP (True Positive), FP (False Positive), TN (True Negative), and FN (False Negative) were calculated based on provided annotations. These metrics enable this invention to quantify the effectiveness of the model in correctly identifying strabismus types and angle changes.

[0043] 1) Overall Accuracy: In real-world diagnostic scenarios, accuracy plays a crucial role in evaluating model performance, where correct diagnosis requires accurate identification of every fine-grained symptom. Mathematically, overall accuracy (Acc) is defined as: .

[0044] 2) Orientation Accuracy: Orientation accuracy is a key indicator for evaluating the predictive performance of a model for directional strabismus. Mathematically, orientation accuracy (Acc) is... Dir Defined as: ; In the formula, TP Dir FP Dir TN Dir and FN Dir Directional accuracy for true positives, false positives, true negatives, and false negatives in directional strabismus samples, respectively.

[0045] 3) Angle accuracy: This invention categorizes the angle accuracy (Acc) of strabismus angle changes. Deg Defined as: ; In the formula, TP Deg FP Deg TN Deg and FN Deg The angular accuracy for true positives, false positives, true negatives, and false negatives of samples classified by angular changes.

[0046] 4) Precision: Precision measures the proportion of correctly predicted positive samples out of all predicted positive samples, reflecting the network's ability to accurately identify positive instances. The formal definition of precision is: .

[0047] 5) Recall: Recall, also known as sensitivity, measures the proportion of correctly predicted positive samples out of all actual positive samples in a dataset. It measures the network's ability to avoid missing positive instances in disease prediction scenarios. The formula for recall is: .

[0048] 6) F1 Score: The F1 score comprehensively evaluates the network's performance as a median average of precision and recall. It provides a balanced metric that considers both precision and recall. The formula for the F1 score is defined as: .

[0049] The applicant compared the method of this invention with state-of-the-art end-to-end classification models, including VGG (Very Deep Convolutional Networks for Large-Scale Image Recognition), ResNet (Deep Residual Learning for Image Recognition), DenseNet (Densely Connected Convolutional Networks), ViT (An image is worth 16x16 words: Transformers for Image Recognition at Scale), and Swin ViT (Swin transformer: Hierarchical Vision Transformer using shifted windows) for fine-grained strabismus recognition. To ensure a fair comparison, a hierarchical classifier was uniformly integrated into each model during the comparison process. Key components such as data partitioning, data augmentation, and training settings remained consistent across all methods.

[0050] Table 1 Performance comparison between the model of this invention and mainstream classification models The experimental results are shown in Table 1. Several observations can be drawn from these results. First, compared with other mainstream classification models, the method of this invention shows superior performance, verifying its effectiveness. Second, using ViT as the base model, the method of this invention represents a substantial improvement over ViT itself, with an overall accuracy increase of 6.6%, which emphasizes the effectiveness of the method of this invention.

[0051] Furthermore, the effectiveness of the method proposed in this invention is verified through experiments.

[0052] 1) Impact of the Attention Module: This invention aims to decouple the global attention mechanism of the original ViT by introducing two distinct modules: intraocular attention and interocular attention, inspired by ophthalmologist observation methods. To evaluate the impact of these attention mechanisms, an ablation study was conducted to assess their effects independently and integrated. The results are summarized in Table 2, demonstrating that both intraocular and interocular attention lead to performance improvements compared to a baseline using only global attention. Notably, the method of this invention achieves optimal performance when both components are integrated simultaneously.

[0053] Table 2 Ablation experiments of different attention modules 2) Computational Cost and Parameters: The hierarchical decoupled attention mechanism proposed in this invention decomposes global attention computation into two independent processes, significantly reducing computational cost. To verify its effectiveness compared to the benchmark ViT model, a comparative analysis was conducted, as shown in Table 3. The results show that the method of this invention reduces computational cost by nearly 50%.

[0054] Table 3 Ablation experiments of different attention modules 3) Visualization: To intuitively understand the hierarchical decoupling attention mechanism of this invention, attention visualization is used to compare it with the benchmark ViT and typical patients. Following the conventional approach to attention visualization [Ccnet: Criss-cross attention for se-mantic segmentation] and [Psanet: Point-wise spatial attention network for scene parsing], this invention first selects a key point (in... Figure 7 and Figure 8 The method of this invention (marked with a red cross x) is used to observe and calculate the attention weights of other image regions, visually demonstrating the ability of this mechanism to collect relevant information.

[0055] Please see Figure 7 In this invention, the intraocular attention guidance model focuses attention on another eye within the same eye region, while the interocular attention guidance model focuses attention on the same eye in different regions to capture its changing patterns. In contrast, the original attention mechanism calculates relationships with all other blocks, potentially merging irrelevant information and complicating attention to fine-grained but crucial details. This visual comparison illustrates how the hierarchical decoupled attention of this invention outperforms the original global attention.

[0056] Please see Figure 8 This invention extends visualization to several patients exhibiting different forms of fine-grained strabismus. For example, the diagnosis of esotropia depends on horizontal gaze position, prompting the model to emphasize this aspect. In cases like exotropia with the V sign and vertical strabismus, attentional mechanisms must take into account variations in both horizontal and vertical eye position. The heatmaps show that the model of this invention produces more interpretable results, which is crucial for medical applications.

[0057] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A fine strabismus diagnosis method based on a hierarchical decoupled attention mechanism, characterized in that, Includes the following steps: Step S11: Obtain nine facial photos of the patient and perform image preprocessing on the nine facial photos to construct a nine-eye bitmap. The nine facial photos are photos of the patient gazing in a specific direction, including the upper left, directly above, upper right, directly left, directly in front, directly right, lower left, directly below, and lower right. Step S12: After dividing the nine-eye bitmap into multiple blocks of the same size, the multiple blocks are mapped into multiple corresponding block embeddings, and the multiple block embeddings are positionally encoded to form a nine-eye bitmap representation. Step S13: Using local multi-head attention, extract the intraocular detail features and interocular variation features from the nine-eye representation map sequentially, and rearrange the intraocular detail features and interocular variation features to form a refined representation map. Step S14, globally pooling the refined characterization map to form a global descriptor, and training and optimizing the global descriptor input into the hierarchical classifier with a combined loss function to output a refined diagnosis result, wherein the combined loss function includes a direction loss function for identifying directional strabismus and a degree loss function for identifying strabismus degree variation , respectively expressed as: ； ； ； In the formula, denotes the category of directional strabismus, denotes the category the true label, denotes the predicted probability of the category 1, denotes the category of strabismus degree change, denotes the true label of the category , and denotes the predicted probability of the category 2.

2. The fine strabismus diagnosis method based on hierarchical decoupled attention mechanism according to claim 1, characterized in that, In step S11, the step of preprocessing the nine facial photographs includes: Step S111, the face mark of each face picture is detected by using the dlib library, and the horizontal distance between the two eyes is calculated according to the face mark and the vertical distance ; Step S112, defining a rectangle with the midpoint between the two eyes as the center, a horizontal distance and a vertical distance as the side length as the main eye area of each face picture, and constructing a nine eye map with nine main eye areas, wherein the length of the main eye area is × , the width is × , the relaxation coefficient =1.2, and the relaxation coefficient =1.

4.

3. The refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism according to claim 1, characterized in that, In step S11, the nine facial photos are obtained by taking pictures of the patient gazing in a specific direction.

4. The refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism according to claim 1, characterized in that, In step S11, the acquisition of the nine facial photos includes the following steps: Video recording equipment was used to capture video of the patient. During the video capture process, the patient's eyes turned sequentially according to the direction of the arrows displayed on the recording equipment. Nine facial photos were extracted from the captured video based on the video frames.

5. A refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism according to claim 1, characterized in that, Step S13 specifically includes: Step S131: Extract the intraocular features of each eye in the nine-eye representation map according to the three independent projections in the Transformer encoder to generate intraocular Query feature value, intraocular Value feature value and intraocular Key feature value for each eye position; Step S132: Multi-head attention is used to extract and interactively compare the intraocular Query feature value, intraocular Value feature value and intraocular Key feature value of each eye position in order to learn the intraocular detail features of each eye position. Step S133: Extract the features of the same corresponding positions of different eye positions in the nine-eye representation map according to the three independent projections in the Transformer encoder, so as to generate inter-eye Query feature value, inter-eye Value feature value and inter-eye Key feature value for different eye positions. Step S134: Multi-head attention is used to extract and combine interocular query feature values, interocular value feature values ​​and interocular key feature values ​​for different eye positions in order to capture the interocular variation features in different gaze directions. Step S135: The intraocular detail features and the interocular variation features are rearranged to form a refined characterization map.

6. The refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism according to claim 1, characterized in that, In step S14, the construction of the strabismus dataset for training optimization includes the following steps: Nine facial photos of multiple patients were obtained, and image preprocessing was performed on the nine facial photos of each patient to construct the corresponding nine-eye bitmap; By having senior ophthalmologists perform fine-grained diagnosis of each patient's nine-eye bitmap, and using each patient's fine-grained diagnosis as a true annotation of the corresponding patient's nine-eye bitmap, a strabismus disease dataset is constructed. The strabismus dataset was divided into training, validation, and test subsets with a split ratio of 6:3:

1.

7. A refined strabismus diagnosis system based on a hierarchical decoupled attention mechanism, characterized in that, include: The image preprocessing module is used to acquire nine facial photos of the patient and perform image preprocessing on the nine facial photos to construct a nine-eye bitmap. The nine facial photos are photos of the patient gazing in a specific direction, which includes the upper left, directly above, upper right, directly left, directly in front, directly right, lower left, directly below, and lower right. The position encoding module is used to map the multiple blocks into corresponding multiple block embeddings after dividing the nine-eye bitmap into multiple blocks of the same size, and to perform position encoding on the multiple block embeddings to form a nine-eye bitmap representation map. The feature extraction and arrangement module is used to sequentially extract intraocular detail features and interocular variation features from the nine-eye representation map using local multi-head attention, and rearrange the intraocular detail features and interocular variation features to form a refined representation map. The pooling and optimization module is used to perform global pooling on the refined representation map to form a global descriptor, and to train and optimize the global descriptor input to the hierarchical classifier using a combined loss function to output a refined diagnostic result. The combined loss function... Includes a direction loss function for directional strabismus recognition. Degree loss function for strabismus degree change recognition They are expressed as follows: ； ； ； In the formula, Indicates the category of directional strabismus. Indicate category The true label, Indicate category The predicted probability of 1 Categories representing changes in strabismus degree, Indicate category The true label, Indicate category The predicted probability of 2.

8. The refined strabismus diagnosis system based on hierarchical decoupling attention mechanism according to claim 7, characterized in that, The feature extraction and arrangement module includes: The first feature extraction unit is used to extract each intraocular feature from the nine-eye representation map based on three independent projections in the Transformer encoder, so as to generate intraocular Query feature value, intraocular Value feature value and intraocular Key feature value for each eye position; The feature comparison learning unit uses multi-head attention to extract and interactively compare the intraocular query feature value, intraocular value value, and intraocular key feature value for each eye position in order to learn the intraocular detailed features of each eye position. The second feature extraction unit is used to extract features from the same corresponding positions of different eye positions in the nine-eye representation map based on the three independent projections in the Transformer encoder, so as to generate inter-eye Query feature values, inter-eye Value feature values ​​and inter-eye Key feature values ​​for different eye positions. The feature combination unit is used to extract and combine interocular query feature values, interocular value feature values ​​and interocular key feature values ​​from different eye positions using multi-head attention, so as to capture interocular variation features in different gaze directions; The feature arrangement unit is used to rearrange the intraocular detail features and the interocular variation features to form a refined characterization map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The processor executes the computer program to implement the refined strabismus diagnosis method based on a hierarchical decoupled attention mechanism as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the refined strabismus diagnosis method based on the hierarchical decoupled attention mechanism as described in any one of claims 1 to 6.

Images