Strabismus type judgment and angle quantization method based on Hirschberg image driving

Abstract

The invention provides a strabismus type determination and angle quantification method based on Hirschberg image driving, and the method comprises the steps: collecting a front face image of a subject, and positioning an eye region to obtain an eye region-of-interest image of the front face image; inputting the eye region-of-interest image into a convolutional neural network model, and judging whether a subject has strabismus possibility based on model output: if yes, determining a contour corresponding to each highlight region through a fixed threshold binarization method and a contour analysis algorithm to determine a cornea reflection point; determining a pupil center and a pupil radius through a low gray threshold segmentation and circularity screening method; and calculating the pixel offset of the cornea reflection point and the pupil center, and combining with a segmented pixel-angle conversion model to obtain the strabismus deflection angles of the two eyes. According to the method, the accuracy, objectivity and interpretability of strabismus evaluation can be improved, so that the method can better serve multiple application scenes such as special diagnosis and treatment, large-scale screening and telemedicine.

Description

Technical Field

[0001] This application relates to the field of ophthalmic medical image analysis and intelligent diagnostic technology, and in particular to a method, device and storage medium for strabismus type determination and angle quantification based on Hirschberg image-driven technology, which can be applied to clinical strabismus screening, telemedicine and primary healthcare institutions. Background Technology

[0002] Currently, the diagnosis of strabismus typically relies on subjective assessment by clinicians, such as the cover test and the Krimsky test. For younger children who cannot cooperate with the examination, the results may be inaccurate. Children's eye muscles are not yet fully developed, and their strabismus presentation may be inaccurate due to poor cooperation. These methods are often limited by patient cooperation, physician experience, and the testing environment. However, many parts of the world lack pediatric ophthalmologists; the United States has only 2.4 pediatric ophthalmologists per 100,000 people, with a training period exceeding 12 years. In developing countries, grassroots institutions lack specialized equipment, resulting in misdiagnosis rates exceeding 30%. Therefore, early detection and accurate assessment of strabismus remain challenging.

[0003] With the development of computer vision and deep learning technologies, image-based automatic strabismus detection and evaluation methods have gradually emerged. Some studies use facial or eye images taken by ordinary cameras or smartphones, and use image processing algorithms to detect corneal reflection points and pupil positions, and estimate strabismus angles by combining the Hirschberg principle; other studies introduce convolutional neural networks to perform binary classification of face or eye images, and even attempt to distinguish different strabismus directions. However, existing image-based technologies still generally have the following shortcomings: (1) There is a limited amount of clinical image data with accurate strabismus type and strabismus degree annotations, and the distribution of samples of different types and different angle ranges is uneven, resulting in insufficient generalization ability of the model in multi-type and multi-angle scenarios. (2) Existing methods mostly rely on the simple geometric relationship between the corneal reflection point and the pupil center in two-dimensional images and the fixed Hirschberg constant, which makes it difficult to balance the quantitative accuracy of small-angle and large-angle strabismus, and is prone to systematic errors when the viewing distance changes or the deviation angle is large. (3) Many deep learning methods only stop at the level of coarse screening of "whether or not strabismus exists". They have limited support for automatic identification of various types of strabismus such as esotropia, exotropia, hypertropia and hypotropia. They mostly use coarse classification rather than fine quantification for strabismus deviation angle. (4) Some methods have complex network structures and opaque internal mechanisms. They lack interpretable mapping between key geometric features and clinical indicators, making it difficult to gain the full trust of clinicians.

[0004] In summary, existing solutions for strabismus detection and strabismus degree quantification based on image technology still have significant shortcomings in areas such as automatic strabismus screening and multi-type recognition, high-precision quantitative assessment of deviation angle, and deployability in resource-constrained environments. Summary of the Invention

[0005] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a method, device and storage medium based on Hirschberg image-driven strabismus type determination and angle quantification to improve the accuracy, objectivity and interpretability of strabismus assessment, thereby better serving the efficiency of various application scenarios such as specialist diagnosis and treatment, large-scale screening and telemedicine.

[0006] In a first aspect, the present invention provides a method for strabismus type determination and angle quantification based on Hirschberg image-driven methods, the method comprising:

[0007] Acquire the first frontal facial image of the first subject, locate the eye region to obtain the first region of interest image of the eye in the first frontal facial image;

[0008] The image of the region of interest of the first eye is input into a deep convolutional neural network model with an integrated attention mechanism to determine whether the first subject is suspected of having strabismus:

[0009] If strabismus is suspected, then:

[0010] The highlighted regions of the first eye region of interest image were determined by a fixed threshold binarization method.

[0011] The contours of each highlighted area are determined by a contour analysis algorithm to identify the first corneal reflection point; the contour analysis algorithm includes the roundness of the corresponding contour and the average gray value of the corresponding contour.

[0012] Within the eyelid mask area, the pupil region is detected by low grayscale threshold segmentation and circularity screening methods to determine the center and radius of the first pupil.

[0013] The pixel offsets between the first corneal reflection point and the pupil center in the horizontal and vertical directions are calculated. Combined with the segmented pixel-angle conversion model, the strabismus angles of both eyes in the horizontal and vertical directions are obtained.

[0014] In a second aspect, the present invention also provides an apparatus comprising one or more processors and a memory, wherein the memory contains instructions executable by the one or more processors to cause the one or more processors to perform a strabismus type determination and angle quantization method based on Hirschberg image-driven according to various embodiments of the present invention.

[0015] Thirdly, the present invention also provides a storage medium storing a computer program that enables a computer to execute a strabismus type determination and angle quantification method based on Hirschberg image driving according to various embodiments of the present invention. Attached Figure Description

[0016] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0017] Figure 1 A flowchart of a strabismus type determination and angle quantization method based on Hirschberg image-driven method provided in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of facial image key point annotation provided in an embodiment of the present invention;

[0019] Figure 3 This is an overview diagram of a CBAM attention mechanism provided in an embodiment of the present invention;

[0020] Figure 4 A ResNet18 network architecture integrating CBAM is provided as an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of a corneal reflective spot detection and optimal contour selection process according to an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram of a pupil detection and optimal contour selection process provided in an embodiment of the present invention;

[0023] Figure 7 A visual representation of corneal reflectance displacement-strabismus direction provided in an embodiment of the present invention;

[0024] Figure 8 An age and gender distribution chart of subjects provided in one embodiment of the present invention;

[0025] Figure 9 This is a comparison chart of the accuracy of a binary classification training set and a validation set provided in one embodiment of the present invention;

[0026] Figure 10 ROC curves for a test set and a validation set are provided in one embodiment of the present invention;

[0027] Figure 11 An automated program for reporting oblique angles is provided as an embodiment of the present invention;

[0028] Figure 12 A consistency analysis chart of clinical and automated measurement results is provided as an embodiment of the present invention;

[0029] Figure 13 This invention provides a correlation analysis between automated measurement results and synoptic results, as provided in one embodiment of the invention.

[0030] Figure 14 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0031] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0032] It should be noted that the strabismus type determination and angle quantization method based on Hirschberg image driving provided in this application does not limit the executing entity. Optionally, the executing entity of the strabismus type determination and angle quantization method based on Hirschberg image driving provided in this application can be a strabismus type determination and angle quantization system based on Hirschberg image driving. Specifically, the executing entity of the strabismus type determination and angle quantization method based on Hirschberg image driving provided in this application can be the strabismus type determination and angle quantization unit based on Hirschberg image driving within the strabismus type determination and angle quantization system based on Hirschberg image driving.

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] Figure 1 This is a flowchart illustrating a method for strabismus type determination and angle quantization based on Hirschberg image-driven approaches, as provided in an embodiment of the present invention. Figure 1 As shown, in this embodiment, the present invention provides a method for strabismus type determination and angle quantization based on Hirschberg image-driven methods, the method comprising:

[0035] S11: Acquire the first frontal facial image of the first subject, locate the eye region to obtain the first region of interest image of the eye in the first frontal facial image;

[0036] S12: Input the image of the region of interest of the first eye into a deep convolutional neural network model with an integrated attention mechanism, and determine whether the first subject has strabismus based on the output of the network model:

[0037] If strabismus is possible, then:

[0038] S131: Determine each bright region of the first eye region of interest image using a fixed threshold binarization method;

[0039] S132: The contour of each bright area is determined by the contour analysis algorithm to determine the first corneal reflection point; wherein, the contour analysis algorithm includes the roundness of the corresponding contour and the average gray value of the corresponding contour.

[0040] S133: Within the eyelid mask, the pupil region is detected by low grayscale threshold segmentation and circularity screening to determine the center and radius of the first pupil.

[0041] S134: Calculate the pixel offset between the first corneal reflection point and the pupil center in the horizontal and vertical directions, and combine it with the segmented pixel-angle conversion model to obtain the strabismus angle of both eyes in the horizontal and vertical directions.

[0042] Specifically, in step S11:

[0043] (1) Subject recruitment

[0044] With informed consent from the subjects or their guardians, a total of 832 subjects aged 3 to 62 years were recruited from the ophthalmology department of the hospital, including 392 strabismus patients and 440 normal controls.

[0045] (2) Imaging equipment and imaging distance

[0046] An eye tracker with an infrared light source was used as the image acquisition device. The subject's chin was fixed to a height-adjustable chin rest to keep the head stable. The fixation target was set directly in front of the camera, and the subject was asked to look directly at the fixed light point. The distance between the camera's optical center and the subject's eyes was set to approximately 55 cm, which falls within the conventional eye-tracking measurement range of 35 cm to 1 m, thus minimizing geometric errors while ensuring image resolution.

[0047] (3) Image acquisition and screening

[0048] A full-face image was captured while the subject was looking directly at a target. The image resolution was, for example, 1280×1024 pixels. If the subject was wearing a mask or had significant occlusion, blinking, or other abnormalities, the image was discarded and recaptured. All images were manually reviewed, and severely blurry, overexposed, or underexposed images were removed.

[0049] (4) Gold Standard Measurement and Labelling

[0050] All subjects underwent clinical strabismus assessment by the same experienced pediatric ophthalmologist. The alternating prism cover test (APCT) and synoptophore examination were used to determine the strabismus type (esotropia, exotropia, hypertropia, hypotropia, or orthotropia) and strabismus deviation angle (expressed in ° or prism diopters PD), which served as the gold standard label for model training and evaluation.

[0051] Figure 2This is a schematic diagram of facial image key point annotation provided in an embodiment of the present invention; as shown below. Figure 2 As shown, the acquired RGB full-face image is used as input. A face detection and 68-point facial keypoint localization model based on the Dlib toolkit is used to process each image, resulting in a coordinate set containing 68 feature points. Points 37-42 correspond to the left eye, and points 43-48 correspond to the right eye. Based on the keypoint set for each eye, the minimum bounding rectangle enclosing the eye is calculated and extended outwards by 20 pixels horizontally and vertically to obtain the ROI (Region of Interest) for both eyes. The ROI regions are cropped to form the eye image for subsequent analysis. The cropped image is further grayscaled to reduce the impact of color differences on the algorithm. Using the keypoint set located at the upper and lower eyelid edges, the Catmull-Rom spline interpolation algorithm is used to generate a smooth eyelid contour curve, forming a closed eyelid contour region. This eyelid contour region is used to construct the eyelid mask for pupil detection, ensuring that the pupil search range is limited to the inside of the eyelid.

[0052] In step S12, ResNet18-CBAM is taken as an example of a deep convolutional neural network model with integrated attention mechanism;

[0053] (1) Data augmentation and dataset partitioning

[0054] Using the cropped eye images as samples, the following data augmentation operations were performed on each image: horizontal mirroring, adding random noise, applying Gaussian blur, and random cropping, increasing the dataset size by 4 times. The augmented images were uniformly scaled or padded to 224×224 pixels and divided into training, validation, and test sets in an 8:1:1 ratio.

[0055] (2) Network Structure

[0056] The ResNet-18 network is used as the basic network structure. The first layer is a convolutional layer that performs a 7×7 convolution operation on the input 224×224×3 RGB image, followed by downsampling with a stride of 2 to obtain a 112×112×64 feature map. This is then further downsampled to 56×56×64 by a 3×3 max-pooling layer. The main body of the network consists of four stages, each containing two residual blocks: the first stage uses 64 3×3 convolutional kernels, the second stage uses 128, the third stage uses 256, and the fourth stage uses 512. All convolutional layers are followed by batch normalization and ReLU activation functions. Residual connections (skip connections) are introduced into each convolutional block, which perform an addition operation between the input and the output of the convolutional layer to ensure smooth gradient propagation throughout the network. When the dimensions of the input and output feature maps are inconsistent, spatial downsampling is used to achieve dimension matching; if the dimensions are consistent, 1×1 convolution is used for channel alignment. After each convolutional block, residual connections sum the input and output to form an identity mapping, thereby enhancing gradient propagation. Global average pooling is performed after the convolutional stage to compress the spatial dimension to 1×1. A fully connected layer is placed at the end, with the number of neurons matching the number of classes in this task (2 classes in total), to output the final classification score for each class. ResNet-18 contains a total of 18 convolutional layers distributed across the aforementioned blocks, supplemented by one average pooling layer and one fully connected layer. The introduction of skip connections ensures smooth gradient propagation throughout the network, making this architecture particularly suitable for deeper network designs. Batch normalization effectively stabilizes and accelerates the model training process. On top of the original ResNet-18 network architecture, a Convolutional Attention Module (CBAM) is integrated into each residual block to enhance the model's feature representation capabilities. This module comprises two sub-modules: spatial attention and channel attention. The spatial attention module evaluates the importance of different spatial locations in the feature map. The process involves calculating the average and maximum values ​​of the feature map along each channel dimension; concatenating the two features and generating a spatial attention map using a 1×1 convolutional layer; and then applying the Sigmoid function to normalize the weights to between 0 and 1 to characterize the importance of each location. The channel attention module evaluates the importance of each feature channel. The process involves performing global average pooling and global max pooling on the feature map to obtain two channel description vectors; inputting these two vectors into a shared multilayer perceptron (MLP), which includes dimensionality reduction and restoration operations; summing the MLP outputs and then applying the Sigmoid function to generate the channel weights.In the residual block, the CBAM module is integrated in the following order: First, after the output of the two residual blocks in each stage and before the addition operation with the shortcut connection, the channel attention module is introduced to adaptively weight the feature channels; then, the spatial attention module is applied to recalibrate the importance of the spatial position of the feature map; finally, the feature map weighted by the dual attention mechanism is added with the shortcut connection feature to complete the residual connection process.

[0057] (3) Training and Reasoning

[0058] The enhanced eye image is input into the ResNet18-CBAM network and trained using the cross-entropy loss function and stochastic gradient descent optimizer. Accuracy and loss are monitored on the validation set during training to prevent overfitting. After training, a model for binary classification is obtained. During the inference phase, the subject's eye image is input into the model, which outputs the probability values ​​of belonging to either "normal" or "possibly strabismus." If it is determined to be "possibly strabismus," steps S131–S134 are executed.

[0059] In step S131, the automatic detection of corneal reflective points and pupil center, as well as the quantization of oblique angles, are algorithmically designed. Firstly, based on the high reflectivity of corneal reflective points, a fixed threshold binarization method is used for preliminary screening. A threshold of 200 is set for the grayscale image of the eye region for binarization. This threshold is an optimized value determined through statistical analysis of a large amount of clinical data, effectively distinguishing the grayscale differences between corneal reflective points and surrounding tissue structures. After binarization, the bright areas in the image are highlighted.

[0060] Figure 5 One of the white particles in c is light reflected from the cornea. In step S132, a contour analysis algorithm is used to extract all connected regions in the binary image. This algorithm can accurately identify boundary contours in the image and establish hierarchical relationships between contours. Based on this, the detected contours are filtered by area, retaining contours with an area in the range of 5 to 20 pixels. This filtering condition is designed based on the physiological characteristics of corneal reflection points: contours with too small an area may be image noise or tiny reflection artifacts, while contours with too large an area may be other non-specific reflective areas. Through this step, most interfering factors are effectively eliminated, improving the accuracy of subsequent processing.

[0061] In the feature extraction stage, the algorithm employs a multi-feature fusion strategy to comprehensively evaluate each contour. First, the circularity feature of each contour is calculated. Circularity is a geometric index characterizing how closely a contour approximates an ideal circle, and its calculation method is as follows:

[0062]

[0063] Where A represents the contour area and P represents the contour perimeter. A perfect circle has a circularity of 1, while irregular shapes have a circularity close to 0. Corneal reflective points, due to their optical reflective properties on the corneal surface, typically exhibit an approximately circular shape; therefore, circularity is an important indicator for identifying true reflective points. Simultaneously, the algorithm also calculates the average grayscale value within each contour, using the following method:

[0064]

[0065] Where I(x) i ,y i ) is the pixel (x) i ,y i The grayscale value of the contour is N, where N is the total number of pixels within the contour. Corneal reflective points, acting as specular reflections of the light source on the corneal surface, are typically the brightest areas in the image, with average grayscale values ​​significantly higher than surrounding tissues. To comprehensively evaluate these two features, a linear weighted scoring function was established. The weight allocation was determined based on clinical experience and preliminary experiments, with roundness weight α set to 0.6 and brightness weight β set to 0.4. The comprehensive score was calculated as follows:

[0066]

[0067] A maximization scoring strategy is employed to select the optimal corneal reflective point from all valid contours. This strategy compares the comprehensive scores of each contour and selects the contour with the highest score as the final corneal reflective point. This method ensures accurate identification of the region that best matches the characteristics of a corneal reflective point, even when multiple candidate contours exist under complex lighting conditions.

[0068] like Figure 5 As shown in d, the minimum circumcircle is calculated for the selected optimal contour, obtaining sub-pixel level center coordinates and radius information. The effect is checked by obtaining the boundary coordinates of the minimum circumcircle contour and redrawing it on the original image version. The final result is shown in d. Figure 5 As shown in e.

[0069] In step S133, based on the low grayscale characteristics of the pupil under infrared illumination, a multi-stage image processing algorithm is used to accurately locate the pupil center. The pupil region typically appears as the darkest circular structure within the eye region in near-infrared images; this optical characteristic provides a reliable physical basis for pupil detection. A smooth curve generated between feature points using the Catmull-Rom spline interpolation algorithm provides a precise region of interest for subsequent pupil segmentation. First, a binary mask is generated based on the smooth eyelid contour, as shown below. Figure 6As shown in b, the analysis scope is strictly limited to the area within the eyelid contour. Subsequently, a fixed threshold binarization process is applied to the eye region image within the mask. The threshold of 55 is the optimal value determined based on extensive experimental data, effectively distinguishing the pupil (low grayscale value) and iris (higher grayscale value) regions. A reverse binarization strategy is used to render the pupil region white and other regions black. In the contour detection stage, connected component analysis is performed on the binary image, such as... Figure 6 As shown in Figure c, all possible candidate regions (multiple circular contours exist) are identified. Considering that multiple candidate regions may exist in a real image (such as interference from external structures like eyelashes and eyelids, image noise, etc.), the algorithm employs a two-stage filtering mechanism: first, contours that are too small (area > 15 pixels) are excluded based on an area threshold, which is set based on the physiological size of the pupil and the image resolution; then, the contour closest to a circle is selected as the candidate pupil based on the circularity feature. This dual guarantee mechanism ensures accurate identification of the pupil region even in complex situations such as eyelash occlusion. The minimum circumcircle of the optimal contour is calculated to obtain... Figure 6 As shown in d, the pupil center coordinates and radius were obtained with sub-pixel precision. The image was then redrawn on the original version for effect detection, and the final result is shown below. Figure 6 As shown in e.

[0070] Figure 7 A visualization of corneal reflectance displacement-strabismus direction is provided in an embodiment of the present invention, and the probability distribution of the direction of each type of strabismus is as follows: Figure 7 As shown.

[0071] In step S134, with Figure 7 The parameter explanations shown use the right eye as an example;

[0072] R1, R2, R3, R4, R5, and R6 represent the horizontal position information of the pupil coordinate system x-axis in the right-eye input image; R′1, R′2, R′3, R′4, R′5, and R′6 represent the vertical position information of the pupil coordinate system y-axis in the right-eye input image. RX r With RY r These represent the horizontal and vertical coordinates of the corneal reflection point of the right eye in the input image, respectively; PX r With PY r These correspond to the horizontal and vertical positions of the right pupil center in the image, respectively; the pupil radius is r, r = R1 = R4 = R′1 = R′4, 2r = R2 = R5 = R′2 = R′5, 3r = R3 = R6 = R′3 = R′6. RX l With RY l These represent the horizontal and vertical coordinates of the reflected light spot from the left cornea in the input image, respectively; PX l With PY lThese correspond to the horizontal and vertical positions of the left eye pupil center in the image, respectively. Other relevant parameters for the left eye are as follows: Figure 7 As shown.

[0073] The core of the algorithm lies in its ability to transform pixel-level information from eye images into clinically meaningful strabismus parameters. This technology is based on the Hirschberg principle, which states that when the eyeball deviates, the corneal reflection point shifts systematically relative to the pupil center. By precisely quantifying this shift, the algorithm can simultaneously assess the degree of strabismus in both the horizontal and vertical directions. The formulas for calculating the horizontal and vertical offsets for the right and left eyes are shown below:

[0074] D Xr =RX r -PX r

[0075] D Yr =RY r -PY r

[0076] D Xl =RX l -PX l

[0077] D Yl =RY l -PY l

[0078] Among them, D Xr and D Yr D represents the pixel offset in the horizontal and vertical directions of the right eye, respectively. Xl and D Yl These represent the pixel offsets in the horizontal and vertical directions for the left eye, respectively. To eliminate the influence of individual differences in pupil size, the algorithm uses relative distance for standardization, calculated as follows:

[0079]

[0080] The angle calculation model uses a piecewise function to more accurately simulate the characteristics in actual clinical measurements. Based on the systematic calibration of corneal reflexes of normal subjects using photographic methods, it is pointed out that the conversion factor for the reflex displacement should be 21PD / mm on the emmetropic plane, which is converted to an angle of approximately 12° / mm. When the reflex point is located within the pupil area (|relative distance|≤1), the angle and distance are linearly related, with each unit of relative distance corresponding to a strabismus angle of 42PD, which is converted to an angle of approximately 24°. When the reflex point is located between the pupil edge and the corneal limbus (12), the algorithm will further adjust the calculation parameters to adapt to the measurement requirements of large-angle strabismus.

[0081] The calculation methods for the strabismus angle θ of both eyes in the horizontal and vertical directions are as follows:

[0082]

[0083]

[0084] The prism power Δ of both eyes in the horizontal and vertical directions is calculated as follows:

[0085]

[0086] This segmented design enables the algorithm to maintain good accuracy and sensitivity across the entire possible measurement range.

[0087] Based on the monocular analysis, the algorithm further performs binocular coordinated strabismus angle analysis. The horizontal angle difference ΔH and vertical angle difference ΔV between the two eyes are calculated to assess the binocular deviation angle, conforming to the clinical criteria for strabismus angle calculation.

[0088] The calculation methods for ΔH and ΔV are as follows:

[0089]

[0090] The core contributions of the above embodiments are mainly reflected in several aspects, including integrated processing of strabismus screening and degree quantification, highly robust automatic detection of key ocular features, and a high-precision angle conversion model under clinical calibration. These breakthroughs overcome the limitations of traditional strabismus assessment methods that rely on subjective human judgment and simplified geometric assumptions, providing new technical paths and implementation ideas for research on intelligent strabismus diagnosis methods and grassroots screening applications. First, an integrated automatic diagnostic process was constructed, from "whether strabismus exists" to "strabismus type + strabismus angle." Within the same system, normal / strabismus binary classification, multi-type strabismus discrimination, and quantitative calculation of binocular deviation angles and strabismus degrees are completed sequentially. This significantly improves the precision of diagnosis compared to existing schemes that can only perform rough screening or only measure horizontal deviation. Second, a corneal reflective point detection algorithm combining fixed threshold segmentation, contour analysis, roundness and brightness weighted scoring, and minimum circumcircle fitting was proposed. Combined with a pupil center localization method using eyelid masking and low grayscale threshold segmentation, this algorithm can stably and automatically extract key structures even under complex lighting conditions and individual differences, significantly reducing reliance on manual annotation and the risk of error propagation. Furthermore, based on the use of deep learning networks to improve the accuracy of strabismus recognition, a clinically calibrated segmented pixel-angle conversion model is introduced. Differential conversion coefficients are applied to different offset intervals to achieve a high-precision mapping from two-dimensional image pixel deviations to commonly used clinical prism powers and angles, effectively reducing the systematic bias caused by the traditional Hirschberg linear assumption. The above embodiments achieve breakthroughs in automatic strabismus screening and multi-type identification, and high-precision quantitative assessment of strabismus deviation angles. They provide a new methodological foundation for applications such as specialized clinical diagnosis, large-scale screening at the grassroots level, and telemedicine services, making strabismus objective assessment and intelligent diagnosis research not only more scientifically rigorous but also possessing broader application potential.

[0091] The determination of strabismus type is based on a systematic analysis of the horizontal and vertical deviations of both eyes.

[0092] In a preferred embodiment, the algorithm first compares the absolute values ​​of the monocular horizontal offsets |D Xl |、|D Xr |and the absolute value of the vertical offset of a single eye|D Yl |、|D Yr Based on the relationship between the magnitudes of |, the eye with the largest absolute value of the offset is identified as the deviating eye. Furthermore, according to ΔH and θ(relative_D) Yl ), θ(relative_D Yr The values ​​of ΔH and ΔH are used for specific classification: For horizontal strabismus, ΔH > 0 indicates exotropia, and ΔH < 0 indicates esotropia. For vertical strabismus, the value of |θ(relative_D) is used to determine the specific type of strabismus. Yl Is |greater than |θ(relative_D)? Yr)|: Yes, then when θ(relative_D Yl When θ(relative_D)>0, it is hypostropia, and θ(relative_D) Yl When the α value is less than 0, it indicates superior strabismus. This diagnostic mechanism fully considers the symmetry of human anatomy, ensuring the accuracy of the diagnosis.

[0093] More preferably, based on the detected binocular coordinated strabismus angle, the severity of strabismus is divided into four levels: 0-5° for mild strabismus, 5-15° for mild strabismus, 15-30° for moderate strabismus, and >30° for severe strabismus. This grading system provides an important reference for the formulation of clinical treatment plans.

[0094] The above embodiments realize the automation, quantification and standardization of strabismus detection, providing a reliable auxiliary tool for ophthalmological clinical diagnosis, and are particularly suitable for large-scale screening and long-term follow-up monitoring scenarios.

[0095] More preferably, in the output report results, the pupil center, corneal reflection point coordinates, strabismus deviation angles, horizontal angle difference, and vertical angle difference of both eyes are marked on the original eye image with different colors or symbols, and the offset vector from the pupil center to the reflection point is plotted. At the same time, the calculated strabismus type, horizontal / vertical deviation angle, and corresponding prism power are displayed in the image / report.

[0096] The algorithm output includes a rich set of parameters, encompassing not only basic strabismus angle and type determination, but also detailed information such as prism diopter conversion, offset pixel values, and relative distance. The algorithm also features result visualization capabilities; in the above embodiments, key feature points and measurement results can be intuitively annotated on the original image, facilitating verification and interpretation by clinicians.

[0097] The performance of the binary classification model and the angle quantification model was evaluated using data from the aforementioned 832 subjects. Results showed that the ResNet18-CBAM-based binary classification model achieved an overall accuracy of approximately 94.69% on the validation set, with recall and precision both around 0.94. In 648 strabismus images, the accuracy rates for ROI detection (left and right eyes) were 100%, pupil center detection was approximately 97%, and corneal reflective spot detection was approximately 95%. Correlation and consistency analyses of the strabismus angles measured in this application with clinical APCT and synoptophore measurements showed Pearson correlation coefficients exceeding 0.90 and root mean square errors of approximately 4°, indicating that the method presented in this application has high consistency with the clinical gold standard in strabismus angle quantification and can be used as a clinical auxiliary assessment tool for large-scale screening and follow-up monitoring.

[0098] Figure 14 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention.

[0099] like Figure 14 As shown, in another aspect, this application also provides a device including one or more central processing units (CPUs) 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage portion 1408 into a random access memory (RAM) 1403. Various programs and data required for device operation are also stored in the RAM 403. The CPU 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0100] The following components are connected to I / O interface 1405: an input section 1406 including a keyboard, mouse, etc.; an output section 1407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN card, modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to I / O interface 1405 as needed. Removable media 1411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1410 as needed so that computer programs read from them can be installed into storage section 1408 as needed.

[0101] In particular, according to embodiments of this disclosure, the methods described in any of the above embodiments can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the methods. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1409, and / or installed from removable medium 1411.

[0102] In another aspect, this application also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus of the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this application.

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0104] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor; for example, each unit can be a software program located in a computer or mobile smart device, or a separately configured hardware device. The names of these units or modules do not, in some cases, constitute a limitation on the unit or module itself.

[0105] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for determining strabismus type and quantifying angle based on Hirschberg image-driven methods, characterized in that, include: A first frontal facial image of the first subject is acquired, and the eye region is located to obtain a first region of interest image of the eye in the first frontal facial image; The image of the first region of interest of the eye is input into a deep convolutional neural network model with an integrated attention mechanism. Based on the output of the network model, it is determined whether the first subject may have strabismus. If strabismus is possible, then: The highlighted regions of the first region of interest image are determined by a fixed threshold binarization method; The contours of each highlighted region are determined by a contour analysis algorithm to identify the first corneal reflection point; wherein, the contour analysis algorithm includes the roundness of the corresponding contour and the average gray value of the corresponding contour. The pupil region was detected within the eyelid mask using low grayscale threshold segmentation and circularity filtering methods to determine the center and radius of the first pupil. Calculate the pixel offset between the first corneal reflection point and the pupil center in the horizontal and vertical directions, and combine it with the segmented pixel-angle conversion model to obtain the strabismus angle of both eyes in the horizontal and vertical directions.

2. The method according to claim 1, characterized in that, The process of acquiring a first frontal facial image of the first subject and locating the eye region to obtain a first region of interest (ROI) image of the first frontal facial image includes: The first frontal facial image of the first subject was acquired using a camera device with an infrared light source at a preset distance of 55cm. The image is converted to grayscale, its size is normalized, and its quality is filtered. The eye region is located using a facial landmark detection method, and the image is cropped to obtain the first eye region of interest image of the first frontal facial image.

3. The method according to claim 1, characterized in that, The contour analysis algorithm further includes weights for the roundness and the average gray value. Determining the corresponding contour of each highlighted region using the contour analysis algorithm to determine the first corneal reflection point includes: The first corneal reflection point is determined by the circularity, the average gray value, and the weight, and the center coordinates of the first corneal reflection point are obtained by fitting the minimum circumcircle. The step of detecting the pupil region within the eyelid mask area using low grayscale threshold segmentation and circularity filtering methods to determine the center and radius of the first pupil includes: A binary mask is generated based on the smooth eyelid contour, and a fixed threshold binarization process is applied to the eye region image within the binary mask to distinguish the pupil region from the iris region. The pupil area is rendered white and other areas are rendered black using a reverse binarization method. When multiple circular contours exist in the pupil region, the circular contour with the highest circularity is selected as the first optimal contour by the circularity filtering method, and the minimum circumcircle of the first optimal contour is calculated to determine the center and radius of the first pupil.

4. The method according to claim 1, characterized in that, The calculation of the pixel offset between the first corneal reflection point and the pupil center in the horizontal and vertical directions includes: The horizontal offset, vertical offset, relative horizontal offset, and relative vertical offset of both eyes are calculated as follows: D Xr =RX r -PX r D Yr =RY r -PY r D Xl =RX l -PX l D Yl =RY l -PY l Among them, RX r With RY r PX represents the horizontal and vertical coordinates of the corneal reflector point of the right eye. r With PY r D represents the horizontal and vertical coordinates of the pupil center; Xr and D Yr RX represents the horizontal and vertical pixel offsets between the right corneal reflection point and the center of the pupil; l With RY l PX represents the horizontal and vertical coordinates of the reflected light spot from the left cornea. l With PY l Let D be the horizontal and vertical coordinates of the center of the left pupil. Xl and D Yl denoted by , where r is the horizontal and vertical pixel offset of the left corneal reflection point from the center of the pupil; and r is the pupil radius, relative_D. Xr relative_D represents the relative horizontal offset between the corneal reflector point of the right eye and the center of the pupil. Yr relative_D represents the relative vertical offset between the corneal reflector point of the right eye and the center of the pupil. Xl relative_D represents the relative horizontal offset between the corneal reflector point of the left eye and the center of the pupil. Yl This represents the relative vertical offset between the left corneal reflector point and the center of the pupil. The combined piecewise pixel-angle conversion model yields the following strabismus angles in both the horizontal and vertical directions: When the corneal reflex point is located within the pupil area (|relative distance|≤1), the angle and distance are linearly related. Each unit of relative distance corresponds to 42PD of strabismus angle, which is converted to an angle of approximately 24°. The calculation methods for the strabismus angle θ of both eyes in the horizontal and vertical directions are as follows: The prism power Δ of both eyes in the horizontal and vertical directions is calculated as follows:

5. The method according to claim 4, characterized in that, After obtaining the strabismus angles of both eyes in the horizontal and vertical directions, the method further includes: Calculate the horizontal angle difference ΔH and the vertical angle difference ΔV between the two eyes to assess the binocular deviation angle; The calculation methods for ΔH and ΔV are as follows:

6. The method according to claim 5, characterized in that, After obtaining the strabismus angles of both eyes in the horizontal and vertical directions, the method further includes: Compare the absolute values ​​of monocular horizontal offset |D Xl |、|D Xr |and the absolute value of the vertical offset of a single eye|D Yl |、|D Yr The size relationship of the two eyes indicates that the eye with the largest absolute value of the offset is the deviated eye. For determining horizontal strabismus: ΔH>0 indicates exotropia, and ΔH<0 indicates esotropia. For determining vertical strabismus: Determine |θ(relative_D) Yl Is |greater than |θ(relative_D)? Yr )|: Yes, then when θ(relative_D Yl When θ(relative_D)>0, it is hypostropia, and θ(relative_D) Yl When ) < 0, it is called upward strabismus.

7. The method according to claim 6, characterized in that, Also includes: If the angle of deviation between the two eyes is 0-5°, it is considered mild strabismus. If the angle of deviation between the two eyes is 5-15°, it is considered mild strabismus. If the angle of deviation between the two eyes is 15-30°, it is considered moderate strabismus; If the angle of deviation between the two eyes is greater than 30°, it is considered severe strabismus.

8. The method according to claim 7, characterized in that, The coordinates of the pupil center and the corneal reflection point are superimposed on the first frontal facial image, along with the strabismus angles of the two eyes in the horizontal and vertical directions, the horizontal angle difference, and the vertical angle difference, generating a structured diagnostic report that includes the presence or absence of strabismus, the type of strabismus, and the strabismus angle.

9. A strabismus type determination and angle quantification device based on Hirschberg image-driven method, characterized in that, The device includes: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1-8.

10. A storage medium storing a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images