Vision health monitoring method based on shooting, tracking and comparison of mobile terminal equipment

Abstract

The invention relates to the technical field of computer vision, in particular to a vision health monitoring method based on shooting, tracking and comparison of mobile terminal equipment. Comprising the following steps: respectively collecting a large number of left-eye, right-eye and double-eye photos of teenagers, carrying out eye cutting and preprocessing by using a newest target detection network yolov10 in combination with opencv, and carrying out uniform resize on the images; a traditional CNN (convolutional neural network) and a Vision Transformers convolutional neural network are adopted to extract eye features in the preprocessed image, and binary classification supervised learning is used to help distinguish myopic eyes and normal eyes; training the model by using a cross validation method; various devices are used for screening. According to the invention, by developing a computer vision algorithm with high accuracy, whether the eyes are shortsighted or not and the shortsightedness degree can be judged according to the eye images; and mobile terminal deployment of the algorithm is realized, so that real-time detection can be carried out on mobile equipment such as a smart phone.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a vision health monitoring method based on shooting, tracking and comparison by a mobile terminal device. Background Art

[0002] With the growing problem of myopia among adolescents, rapid and accurate myopia detection methods are becoming increasingly important. Traditional detection methods rely on medical equipment, which is costly and complex to operate. In recent years, with the development of computer vision technology, image-based myopia detection has become a research hotspot. However, existing technologies still have shortcomings in image preprocessing, feature extraction, and model building, resulting in limited detection accuracy and generalization capabilities. Furthermore, directly applying Vision Tolerance (Vision Tolerance) to myopia degree segmentation still faces challenges, including dataset size limitations and insufficient model optimization for this task.

[0003] This technical solution, inspired by papers such as "Application of Deep Learning-Based Eye Image Analysis in Myopia Detection" and "Recent Advances in Vision Transformers in Medical Image Classification," aims to fill the gap in Vision Transformers (Vision Transformers) applications in myopia detection and promote the development and application of related technologies. To this end, we propose a vision health monitoring method based on tracking and comparison of images captured by mobile devices. Summary of the Invention

[0004] The purpose of the present invention is to provide a vision health monitoring method based on shooting, tracking and comparison by a mobile terminal device to solve the problems raised in the above background technology.

[0005] To solve the above technical problems, the present invention provides a method for monitoring vision health based on shooting, tracking and comparison by a mobile terminal device, comprising the following steps:

[0006] S1. Data collection and image preprocessing: We collected a large number of photos of the left eye, right eye, and both eyes of teenagers. We used the latest object detection network YOLOv10 combined with OpenCV to perform eye cropping and preprocessing, and uniformly resized the images to improve image quality and consistency.

[0007] S2. Model Construction: We use traditional convolutional neural networks (CNNs) and Vision Transformers convolutional neural networks to extract eye features from pre-processed images and use binary supervised learning to help distinguish myopic eyes from normal eyes.

[0008] S3. Model training and evaluation: Use cross-validation to train the model to avoid overfitting and improve the model's generalization ability. During the training process, adjust model parameters based on the performance of the validation set to optimize model performance. Collect prediction results from the training process and evaluate the model's classification performance using multiple parameters.

[0009] S4. Multi-device screening application: The algorithm is embedded in web applications and multi-terminal mobile applications, which can perform vision screening anytime and anywhere, and support external camera devices to improve image quality.

[0010] As a further improvement of this technical solution, in step S1, the core idea of ​​yolov10 is to divide the input image into multiple grid cells, each grid cell is responsible for predicting a certain number of bounding boxes and the category probability of the targets in these bounding boxes; it consists of a feature extraction layer and a prediction layer; the feature extraction layer contains multiple convolution layers and pooling layers for extracting image features; the convolution layer performs convolution operations by sliding the convolution kernel on the image; the pooling layer is used to reduce the amount of data; the last few layers of the network are prediction layers, which predict the position of the bounding box and the category of the target based on the previously extracted features; for each grid cell, it predicts the coordinates of multiple bounding boxes ,in and are the coordinates of the bounding box center relative to the grid cell, and is the width and height of the bounding box. At the same time, the probability of the target belonging to different categories in each bounding box is also predicted ,in Indicates category;

[0011] Yolov10 combined with opencv for eye cropping specifically includes the following:

[0012] S1.1. Haar feature difference calculation assistance: During the eye cropping process, Haar feature difference calculation is used to help detect edges in the image;

[0013] S1.2. Cropping based on yolov10 prediction results:

[0014] First, the image is fed into the yolov10 network for object detection; the network outputs the predicted bounding box coordinates for each grid cell. and class probabilities ;

[0015] Then, the bounding boxes of the eye category are filtered out, and the image is cropped according to the coordinate information of these bounding boxes.

[0016] As a further improvement of this technical solution, in step S1.1, Haar feature difference calculation is used to help detect edges in the image. The specific algorithm is:

[0017] ;

[0018] in, Indicated in coordinates The value obtained by calculating the Haar feature difference is used to detect the edge in the image to help determine the boundary of the eye; It is used to traverse the area and Coordinate variables of and Represent two different areas in the image and are used to calculate the difference to highlight the edge features; It is a function of the pixel value of the image, which represents the pixel value of the image at a certain coordinate;

[0019] Different regions may contribute differently to feature differences. Introducing a weight factor can better reflect this difference. The optimized formula is:

[0020] ;

[0021] in, and They are regions and The weight factor can be obtained based on prior knowledge, experience or through learning.

[0022] Calculating Haar feature differences at different scales can capture features at different levels of the image. On the basis of introducing weight factors, combined with multi-scale analysis, the robustness and accuracy of feature difference calculation can be improved. The optimized formula is:

[0023] ;

[0024] in, Represents different scales, and In scale The area below, and is the corresponding weight factor, It is on scale Image pixel value function under ;

[0025] The features of a pixel in an image are often related to the features of its neighboring pixels. When calculating Haar feature differences, considering neighborhood information can improve the accuracy and stability of the calculation; the optimized formula is:

[0026] ;

[0027] in, Is a function that represents pixel points The influence of neighborhood information on its eigenvalues.

[0028] As a further improvement of the present technical solution, in step S1, the preprocessing of the cropped eye image includes grayscale conversion, noise reduction, image enhancement, and image normalization; wherein the image noise reduction includes mean filtering, median filtering, and Gaussian filtering; the image enhancement includes contrast enhancement (histogram equalization and grayscale stretching) and sharpening; and the image normalization includes size normalization and grayscale normalization.

[0029] Among them, grayscale determines the weight according to the sensitivity of the human eye to different colors. The specific algorithm is:

[0030] ;

[0031] in, Indicates the grayscale value of a certain pixel in the image after grayscale processing; They represent the pixel values ​​of the red, green, and blue channels of a certain pixel in a color image; 0.299, 0.587, and 0.114 are the sensitivity ratios of the human eye to red, green, and blue, respectively.

[0032] As a further improvement of this technical solution, the Gaussian filter is used for noise reduction. The specific algorithm is:

[0033] ;

[0034] in, Indicates that after Gaussian filtering, the coordinates The pixel value at ; It is the coordinate variable used to traverse the filter kernel; is the filter kernel The weight value at the coordinate is used to determine the weighting method for surrounding pixels; Indicates the original image at coordinates The pixel value at , that is, the original pixel value participating in the weighted average calculation; Represents the filter kernel used by Gaussian filtering. It is a two-dimensional matrix. The size and weight distribution determine the effect of the filtering.

[0035] Adaptive weights are introduced to optimize the Gaussian filtering algorithm:

[0036] First consider the local variance: calculate the original image in The variance in the local area centered on ;

[0037] Secondly, adjust the weight: adjust the weight of the filter kernel according to the local variance ;

[0038] The final optimized formula, that is, the filtered pixel value becomes:

[0039] ;

[0040] in, is the adjusted weight;

[0041] The Gaussian filtering algorithm is also optimized by combining multi-scale filtering:

[0042] First, multiple Gaussian filter kernels of different scales are used. For each scale of the filter kernel, the filtered pixel value is calculated according to the original formula;

[0043] Then perform weighted fusion: perform weighted fusion based on the results of filtering at different scales. Assume that the pixel values ​​after filtering at different scales are ,in represents the scale, and the weight is . Then the optimized pixel value is:

[0044] ;

[0045] in, .

[0046] As a further improvement of this technical solution, in step S2, a model is constructed by combining a traditional convolutional neural network (CNN) and a Vision Transformers convolutional neural network. A traditional convolutional neural network (CNN) includes convolution operations, pooling operations, and fully connected layers. The working principle of convolution is as follows:

[0047] ;

[0048] in, Is the pixel value of the convolution output, expressed at coordinates The new pixel value obtained after the convolution operation; is the pixel value of the input image, that is, the original eye image at coordinates The pixel value at ; Is the convolution kernel, which is a small matrix used to slide on the image for convolution operation. Its size and element value determine the effect of convolution; is the bias, which is a constant term used to adjust the position of the convolution result; It is an activation function (such as ReLU), which is used to introduce nonlinear factors and increase the expressive power of the model;

[0049] The purpose of the pooling operation is to reduce the amount of data while retaining important feature information; common pooling methods include maximum pooling and average pooling:

[0050] Max pooling is to select the maximum value in a small area as the output;

[0051] Average pooling calculates the average value of pixel values ​​in a small area as output;

[0052] The fully connected layer is used to integrate the feature vectors obtained by the previous convolution and pooling operations and map them to the final output dimension; each of its neurons is connected to all neurons in the previous layer.

[0053] As a further improvement to this technical solution, the core of the Vision Transformers convolutional neural network is the self-attention mechanism; it can calculate the correlation between each position in the image and the rest of the positions, thereby better capturing global information;

[0054] Among them, the basic formula of self-attention is:

[0055] ;

[0056] in, They are the query vector, key vector, and value vector obtained by linearly transforming the input features; is the dimension of the key vector; The function is used to normalize the calculated weights so that their sum is 1.

[0057] The Vision Transformers convolutional neural network also includes a multi-layer perceptron (MLP): After the self-attention mechanism, a multi-layer perceptron (MLP) is usually connected to further process the features; the MLP consists of multiple fully connected layers and is used to perform nonlinear transformations on the features processed by the self-attention mechanism, thereby increasing the expressive power of the model.

[0058] As a further improvement of the present technical solution, in step S2, the extracted eye features include at least three related features of the eyeball structure, retinal structure and eye appearance; wherein:

[0059] Relevant features of the eyeball structure include eyeball shape, eye white pattern, eyeball axial length, and lens shape and position;

[0060] Relevant features of retinal structure include retinal vascularity and retinal thickness;

[0061] Relevant features of ocular appearance include eyelid morphology and interocular distance.

[0062] As a further improvement of the present technical solution, in step S3, the model is trained using a cross-validation method, specifically including:

[0063] S3.1, divide the eye image dataset processed in step S1 into subsets of roughly equal size (usually The value of is 5 or 10, here we assume );

[0064] S3.2, proceed Round training and validation: In each round, one of the subsets is selected as the validation set, and the rest The model is trained using the training set. During the training process, the model learns the relationship between the features of the eye images in the training set and the corresponding labels of myopia or normal eyes.

[0065] S3.3. After each round of training, the model is validated using the validation set of the current round to evaluate the performance of the model on data that was not involved in the training.

[0066] As a further improvement of the present technical solution, in step S3, the parameters used to evaluate the classification performance of the model include at least accuracy, precision, recall, and F1-score, and the F1-score is used as the final evaluation criterion; wherein:

[0067] Accuracy refers to the ratio of the number of samples predicted as positive or negative and correctly predicted to the total number of samples. The calculation formula is:

[0068] ;

[0069] Precision refers to the ratio of the number of samples predicted to be positive and actually positive to the number of samples predicted to be positive. The calculation formula is:

[0070] ;

[0071] It reflects the accuracy of the model's prediction results.

[0072] Recall refers to the ratio of the number of samples predicted to be positive and actually positive to the number of samples that are actually positive. The calculation formula is:

[0073] ;

[0074] It reflects the model's ability to correctly identify positive examples.

[0075] in, Indicates the number of samples that are predicted to be positive and are actually positive. Indicates the number of samples predicted to be positive but actually negative. Indicates the number of samples predicted to be negative but actually positive. Indicates the number of samples that are predicted to be negative and are actually negative; Indicates the number of samples predicted as positive examples; Indicates the number of samples that are actually positive examples;

[0076] F1 score is an evaluation indicator that comprehensively considers precision and recall. The formula is:

[0077] ;

[0078] It provides a more comprehensive evaluation of model performance, balancing the relationship between precision and recall.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] 1. This vision health monitoring method, based on mobile device tracking and comparison, develops a highly accurate computer vision algorithm that can determine myopia and the degree of myopia based on eye images. The algorithm is deployed on mobile devices, enabling real-time detection on smartphones and other mobile devices. A user-friendly interface is also provided, allowing ordinary users to easily use the application for self-assessment of myopia.

[0081] 2. This vision health monitoring method based on tracking and comparison using mobile terminal devices uses eye photos taken with mobile phones and, through deep learning techniques related to computer vision, a classification model based on the Vision Transformers network structure is developed. This model is used to determine whether a person is nearsighted and to distinguish the degree of myopia. Currently, the highest accuracy rate achieved in the binary classification task (determining whether a person is nearsighted) on existing data is 72%. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is an exemplary overall method flow chart of the present invention. DETAILED DESCRIPTION

[0083] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0084] Example 1

[0085] like Figure 1 As shown, this embodiment provides a vision health monitoring method based on shooting, tracking and comparison by a mobile terminal device, including the following steps:

[0086] S1. Data collection and image preprocessing: We collected a large number of photos of the left eye, right eye, and both eyes of teenagers. We used the latest object detection network YOLOv10 combined with OpenCV to perform eye cropping and preprocessing, and uniformly resized the images to improve image quality and consistency.

[0087] Specifically, eye photos are specifically targeted at adolescents. This is because adolescents are at high risk for myopia, and their eye photos are more easily distinguishable from those with normal vision. Collection requirements include collecting photos of the left eye, right eye, and both eyes. During the collection process, ensure that the photos clearly capture eye features. For example, the subject is required to look directly at the camera, maintain even facial lighting, and avoid shadows obscuring the eyes. Furthermore, the subject is instructed to keep their head steady to prevent blurring of the photo due to shaking.

[0088] At the same time, Yolov10 has advanced object detection capabilities and can accurately identify the eye area in the image. OpenCV provides a wealth of image processing functions to help further optimize the cropping process.

[0089] In this step, yolov10 is an advanced target detection network based on a deep convolutional neural network architecture. The core idea is to divide the input image into multiple grid cells, each grid cell is responsible for predicting a certain number of bounding boxes and the category probability of the targets in these bounding boxes; it consists of a feature extraction layer and a prediction layer; the feature extraction layer contains multiple convolution layers and pooling layers for extracting image features; the convolution layer performs convolution operations by sliding the convolution kernel on the image; the pooling layer is used to reduce the amount of data, and the most common ones are maximum pooling and average pooling. For example, maximum pooling selects the maximum value in a small area as the output. The last few layers of the network are prediction layers, which predict the position of the bounding box and the category of the target based on the previously extracted features. For each grid cell, it predicts the coordinates of multiple bounding boxes. ,in and are the coordinates of the bounding box center relative to the grid cell, and is the width and height of the bounding box. At the same time, the probability of the target belonging to different categories in each bounding box is also predicted ,in Indicates category;

[0090] Yolov10 combined with opencv for eye cropping specifically includes the following:

[0091] S1.1. Haar feature difference calculation assistance (OpenCV related): During the eye cropping process, Haar feature difference calculation is used to help detect edges in the image;

[0092] S1.2. Cropping based on yolov10 prediction results:

[0093] First, the image is fed into the yolov10 network for object detection; the network outputs the predicted bounding box coordinates for each grid cell. and class probabilities ;

[0094] Then, we filter out the bounding boxes of the eye category (assuming the category label is "eye") and crop the image based on the coordinate information of these bounding boxes. For example, if the coordinates of a bounding box are , then the cropped eye image area is , that is, the coordinates of the upper left corner of the image are intercepted as , the coordinates of the lower right corner are The rectangular area is taken as the eye image.

[0095] Further optimization is also included: To improve cropping accuracy, the cropped eye image can be processed. For example, the cropped area can be fine-tuned based on the eye's shape characteristics, or any interfering edges can be removed. This may involve algorithms based on morphological operations, such as erosion and dilation (common image processing operations in OpenCV). Erosion removes small objects from an image, while dilation fills small holes. These operations make the cropped eye image cleaner and more accurate, making it more conducive to subsequent image preprocessing and model building.

[0096] Furthermore, in step S1.1, Haar feature difference calculation is used to help detect edges in the image. The specific algorithm is:

[0097] ;

[0098] in, Indicated in coordinates The value obtained by calculating the Haar feature difference is used to detect the edge in the image to help determine the boundary of the eye; It is used to traverse the area and Coordinate variables of and Represent two different areas in the image and are used to calculate the difference to highlight the edge features; It is a function of the pixel value of the image, which represents the pixel value of the image at a certain coordinate; for example Indicated in coordinates The pixel value at ;

[0099] Different regions may contribute differently to feature differences, and introducing weight factors can better reflect these differences. For example, for some key regions or regions with specific characteristics, larger weights can be given to highlight the importance of these regions in feature difference calculation. The optimized formula is:

[0100] ;

[0101] in, and They are regions and The weight factor can be obtained based on prior knowledge, experience, or learning. For example, the average importance of different regions can be determined by analyzing a large number of samples, and the corresponding weight factor can be set.

[0102] Calculating Haar feature differences at different scales can capture features at different levels of the image. For example, at smaller scales, we can focus on the image's details; at larger scales, we can capture the image's overall structure and contours. By introducing weight factors and combining them with multi-scale analysis, we can improve the robustness and accuracy of feature difference calculations. The optimized formula is:

[0103] ;

[0104] in, Represents different scales, and In scale The area below, and is the corresponding weight factor, It is on scale In actual calculation, the original image can be downsampled to different scales first, and then the feature difference is calculated at each scale, and the results of different scales are fused.

[0105] The features of a pixel in an image are often related to the features of its neighboring pixels. Considering this neighborhood information when calculating Haar feature differences can improve accuracy and stability. For example, the features of pixels within a certain range around a central pixel can be weighted averaged and used as the feature value for the central pixel in the feature difference calculation.

[0106] The optimized formula is:

[0107] ;

[0108] in, Is a function that represents pixel points The neighborhood information function can be designed according to specific needs, for example, Gaussian function, mean function, etc.

[0109] The above optimization methods can improve the performance of the Haar feature difference calculation formula, making it more accurate and effective in tasks such as image feature extraction and edge detection. In practical applications, you can choose the appropriate optimization method based on the specific problem and data characteristics, and make appropriate adjustments and improvements.

[0110] Furthermore, in step S1, the cropped eye image is preprocessed including grayscale conversion, noise reduction, image enhancement, and image normalization; wherein image noise reduction includes mean filtering, median filtering, and Gaussian filtering; image enhancement includes contrast enhancement (including histogram equalization and grayscale stretching) and sharpening; and image normalization includes size normalization and grayscale normalization.

[0111] Among them, grayscale determines the weight according to the sensitivity of the human eye to different colors. The specific algorithm is:

[0112] ;

[0113] in, Indicates the grayscale value of a certain pixel in the image after grayscale processing; They represent the pixel values ​​of the red, green, and blue channels of a certain pixel in a color image; 0.299, 0.587, and 0.114 are the sensitivity ratios of the human eye to red, green, and blue, respectively.

[0114] Specifically, grayscale conversion involves converting a color image into a grayscale image. This conversion is based on the fact that the human eye has different sensitivities to different colors, with green being the most sensitive and blue the least. Grayscale images contain only one channel of information, reducing the amount of image data compared to the three channels of color images. Grayscale images can also more clearly highlight features such as image structure and texture, particularly the structural characteristics of the eye, making subsequent feature extraction and analysis easier.

[0115] Specifically, Gaussian filtering is used for noise reduction, and the specific algorithm is:

[0116] ;

[0117] in, Indicates that after Gaussian filtering, the coordinates The pixel value at ; It is the coordinate variable used to traverse the filter kernel; is the filter kernel The weight value at the coordinate is used to determine how to weight the surrounding pixels. Different filter kernel settings will have different value distributions. Indicates the original image at coordinates The pixel value at , that is, the original pixel value participating in the weighted average calculation; Represents the filter kernel used by Gaussian filtering. It is a two-dimensional matrix. The size and weight distribution determine the filtering effect. For example, the common 3x3 or 5x5 filter kernel matrix has internal elements of value.

[0118] Adaptive weights are introduced to optimize the Gaussian filtering algorithm:

[0119] First consider the local variance: calculate the original image in The variance in the local area centered on Regions with larger variances usually mean that the image has more details or is noisy, and require smaller filtering strengths; regions with smaller variances may be smoother regions, and larger filtering strengths can be used.

[0120] Secondly, adjust the weight: adjust the weight of the filter kernel according to the local variance You can use a function To calculate the adjusted weights, for example ,in is a pre-set maximum variance value. The adjusted weight is .

[0121] The final optimized formula, that is, the filtered pixel value becomes:

[0122] ;

[0123] in, is the adjusted weight;

[0124] The Gaussian filtering algorithm is also optimized by combining multi-scale filtering:

[0125] First, multiple Gaussian filter kernels of different scales are used, such as small-scale, medium-scale, and large-scale filter kernels. For each scale filter kernel, the filtered pixel value is calculated according to the original formula;

[0126] Then perform weighted fusion: perform weighted fusion based on the results of filtering at different scales. Assume that the pixel values ​​after filtering at different scales are ,in represents the scale, and the weight is . Then the optimized pixel value is:

[0127] ;

[0128] in, The weights can be obtained based on experience or through learning, for example, the weights can be determined based on local features of the image or by training a small neural network.

[0129] Through the above optimization, Gaussian filtering can be made more adaptable to the local features of the image, and better preserve the details of the image while removing noise.

[0130] S2. Model Construction: We use traditional convolutional neural networks (CNNs) and Vision Transformers convolutional neural networks to extract eye features from pre-processed images and use binary supervised learning to help distinguish myopic eyes from normal eyes.

[0131] In this step, the model is constructed by combining traditional convolutional neural networks (CNN) and Vision Transformers convolutional neural networks. These two network structures each have their own advantages. CNN performs well in local feature extraction and can effectively capture local texture and detail information in the image. Vision Transformers, on the other hand, excels at processing global information and can establish long-distance dependencies to better understand the overall structure and semantics of the image. By combining the two, the features of the eye image can be more comprehensively extracted to better distinguish between myopic eyes and normal eyes. The traditional convolutional neural network (CNN) includes convolution operations, pooling operations, and fully connected layers. The working principle of convolution is as follows:

[0132] ;

[0133] in, Is the pixel value of the convolution output, expressed at coordinates The new pixel value obtained after the convolution operation; is the pixel value of the input image, that is, the original eye image at coordinates The pixel value at ; Is the convolution kernel, which is a small matrix used to slide on the image for convolution operation. Its size and element value determine the effect of convolution; is the bias, which is a constant term used to adjust the position of the convolution result; It is an activation function (such as ReLU), which is used to introduce nonlinear factors and increase the expressive power of the model. The ReLU function is defined as , that is, when hour, ;then , .

[0134] The purpose of pooling is to reduce the amount of data while retaining important feature information. Common pooling methods include maximum pooling and average pooling: Maximum pooling is to select the maximum value in a small area as the output, for example, in a In the area, the maximum value of the four pixel values ​​is selected; average pooling calculates the average value of the pixel values ​​in a small area as the output;

[0135] The fully connected layer is used to integrate the feature vectors obtained by the previous convolution and pooling operations and map them to the final output dimension; each of its neurons is connected to all neurons in the previous layer.

[0136] Furthermore, the core of the Vision Transformers convolutional neural network is the self-attention mechanism; it can calculate the correlation between each position in the image and the rest of the positions, thereby better capturing global information;

[0137] Among them, the basic formula of self-attention is:

[0138] ;

[0139] in, They are the query vector, key vector, and value vector obtained by linearly transforming the input features; is the dimension of the key vector; The function is used to normalize the calculated weights so that their sum is 1.

[0140] Through the self-attention mechanism, the model can dynamically adjust the weights according to the relevance of different positions in the image, thereby better focusing on important feature areas.

[0141] The Vision Transformers convolutional neural network also includes a multi-layer perceptron (MLP): After the self-attention mechanism, a multi-layer perceptron (MLP) is usually connected to further process the features; the MLP consists of multiple fully connected layers and is used to perform nonlinear transformations on the features processed by the self-attention mechanism, thereby increasing the expressive power of the model.

[0142] Specifically, the combination of traditional convolutional neural networks (CNNs) and Vision Transformers convolutional neural networks includes:

[0143] The first one is feature fusion, which includes:

[0144] (1) Early fusion: Fusion is performed at lower layers of the network. First, a CNN is used to perform preliminary feature extraction on the input image, generating a set of local feature maps. These feature maps are then fed into the Vision Transformer encoder and combined with the Transformer's self-attention mechanism to extract higher-level features. During this process, fusion algorithms such as concatenation or weighted summation can be used to fuse the features extracted by the CNN with those extracted by the Transformer. For example, the feature map extracted by the CNN and the feature vector output by the Transformer encoder can be concatenated and then fed into subsequent network layers for further processing.

[0145] (2) Late fusion: Fusion is performed at a higher level in the network. CNN and Vision Transformers are used to extract features from the image independently, resulting in two different sets of feature representations. Then, in the later stages of the network, these two sets of features are fused together. Fusion strategies such as feature weighting and multi-layer perceptron (MLP) fusion can be used. For example, an MLP can be used to learn how to fuse the features extracted by CNN and Vision Transformers. The formula can be expressed as ,in is the fused feature. is the feature extracted by CNN, It is the feature extracted by Vision Transformers. MLP is a multi-layer perceptron that fuses the input features by learning weight parameters.

[0146] The second type is a hybrid architecture; it includes:

[0147] (1) Build a hybrid network: Design a hybrid network architecture that combines CNN and Vision Transformer modules. For example, you can use CNN modules at the front end of the network to extract local features, then introduce Vision Transformer modules in the middle layers to process global information, and finally use fully connected layers or other classifiers at the back end for classification. In this hybrid network, the ratio and connection method of CNN and Vision Transformer modules can be flexibly adjusted according to specific task requirements and data characteristics.

[0148] (2) Attention Mechanism Fusion: Combining the CNN's attention mechanism with the Vision Transformer's self-attention mechanism. For example, a Transformer-like self-attention module can be introduced into a CNN, or the CNN's convolutional attention mechanism can be used in Vision Transformers to enhance the focus on local features. This can further improve the model's ability to extract features at different levels while strengthening the interaction between local and global information.

[0149] Furthermore, in step S2, the extracted eye features include at least three related features of the eyeball structure, retinal structure and eye appearance; wherein:

[0150] Relevant features of the eyeball structure include eyeball shape, eye white pattern, eyeball axial length, and lens shape and position;

[0151] Relevant features of retinal structure include retinal vascularity and retinal thickness;

[0152] Relevant features of eye appearance include eyelid shape and eye spacing. Specific features are as follows:

[0153] 1. Axial length of the eyeball: Myopia is often caused by an excessively long eyeball. While it's impossible to accurately measure the axial length of the eyeball directly in eye images, it can be indirectly reflected through certain features. For example, in an image taken from the side, the eyeball of a myopic eye may appear more prominent, suggesting a relatively long axial length.

[0154] 2. Lens-related characteristics: The shape and position of the lens may change in myopic eyes. In normal eyes, the lens can adjust the focal length normally so that the image is clearly focused on the retina. In myopic eyes, the lens may become convex and its adjustment ability may be affected. By analyzing eye images, we can observe features such as changes in grayscale values ​​and boundary clarity in the lens area to distinguish myopic eyes from normal eyes. For example, if the grayscale values ​​of the lens area are relatively uniform and the boundaries are clear, it may be more like a normal eye; if there are abnormal changes in the grayscale values ​​and blurred boundaries, it may indicate the possibility of myopia.

[0155] 3. Retinal Vascular Distribution: The distribution of retinal vessels may differ between myopic and normal eyes. In normal eyes, retinal vessels are relatively regular and uniform in thickness. However, in myopia, changes in eye structure may cause retinal vessels to be stretched or squeezed, resulting in distortion, thinning, or localized changes in vessel density. Morphological analysis of retinal vessels in eye images can serve as a reference feature for distinguishing myopic and normal eyes.

[0156] Retinal thickness: Myopic eyes may have different retinal thickness than normal eyes. Long-term myopia may cause retinal thinning, especially in key areas such as the macula. Advanced eye imaging techniques or in-depth analysis of eye images can be used to obtain information related to retinal thickness to help distinguish myopic eyes from normal eyes.

[0157] 5. Eyelid morphology: While there's no direct causal relationship between eyelid morphology and myopia, some studies have found subtle changes in the eyelids of myopic patients. For example, the upper eyelids of myopic patients may be relatively loose, or the eyelid folds may appear differently. These features can serve as auxiliary information, combined with other ocular features, to distinguish myopic patients from normal eyes.

[0158] 6. Interocular distance: Interocular distance may differ between myopic and normal eyes. While this difference may not be significant, large-scale statistical analysis may reveal that the average interocular distance of myopic patients is different from that of normal eyes. Parameters related to interocular distance can be measured in eye images and used as a potential distinguishing feature.

[0159] S3. Model training and evaluation: Use cross-validation to train the model to avoid overfitting and improve the model's generalization ability. During the training process, adjust model parameters based on the performance of the validation set to optimize model performance. Collect prediction results from the training process and evaluate the model's classification performance using multiple parameters.

[0160] Cross-validation is chosen for model training primarily to avoid overfitting. When training data is limited, a model can easily overlearn the features of the training data, resulting in reduced generalization ability on new data. Cross-validation divides a dataset into multiple subsets, alternately using one subset as the validation set and the remaining subsets as the training set. Repeating the training and validation process multiple times allows for a more comprehensive assessment of model performance and improves its generalization ability.

[0161] In this step, the model is trained using the cross-validation method, which includes:

[0162] S3.1, divide the eye image dataset processed in step S1 into subsets of roughly equal size (usually The value of is 5 or 10, here we assume );

[0163] S3.2, proceed Round training and validation: In each round, one of the subsets is selected as the validation set, and the rest The first subset is used as the validation set, and the second to fifth subsets are used as the training set. The model is trained using the training set. During the training process, the model learns the relationship between the features of the eye images in the training set and the corresponding labels of myopia or normal eyes.

[0164] S3.3. After each round of training, the model is validated using the validation set of the current round to evaluate the performance of the model on data that was not involved in the training.

[0165] Furthermore, when adjusting model parameters:

[0166] First, adjust based on validation set performance: During training, closely monitor validation set performance. Validation set performance metrics (such as precision, recall, and F1 score) can reflect the model's ability to generalize to unseen data. If validation set accuracy begins to decline, or other performance metrics show suboptimal performance, the model may be overfitting.

[0167] Second, the parameters to adjust and how to adjust them: Common model parameters that require adjustment include the size and number of convolutional kernels in the convolutional neural network, the number of neural network layers, and the learning rate. For example, if you find that the model's accuracy on the validation set is low, you can try reducing the learning rate so that the model learns the features in the training data more slowly to avoid overfitting. Alternatively, you can increase the number of neural network layers to improve the model's expressiveness, but at the same time, be careful to prevent exacerbating overfitting. If you find that the model is overfitting, you can try adding regularization terms, such as L1 or L2 regularization, to limit the size of the model parameters and prevent the model from becoming too complex.

[0168] Furthermore, in step S3, the parameters used to evaluate the classification performance of the model include at least accuracy, precision, recall, and F1-score, with the F1-score being used as the final evaluation criterion; wherein:

[0169] Accuracy refers to the ratio of the number of samples predicted as positive or negative and correctly predicted to the total number of samples. The calculation formula is:

[0170] ;

[0171] Precision refers to the ratio of the number of samples predicted to be positive and actually positive to the number of samples predicted to be positive. The calculation formula is:

[0172] ;

[0173] It reflects the accuracy of the model's prediction results.

[0174] Recall refers to the ratio of the number of samples predicted to be positive and actually positive to the number of samples that are actually positive. The calculation formula is:

[0175] ;

[0176] It reflects the model's ability to correctly identify positive examples.

[0177] in, Indicates the number of samples that are predicted to be positive and are actually positive. Indicates the number of samples predicted to be positive but actually negative. Indicates the number of samples predicted to be negative but actually positive. Indicates the number of samples that are predicted to be negative and are actually negative; Indicates the number of samples predicted as positive examples; Indicates the number of samples that are actually positive examples;

[0178] F1 score is an evaluation indicator that comprehensively considers precision and recall. The formula is:

[0179] ;

[0180] It provides a more comprehensive evaluation of model performance, balancing the relationship between precision and recall.

[0181] S4. Multi-device screening application: The algorithm is embedded in web applications and multi-terminal mobile applications, which can perform vision screening anytime and anywhere, and support external camera devices to improve image quality.

[0182] Specifically, the model application includes the following steps:

[0183] A1. Mobile terminal device adaptation

[0184] Installation and Setup: A dedicated mobile application has been developed, supporting major mobile operating systems such as iOS and Android. Users can download and install the application from the app store. Once installed, the application will guide the user through the necessary settings, such as authorizing camera access and allowing storage permissions, to ensure the application can function properly and save captured eye images.

[0185] Image Capture Guidance: Provide clear image capture guidance within the app interface. For example, use both graphic and text to demonstrate the correct shooting posture and distance, ensuring users can accurately capture clear images of the eyes. Use animated demonstrations or video tutorials to help users better understand the shooting requirements. Additionally, the app can configure autofocus and exposure adjustment to adapt to different shooting environments and improve image quality.

[0186] A2. Eye image capture and upload

[0187] Shooting process: After opening the app, the user enters the image capture interface. The app automatically invokes the mobile device's camera and displays a real-time preview. Following the instructions, the user points the camera at their eyes, adjusting the distance and angle to ensure their eyes are fully within the frame and clearly visible. The app may provide guide lines or frames to help users better position their eyes. After the capture is complete, the user can review the image and retake it if unsatisfactory.

[0188] Image Upload: Completed eye images are automatically saved to the mobile device's local storage. Users can choose to upload the image for analysis immediately or at a later time. During the upload process, the app displays a progress bar and ensures secure transmission of the image data. Encryption and data compression methods can be used to improve upload speed and data security. The app also performs a preliminary quality check on the uploaded image, checking for clarity and completeness. If the image quality does not meet the requirements, the app prompts the user to retake or adjust the uploaded image.

[0189] A3. Display of vision monitoring results

[0190] Real-time analysis and feedback: Once the image is successfully uploaded, the server-side vision monitoring model immediately analyzes the image. During the analysis, the model extracts features from the eye image and compares them with a pre-trained model to determine whether the user's eyes are nearsighted or normal. The analysis results are quickly returned to the mobile device and displayed to the user on the application interface. Results can be presented in a concise and clear manner, such as text descriptions ("nearsightedness" or "normal eyes") or icons (e.g., glasses icon for nearsightedness, clear eyes icon for normal eyes).

[0191] Detailed report generation: In addition to simply displaying results, the app can also generate detailed vision monitoring reports. These reports can include additional information, such as eye features extracted during analysis, comparisons with normal or nearsighted eyes, and potential vision risk warnings. Reports can be presented as PDFs or web pages, making them easy for users to save and share. The reports can also provide recommendations, such as regular vision exams, maintaining good eye hygiene, and managing eye time appropriately, to help users better protect their vision.

[0192] It also includes result analysis, including the following steps:

[0193] B1. Accuracy Assessment

[0194] Comparative Validation: To evaluate the model's accuracy, compare the model's analysis results with the diagnoses of professional ophthalmologists. A certain number of eye image samples can be collected and analyzed and judged by both the model and ophthalmologists. The results are compared and evaluation metrics such as precision, recall, and F1 score are calculated. If the model's accuracy is low, further optimization of model parameters, increased training data volume, and improved image preprocessing methods can be used to improve model performance.

[0195] Cross-validation: Using cross-validation, the dataset is divided into multiple subsets. The model is trained and validated multiple times, alternating between using one subset as the validation set and the remaining subsets as the training set. Cross-validation provides a more comprehensive assessment of the model's generalization and stability. If the model's performance varies significantly across different subsets, it indicates that the model may be overfitting or underfitting, and further adjustments to the model structure and parameters are needed.

[0196] B2. User Feedback Collection

[0197] Establish feedback channels: Set up user feedback channels within mobile apps, such as feedback buttons and online questionnaires. Users can use these channels to provide developers with feedback on their experience, their opinions on the accuracy of results, and suggestions for improvement. Developers can regularly collect and analyze user feedback to understand user needs and issues, allowing them to make timely improvements and optimizations to their apps.

[0198] Feedback Analysis and Application: User feedback is categorized and analyzed to extract valuable information. For example, if many users report inaccurate results, further investigation can be conducted to determine the cause, which could be image quality issues, model performance issues, or improper user operation. Based on the feedback analysis results, appropriate improvement measures can be implemented, such as optimizing image acquisition guidance, improving model accuracy, and refining the user interface. Furthermore, timely responses to user feedback ensure that users feel the developer's attention and value, thereby improving user satisfaction.

[0199] B3. Continuous improvement and optimization

[0200] Data Updates and Model Optimization: Over time, new eye image data is continuously collected and added to the training set to continuously optimize and improve the model. The model can be retrained regularly and its parameters adjusted to adapt to changing real-world conditions. At the same time, we monitor the latest research findings and technological developments in ophthalmology, promptly introducing new feature extraction methods and model structures to improve model performance and accuracy.

[0201] Application Functionality Expansion: Continuously expand the functionality of the mobile application based on user needs and market feedback. For example, we could add a vision trend analysis feature to allow users to track their vision changes over time; or add an eye health knowledge module to provide more information on eye hygiene and myopia prevention. We could also collaborate with professional ophthalmology institutions to provide users with online consultations, appointment bookings, and other services, enhancing the app's practicality and value.

[0202] In summary, by collecting data and conducting application tests, the above-mentioned method for detecting myopia in adolescents based on the combination of hybrid convolutional neural network and VisionTransformer comprehensively measures the classification performance of the model by calculating evaluation indicators such as accuracy, precision, and recall rate on the test set. The accuracy rate is 12% higher than that of the traditional CNN model, and the precision and recall rates are increased by 10% and 15% respectively; it is more than 80% faster than the traditional myopia scale test.

[0203] Those skilled in the art will appreciate that the process of implementing all or part of the steps of the above embodiments may be accomplished by hardware, or by instructing related hardware through a program.

[0204] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A vision health monitoring method based on shooting, tracking and comparison of mobile terminal devices, characterized in that: The steps include: S1. Data collection and image preprocessing: A large number of photos of the left eye, right eye, and both eyes of teenagers were collected. The latest object detection network YOLOv10 was used in combination with OpenCV to perform eye cropping and preprocessing, and the images were uniformly resized. S2. Model construction: We use traditional convolutional neural networks (CNNs) and Vision Transformers (VTs) to extract eye features from preprocessed images and use binary supervised learning to help distinguish myopic eyes from normal eyes. S3. Model training and evaluation: Use cross-validation to train the model. During training, adjust model parameters based on the performance of the validation set to optimize model performance. Calculate the prediction results of the training process and evaluate the classification performance of the model using multiple parameters. S4. Multi-device screening application: The algorithm is embedded in web applications and multi-terminal mobile applications, and supports external camera devices to improve image quality.

2. The vision health monitoring method based on shooting, tracking and comparison by a mobile terminal device according to claim 1 is characterized in that: In step S1, the core idea of ​​yolov10 is to divide the input image into multiple grid cells, each grid cell is responsible for predicting a certain number of bounding boxes and the category probability of the objects in these bounding boxes; it consists of a feature extraction layer and a prediction layer; the feature extraction layer contains multiple convolutional layers and pooling layers for extracting image features; the convolution layer performs convolution operations by sliding the convolution kernel on the image; the pooling layer is used to reduce the amount of data; the last few layers of the network are prediction layers, which predict the position of the bounding box and the category of the target based on the previously extracted features; for each grid cell, it predicts the coordinates of multiple bounding boxes ,in and are the coordinates of the bounding box center relative to the grid cell, and is the width and height of the bounding box; at the same time, the probability of the target belonging to different categories in each bounding box is also predicted ,in Indicates category; Yolov10 combined with opencv for eye cropping specifically includes the following: S1.

1. Haar feature difference calculation assistance: During the eye cropping process, Haar feature difference calculation is used to help detect edges in the image; S1.

2. Cropping based on yolov10 prediction results: First, the image is fed into the yolov10 network for object detection; the network outputs the predicted bounding box coordinates for each grid cell. and class probabilities ; Then, the bounding boxes of the eye category are filtered out, and the image is cropped according to the coordinate information of these bounding boxes.

3. The vision health monitoring method based on mobile terminal device shooting, tracking and comparison according to claim 2 is characterized in that: In step S1.1, Haar feature difference calculation is used to help detect edges in the image. The specific algorithm is: ； in, Indicated in coordinates The value obtained by calculating the Haar feature difference is used to detect the edge in the image to help determine the boundary of the eye; It is used to traverse the area and Coordinate variables of and Represent two different areas in the image and are used to calculate the difference to highlight the edge features; It is a function of the pixel value of the image, which represents the pixel value of the image at a certain coordinate; Different regions may contribute differently to feature differences. Introducing a weight factor can better reflect this difference. The optimized formula is: ； in, and They are regions and The weight factor can be obtained based on prior knowledge, experience or learning; Calculating Haar feature differences at different scales can capture features at different levels of the image. On the basis of introducing weight factors, combined with multi-scale analysis, the robustness and accuracy of feature difference calculation can be improved. The optimized formula is: ； in, Represents different scales, and In scale The area below, and is the corresponding weight factor, It is on scale Image pixel value function under ; The features of a pixel in an image are often related to the features of its neighboring pixels. When calculating Haar feature differences, considering neighborhood information can improve the accuracy and stability of the calculation. The optimized formula is: ； in, Is a function that represents pixel The influence of neighborhood information on its eigenvalues.

4. The vision health monitoring method based on mobile terminal device shooting, tracking and comparison according to claim 1 is characterized in that: In step S1, the cropped eye image is preprocessed, including grayscale conversion, noise reduction, image enhancement, and image normalization; wherein image noise reduction includes mean filtering, median filtering, and Gaussian filtering; image enhancement includes contrast enhancement and sharpening; and image normalization includes size normalization and grayscale normalization; Among them, grayscale determines the weight according to the sensitivity of the human eye to different colors. The specific algorithm is: ； in, Indicates the grayscale value of a certain pixel in the image after grayscale processing; They represent the pixel values ​​of the red, green, and blue channels of a certain pixel in a color image respectively; 0.299, 0.587, and 0.114 are the sensitivity ratios of the human eye to red, green, and blue, respectively.

5. The method for monitoring vision health based on shooting, tracking and comparison by a mobile terminal device according to claim 4, characterized in that: The Gaussian filter is used for noise reduction, and the specific algorithm is: ； in, Indicates that after Gaussian filtering, the coordinates The pixel value at ; It is the coordinate variable used to traverse the filter kernel; is the filter kernel The weight value at the coordinate is used to determine the weighting method for surrounding pixels; Indicates the original image at coordinates The pixel value at , that is, the original pixel value participating in the weighted average calculation; Represents the filter kernel used by Gaussian filtering. It is a two-dimensional matrix. The size and weight distribution determine the effect of the filtering. Adaptive weights are introduced to optimize the Gaussian filtering algorithm: First consider the local variance: calculate the original image in The variance in the local area centered on ; Secondly, adjust the weight: adjust the weight of the filter kernel according to the local variance ; The final optimized formula, that is, the filtered pixel value becomes: ； in, is the adjusted weight; The Gaussian filtering algorithm is also optimized by combining multi-scale filtering: First, multiple Gaussian filter kernels of different scales are used; for each scale of the filter kernel, the filtered pixel value is calculated according to the original formula; Then perform weighted fusion: perform weighted fusion according to the results after filtering at different scales; suppose the pixel values ​​after filtering at different scales are ,in represents the scale, and the weight is ; The optimized pixel value is: ； in, .

6. The method for monitoring vision health based on shooting, tracking and comparison by a mobile terminal device according to claim 1, characterized in that: In step S2, a model is constructed by combining a traditional convolutional neural network (CNN) and a Vision Transformers convolutional neural network. The traditional convolutional neural network (CNN) includes a convolution operation, a pooling operation, and a fully connected layer. The working principle of convolution is as follows: ； in, Is the pixel value of the convolution output, expressed at coordinates The new pixel value obtained after the convolution operation; is the pixel value of the input image, that is, the original eye image at coordinates The pixel value at ; Is the convolution kernel, which is a small matrix used to slide on the image for convolution operation. Its size and element value determine the effect of convolution; is the bias, which is a constant term used to adjust the position of the convolution result; It is an activation function, which is used to introduce nonlinear factors and increase the expressive power of the model; The purpose of the pooling operation is to reduce the amount of data while retaining important feature information; common pooling methods include maximum pooling and average pooling: Max pooling is to select the maximum value in a small area as the output; Average pooling calculates the average value of pixel values ​​in a small area as output; The fully connected layer is used to integrate the feature vectors obtained by the previous convolution and pooling operations and map them to the final output dimension; each of its neurons is connected to all neurons in the previous layer.

7. The method for monitoring vision health based on shooting, tracking and comparison by a mobile terminal device according to claim 6, characterized in that: The core of the Vision Transformers convolutional neural network is the self-attention mechanism; it can calculate the correlation between each position in the image and the rest of the positions, thereby better capturing global information; Among them, the basic formula of self-attention is: ； in, They are the query vector, key vector, and value vector obtained by linearly transforming the input features; is the dimension of the key vector; The function is used to normalize the calculated weights so that their sum is 1; The Vision Transformers convolutional neural network also includes a multi-layer perceptron (MLP). After the self-attention mechanism, a multi-layer perceptron is usually connected to further process features. The MLP consists of multiple fully connected layers and is used to perform nonlinear transformations on features processed by the self-attention mechanism, thereby increasing the expressive power of the model.

8. The method for monitoring vision health based on shooting, tracking and comparison by a mobile terminal device according to claim 1, characterized in that: In step S2, the extracted eye features include at least three related features of the eyeball structure, retinal structure and eye appearance; wherein: Relevant features of the eyeball structure include eyeball shape, eye white pattern, eyeball axial length, and lens shape and position; Relevant features of retinal structure include retinal vascularity and retinal thickness; Relevant features of ocular appearance include eyelid morphology and interocular distance.

9. The method for monitoring vision health based on shooting, tracking and comparison by a mobile terminal device according to claim 1, characterized in that: In step S3, the model is trained using a cross-validation method, specifically including: S3.1, divide the eye image dataset processed in step S1 into subsets of roughly equal size; S3.2, proceed Round training and validation: In each round, one of the subsets is selected as the validation set, and the rest The model is trained using the training set. During the training process, the model learns the relationship between the features of the eye images in the training set and the corresponding labels of myopia or normal eyes. S3.

3. After each round of training, the model is validated using the validation set of the current round to evaluate the performance of the model on data that was not involved in the training.

10. The method for monitoring vision health based on shooting, tracking and comparison by a mobile terminal device according to claim 9, characterized in that: In step S3, the parameters used to evaluate the classification performance of the model include at least accuracy, precision, recall, and F1-score, with the F1-score being used as the final evaluation criterion; wherein: Accuracy refers to the ratio of the number of samples that are predicted to be positive or negative and are correctly predicted to the total number of samples. The calculation formula is: ； Precision refers to the ratio of the number of samples predicted to be positive and actually positive to the number of samples predicted to be positive. The calculation formula is: ； Recall refers to the ratio of the number of samples predicted to be positive and actually positive to the number of samples that are actually positive. The calculation formula is: ； in, Indicates the number of samples that are predicted to be positive and are actually positive. Indicates the number of samples predicted to be positive but actually negative. Indicates the number of samples predicted to be negative but actually positive. Indicates the number of samples that are predicted to be negative and are actually negative; Indicates the number of samples predicted as positive examples; Indicates the number of samples that are actually positive examples; The F1 score is an evaluation indicator that comprehensively considers precision and recall. The formula is: ； It provides a more comprehensive evaluation of model performance to balance the relationship between precision and recall.

Images