Eye physiotherapy and training method based on AR technology

Abstract

The invention discloses an AR technology-based eye physiotherapy and training method. The method comprises the steps of obtaining eye image data containing pupil edge contour features and a pupil size time sequence; preprocessing the image, wherein the preprocessing comprises standardization, Min-Max normalization and Canny edge detection processing; constructing a hybrid neural network model, wherein the model is formed by integrating a GRU unit and an LVT through a parallel architecture; according to a pupil recognition result output by the model, real-time recognition and classification of eye pupils and pupil sizes are completed; eye pupils are identified by utilizing edge detection, and a smooth and continuous pupil size time sequence is formed as an input GRU and Lite-ViT hybrid network according to up-sampling of samples, so that eye pupil identification and pupil size classification can be completed.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and AI-assisted technology, specifically to an eye therapy and training method based on AR technology. Background Technology

[0002] Eye-tracking research offers significant prospects for medical image analysis and interpretation. Eye tracking, a technology that monitors and records eye movements, provides valuable insights into human visual attention patterns. By leveraging human eye gaze data to extract meaningful features and insights from medical images, our understanding of how radiologists and other medical professionals monitor, interpret, and interpret images for diagnostic purposes can be improved. Eye-tracking data embeds complex human visual attention patterns, providing a bridge to integrating artificial intelligence (AI) development with human cognition. This integration allows for novel approaches to incorporate domain knowledge into machine learning (ML) and deep learning (DL) methods, enhancing their consistency with human-like perception and decision-making. Furthermore, the extensive collection of eye-tracking data enables novel ML / DL methods to analyze human visual patterns, paving the way for a better understanding of human vision, attention, and cognition; the application of deep learning in eye-motion tracking, i.e., eye-tracking technology, has revolutionized this field. However, existing methods suffer from poor timeliness and low accuracy on eye-tracking devices. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide an eye therapy and training method based on AR technology, which solves the problems of poor timeliness and low accuracy of models used on eye-tracking devices.

[0004] Technical solution: The present invention provides an eye therapy and training method based on AR technology, comprising:

[0005] (1) Obtain eye image data containing pupil edge contour features and pupil size time series;

[0006] (2) Image preprocessing includes standardization, Min-Max normalization and Canny edge detection;

[0007] (3) Construct a hybrid neural network model, which integrates GRU units and LVT through a parallel architecture. The GRU units are used to process pupil size time series data; the LVT model contains a convolutional self-attention (CSA) module and at least one recursive dilated self-attention (RASA) module to obtain the spatial features of the pupil.

[0008] (4) Based on the pupil recognition results output by the model, complete the real-time recognition and classification of eye pupils and pupil size.

[0009] Further, the preprocessing is as follows: The input image size is standardized to S using bilinear interpolation. W =(W w +H w W w and H w The working width and height are respectively; the original aspect ratio is maintained; the obtained image is linearly normalized using the Min-Max method; edge detection is performed using Canny edges, and each edge segment is first fitted by a set of dominant points according to the k-nearest neighbor method to obtain an ellipse fit; the pupil size time series data consists of data points with unequal spacing.

[0010] Furthermore, the Convolutional Self-Attention (CSA) module of the LVT model performs the following operations:

[0011] Local features are calculated using sliding window convolution: y i =∑ j∈N(i) W i→j x j ;

[0012] Global feature weights are calculated using a multi-head self-attention mechanism: y i =∑ j∈N(i) α i→j W V X j and Where α i→j ∈(0,1) is a scalar used to control the contribution of each spatial location value in the summation. α is normalized using soft maximum operation to make α... i→j =1;

[0013] The hidden state is updated recursively using the following formula:

[0014] x t+1 =ASA(F(X) t ,h t-1 )),h t-1 =X t-1 ,X t =ASA(F(X) t-1 ,h t-2 Where t is the step size, and h∈R d×H×W It is in a hidden state.

[0015] Furthermore, the Recursive Dilated Self-Attention (RASA) module employs multi-scale dilated convolutions to process the query vector:

[0016]

[0017] Multi-scale contextual features are captured by using convolutional layers with parallel dilation rates of {1, 3, 5}.

[0018] Furthermore, the hybrid neural network employs a feature-level fusion approach, fusing the temporal features output by the GRU with the spatial features output by the LVT through a fully connected layer.

[0019] Furthermore, we select the mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R²). 2 As the evaluation criterion for the model, the present invention provides an eye therapy and training system based on AR technology, comprising:

[0020] The data acquisition module is used to acquire eye image data including pupil edge contour features and pupil size time series;

[0021] The data processing module is used to perform image size standardization, Min-Max normalization, and Canny edge detection processing;

[0022] The hybrid neural network module, which integrates GRU units and LVTs through a parallel architecture, is used to input the processed spatiotemporal features into the GRU-LVT hybrid model for training.

[0023] The feedback control module, based on the pupil recognition results output by the model, completes real-time recognition and classification of the pupil and pupil size.

[0024] An electronic device according to the present invention includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the steps of any of the methods described herein.

[0025] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described herein.

[0026] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: Based on different pupil features, this invention adds multiple discrimination methods to the original data, making the final prediction results more accurate and the data more effective; the GRU-LVT models constructed in this invention all use lightweight models, meeting real-time requirements and reducing computational load on mobile devices; this invention's model does not simply use GRU or Vision Transformer models, but combines both, making the model more accurate; the model's use on mobile devices not only satisfies the needs of convenient computation and low computational resource consumption, resulting in better real-time performance, but also provides more accurate prediction results. Attached Figure Description

[0027] Figure 1 This is a flowchart of the present invention;

[0028] Figure 2 This is a schematic diagram of the LVT network model of the present invention;

[0029] Figure 3 This is the GRU-LVT network model fusion structure of the present invention. Detailed Implementation

[0030] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0031] like Figure 1 As shown, this embodiment of the invention provides an eye therapy and training method based on AR technology, including the following steps:

[0032] Step S1: Collect pupil data and determine features; divide the dataset into training, testing, and validation sets; including the following steps:

[0033] Step S1.1: Select images from multiple sources, including taking photos, searching and downloading open-source eye photos online, and searching for publicly available resources in literature, and conduct preliminary screening of the selected images;

[0034] Step S1.2: The pupil image data is divided into pupil edge contour data and pupil size data. The pupil contour data is determined by three indicators to determine its confidence measure, while the pupil size data is processed to generate a time series.

[0035] Step S1.3: For the pupil data in the dataset, randomly select 70% as the training set, 20% as the test set, and 10% as the validation set.

[0036] Step S2: Represent the pupil data in the dataset as a time series and perform standardization; the specific method is as follows:

[0037] Before processing, the input image is reduced to the working size using bilinear interpolation, S W =(W w +H w W w and H w These are the working width and height, respectively. During dimensionality reduction, the original aspect ratio is used.

[0038] The obtained image is linearly normalized using the Min-Max method;

[0039] Edge detection was performed using Canny Edge, where each edge segment was first approximated by a set of dominant points using the k-nearest neighbor method to produce a better ellipse fit. The pupil size time series data consisted of data points with unequal spacing.

[0040] Step S3: Construct an eye pupil recognition model; the eye pupil recognition model takes the processed eye pupil image data in a standardized format as input and outputs the predicted results of the eye pupil, including accurately identifying the edge of the pupil to determine the pupil state, and outputting the eye pupil recognition accuracy result; perform multiple rounds of repeated training, continuously adjusting the parameters to make the GRU and Lite-ViT hybrid network model more accurate; the details are as follows:

[0041] A gated recurrent unit (GRU) that takes pupil feature data as input; the GRU mainly consists of two gating structures: an update gate and a reset gate. The calculation formula for the update gate is Z. t =σ(W z ·[h t-1 ,X t ]+b z ), where W z This is the weight matrix of the update gate, and it's the sigmoid function, whose output value is between 0 and 1. When Z... t When Z approaches 1, it indicates that more of the past hidden states are retained; when Z approaches 1... t When the value is close to 0, it indicates that the current input is used more frequently to update the hidden state; the formula for calculating the reset gate is r. t =σ(W r ·[h t-1 ,X t ]+b r The reset of the door confirmed the hidden state of the previous moment. t-1 How to influence the candidate hidden state h at the current moment? t His input r t It determines whether or not to ignore information from past hidden states.

[0042] The LVT (Lite Vision Transformer) model in the pupil recognition model is primarily based on four self-attention modules, including one Convolutional Self-Attention Module (CSA) and three Resursive Attention Modules (RASA). The first step of the Convolutional Self-Attention Module (CSA) is convolution, followed by analysis of the input and output feature vectors, where d represents the number of channels. The index space is then defined. Convolution is computed using a sliding window. Within each window, the convolution formula can be written as: y i =∑ j∈N(i) W i→j x j Then, further analysis of Self-Attention reveals that it requires three projection matrices to calculate the query, key, and value. The formula for Self-Attention can be written as: y i =∑ j∈N(i) αi→j W V X j and Where α i→j ∈(0,1) is a scalar used to control the contribution of each spatial location value in the summation. α is normalized using soft maximum operation to make α... i→j =1, meaning the number of learnable matrices is 3 instead of k, compared to a convolution with the same kernel size k. 2 For the Resursive Attention (RASA) module, unlike convolution, the feature response of self-attention is a weighted sum of projected input vectors from all spatial locations. These weights are determined by the similarity between the query and the key and represent the strength of the relationship between any pair of feature vectors. Therefore, multi-scale information is added when generating the weights. Specifically, the computation of the query is upgraded from a 1×1 convolution to the following operation;

[0043]

[0044] First, a 1×1 convolution is used to apply a linear projection. Then, three convolutions with different dilation rates but sharing a kernel are applied to capture multi-scale context. The parameter overhead is further reduced by setting the number of groups to the number of feature channels. Then, the parallel features at different scales are weighted and summed.

[0045] Secondly, the other three (RASA) methods increase their depth without increasing the number of parameters used. Combined with Atrous Self-Attention (ASA), they propose Recursive Self-Attention (RASA), whose formula can be written as:

[0046] x t+1 =ASA(F(X) t ,h t-1 )),h t-1 =X t-1 ,X t =ASA(F(X) t-1 ,h t-2 ))

[0047] Where t is the step size, and h∈R d×H×W It is in a hidden state;

[0048] Step S4: Input the standardized pupil data from the test set into the GRU and LVT hybrid network model and test the various indicators of the model;

[0049] Based on step S4, the mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R²) are selected to assess the performance of the pupil model. 2 () is the evaluation criterion for the model.

[0050] Step S5: Apply the GRU-LVT hybrid network model to complete the real-time recognition and classification of pupil size and pupil size.

Claims

1. An eye therapy and training method based on AR technology, characterized in that, include: (1) Obtain eye image data containing pupil edge contour features and pupil size time series; (2) Image preprocessing includes standardization, Min-Max normalization and Canny edge detection; (3) Construct a hybrid neural network model, which integrates GRU units and LVT through a parallel architecture. The GRU units are used to process pupil size time series data; the LVT model contains a convolutional self-attention (CSA) module and at least one recursive dilated self-attention (RASA) module to obtain the spatial features of the pupil. (4) Based on the pupil recognition results output by the model, complete the real-time recognition and classification of eye pupils and pupil size.

2. The eye therapy and training method based on AR technology according to claim 1, characterized in that, The preprocessing is as follows: The input image size is standardized to S using bilinear interpolation. W =(W w +H w W w and H w The working width and height are respectively; the original aspect ratio is maintained; the obtained image is linearly normalized using the Min-Max method; edge detection is performed using Canny edges, and each edge segment is first fitted by a set of dominant points according to the k-nearest neighbor method to obtain an ellipse fit; the pupil size time series data consists of data points with unequal spacing.

3. The eye therapy and training method based on AR technology according to claim 1, characterized in that, The Convolutional Self-Attention (CSA) module of the LVT model performs the following operations: Local features are calculated using sliding window convolution: y i =∑ j∈N(i) W i→j x j ; Global feature weights are calculated using a multi-head self-attention mechanism: y i =∑ j∈N(i) α i→j W V X j and Where α i→j ∈(0,1) is a scalar used to control the contribution of each spatial location value in the summation. α is normalized using soft maximum operation to make α... i→j =1; The hidden state is updated recursively using the following formula: x t+1 =ASA(F(X) t ,h t-1 )),h t-1 =X t-1 ,X t =ASA(F(X) t-1 ,h t-2 Where t is the step size, and h∈R d×H×W It is in a hidden state.

4. The eye therapy and training method based on AR technology according to claim 1, characterized in that, The Recursive Dilated Self-Attention (RASA) module uses multi-scale dilated convolutions to process query vectors. Multi-scale contextual features are captured by using convolutional layers with parallel dilation rates of {1, 3, 5}.

5. The eye therapy and training method based on AR technology according to claim 4, characterized in that, Hybrid neural networks employ a feature-level fusion approach, fusing the temporal features output by the GRU with the spatial features output by the LVT through a fully connected layer.

6. The eye therapy and training method based on AR technology according to claim 5, characterized in that, Select the mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R²). 2 The most important evaluation criterion for the model.

7. An eye therapy and training system based on AR technology, characterized in that, include: The data acquisition module is used to acquire eye image data including pupil edge contour features and pupil size time series; The data processing module is used to perform image size standardization, Min-Max normalization, and Canny edge detection processing; The hybrid neural network module, which integrates GRU units and LVTs through a parallel architecture, is used to input the processed spatiotemporal features into the GRU-LVT hybrid model for training. The feedback control module, based on the pupil recognition results output by the model, completes real-time recognition and classification of the pupil and pupil size.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images