Screening method and screening device for distinguishing early glaucoma patients

Abstract

The invention discloses a screening method and a screening device for distinguishing early glaucoma patients. The screening method comprises the following steps that a subject wears VR equipment, a fixation point is calibrated, a pre-detection scene image is displayed to collect eye movement and head movement data for calibration, then the detection scene image is displayed, and corresponding data is collected. And merging the data by taking a time point as a reference, and substituting the merged data into a glaucoma-normal person classification model for evaluation so as to judge whether the subject has glaucoma characteristics or not. According to the screening method and the screening device provided by the invention, the eye movement and head movement features of the subject are analyzed based on the virtual reality equipment, and the feature model of the glaucoma patient is established, so that early glaucoma patients can be effectively distinguished, and technical support is provided for early screening.

Description

TECHNICAL FIELD

[0001] The present application relates to a screening method for distinguishing early glaucoma patients, and also relates to a corresponding screening device, and belongs to the technical field of medical auxiliary diagnosis. BACKGROUND

[0002] Glaucoma is a group of irreversible blinding eye diseases characterized by progressive damage to the optic nerve. Related studies have shown that compared with elderly people with normal vision, elderly people with glaucoma are 3-4 times more likely to fall within a year, and it has been confirmed that severe visual impairment is associated with falls.

[0003] The traditional glaucoma diagnosis method is fundus medical image analysis, which comprehensively detects and identifies the fundus lesions of patients by analyzing the shape, color and structure of the fundus of the patients. For example, in the Chinese invention patent with the patent number ZL 202410343063.0, a fundus medical image analysis method based on computer vision technology is disclosed. The method collects the fundus medical image of the patient, and introduces an image processing and analysis algorithm based on computer vision technology and artificial intelligence technology in the backend to analyze the fundus medical image.

[0004] Similarly, the prior art is to detect and identify the fundus lesions of patients by comprehensively analyzing the shape, color and structure of the fundus of the patients. However, in reality, the early symptoms of primary glaucoma are hidden, and only 10%-50% of the patients with glaucoma are aware of their condition, and 75% of the patients who seek medical treatment have developed to the middle and late stages of glaucoma. Therefore, early glaucoma screening is crucial. SUMMARY

[0005] The primary technical problem to be solved by the present application is to provide a screening method for distinguishing early glaucoma patients.

[0006] Another technical problem to be solved by the present application is to provide a screening device for executing the above-mentioned screening method.

[0007] To achieve the above technical purposes, the present application adopts the following technical solutions: According to the first aspect of the embodiment of the present application, a screening method for distinguishing early glaucoma patients is provided, which comprises the following steps: Step 11: the subject wears a VR device to calibrate the gaze point; Step 12: enter the pre-detection time, display the image of the pre-detection scene on the display module of the VR device, collect the eye movement data and head movement data of the subject when observing the image, and calibrate; Step 13: enter the detection time, display the image of the detection scene on the display module, and collect the eye movement data and head movement data of the subject when observing the image; Step 14: merge the eye movement data and the head movement data based on time points; Step 15: input the merged data obtained in the above step into the glaucoma-normal person classification model for evaluation to determine whether the subject meets the characteristics of glaucoma patients.

[0008] Preferably, the detection scene shown in step 13 displays all instance objects in the virtual reality environment as sub-objects of the VR device camera, so that the relative positions of the eye position and the specified field of view range do not change.

[0009] Preferably, step 13 includes the following sub-steps: Sub-step 131: enter the gaze detection time, display the gaze detection content on the display module, and the subject gazes at the detection content; the gaze detection content is an instance object in a static state within a 10° and 24° circular field of view range in a white and black panoramic background; Sub-step 132: enter the tracking detection time, display the tracking detection content on the display module, and the subject gazes at the tracking detection content; the tracking detection content is an instance object in a moving state within a 10° and 24° circular field of view range in a white and black panoramic background; Sub-step 133: enter the matching detection time, display the matching detection content on the display module, and the subject gazes at the matching detection content; the matching detection content is a plurality of instance objects in a static state within a field of view range in a white and black panoramic background.

[0010] Preferably, the panoramic background of step 12 and step 13 is white and black, and the white and black backgrounds appear alternately when the steps are switched.

[0011] Preferably, step 14 includes the following sub-steps: Sub-step 141: read the eye movement data and the head movement data into a DataFrame table type data structure; Sub-step 142: store the timestamp difference of the eye movement data and the head movement data; store the minimum timestamp difference found, and the initial value is set to infinity; record the index of the last merged row in the head movement data, and the initial value is 0; Sub-step 143: traverse each row of the eye movement data from the 0th row to the mth row, where m is a positive integer; Sub-step 144: traverse each row of the head movement data from the index specified row to the ith row, where i is a positive integer; Sub-step 145: Calculate the timestamp difference of the current row; if the difference is less than the minimum timestamp difference of the current record, update the minimum timestamp difference; otherwise, merge the mth row of the eye movement data and the ith row of the head movement data; reset the timestamp difference and the minimum timestamp difference, and update the index i which is the index of the current row.

[0012] Sub-step 146: exit sub-step 144 and continue sub-step 143 until the eye movement data is traversed; Sub-step 147: save the merged timing data.

[0013] Preferably, the step 13 comprises the following sub-steps: Sub-step 134: evaluate the eye movement data in the detection time to obtain a task index; Sub-step 135: compare the task index with the data of the baseline experiment to determine whether the subject meets the characteristics of glaucoma patients.

[0014] Preferably, the method for obtaining the glaucoma-normal person classification model comprises the following steps: Step 21: diagnose the eye health of the subject through medical examination; Step 22: the subject wears a VR device to calibrate the gaze point; Step 23: enter the pre-detection time, display the image of the detection scene on the display module, collect the eye movement and head movement data of the subject when observing the image, and calibrate; Step 24: enter the detection time, display the image of the detection scene on the display module, and collect the eye movement and head movement data of the subject when observing the image; Step 25: merge the eye movement data and the head movement data based on the time point; Step 26: obtain sufficient data of the diagnosed subject to form a data set, and divide the training set and the test set of the data set; Step 27: based on the 1DCNN model and the training set, train and establish the glaucoma-normal person classification model, and evaluate the model with the test set.

[0015] Preferably, the step 27 comprises the following sub-steps: Sub-step 271: the network model is based on the 1DCNN model and the training set, and adds LSTM or Attention layer based on CNN to construct 1D-CNN model, LSTM+1DCNN model, 1DCNN+Attention model and LSTM+1DCNN+Attention model for training; Sub-step 272: obtain model evaluation indexes based on the test set; Sub-step 273: Select the best model according to the evaluation index.

[0016] According to a second aspect of the embodiments of the present application, a screening device for distinguishing early glaucoma patients is provided, comprising a VR device and a control device; wherein, The VR device comprises a display module, a head movement sensor, an eye movement sensor, an audio module and a communication module; the display module is used to display images; the head movement sensor is used to acquire head movement data; the eye movement sensor is used to acquire eye movement data; the audio module is used to play audio prompts; and the communication module is used to realize data transmission. The control device comprises a computing module and a storage module, and the computing module reads a computer program in the storage module and is used to execute the screening method for distinguishing early glaucoma patients.

[0017] Compared with the prior art, the present application provides a screening method for early glaucoma patients based on virtual reality technology and a corresponding screening device. The method can effectively distinguish early glaucoma patients and normal people by making the subjects wear VR devices and collecting a series of eye movement and head movement data. In a virtual reality environment, the eye movement and head movement characteristics generated by the subjects under different scenes and tasks are recorded and analyzed, and these characteristics are significantly different between early glaucoma patients and normal people. By combining the collected data and substituting them into a glaucoma-normal person classification model for evaluation, it can be accurately judged whether the subject meets the characteristics of early glaucoma patients. The present application improves the accuracy of early glaucoma screening and helps to reduce the risk of irreversible vision loss caused by glaucoma. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1A For the white background scene of the eye movement detection scene in the embodiments of the present application, a white background scene schematic diagram is shown; Figure 1B For the white background detection picture and the ball outside scene of the eye movement detection scene in the embodiments of the present application, a white background detection picture and a ball outside scene schematic diagram are shown; Figure 1C For the black background scene of the eye movement detection scene in the embodiments of the present application, a black background scene schematic diagram is shown; Figure 1D For the black background detection picture and the ball outside scene of the eye movement detection scene in the embodiments of the present application, a black background detection picture and a ball outside scene schematic diagram are shown; Figure 2 For the screen recording schematic diagram of the eye movement detection in the embodiments of the present application, a screen recording schematic diagram is shown; Figure 3 For the distribution ratio schematic diagram of the training set, the test set and the two types of labels of the data set in the embodiments of the present application, a distribution ratio schematic diagram is shown; Figure 4A For the loss function descent curve of the training set in the embodiments of the present application, a loss function descent curve schematic diagram is shown; Figure 4B Figure 2 shows the loss function curve of the test set in an embodiment of the present application. Figure 5A Figure 3 shows the confusion matrix of the 1DCNN model in an embodiment of the present application. Figure 5B Figure 4 shows the confusion matrix of the 1DCNN-Attention model in an embodiment of the present application. Figure 5C Figure 5 shows the confusion matrix of the LSTM-1DCNN model in an embodiment of the present application. Figure 5D Figure 6 shows the confusion matrix of the LSTM-1DCNN-Attention model in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical content of the present application will be described in detail below in combination with the drawings and specific embodiments.

[0020] The technical concept of the embodiments of the present application is to identify early glaucoma patients by using the eye movement and head movement characteristics of glaucoma patients in a virtual reality environment. In the early stage of glaucoma, although the central vision of the patient may not be impaired, their peripheral vision has already appeared abnormal. In order to compensate for the decline in visual function caused by this visual field damage, the patient will unconsciously exhibit different eye movement and head movement behaviors from normal people. Therefore, by analyzing these behavior characteristics in a specific virtual reality scene and task, early glaucoma patients can be distinguished because these characteristics are significantly different from the normal population. In short, the present application realizes the screening of early glaucoma by identifying the differences in eye movement and head movement behaviors between early glaucoma patients and normal people.

[0021] In an embodiment of the present application, we use the HTC VIVE Pro Eye VR headset (referred to as VR device) as the detection device. The software system is developed using the SteamVR plug-in, which enables the system to process input and output information related to the VR controller and load the 3D model of the VR controller. In order to collect eye movement data, we use the SRanipal software development kit (SDK), which can collect directional vector information of eye movement; and the head movement data is collected through the OpenVR API, which can obtain quaternion data of head movement. In addition, we have built a human motion experiment system based on Unity3D, which is specially designed to build a detection platform for glaucoma patients in order to analyze eye movement and head movement.

[0022] First embodiment The first embodiment of the present application provides a screening method for distinguishing early glaucoma patients, which at least comprises the following steps.

[0023] Step 11: The subject wears the VR device and calibrates the gaze point.

[0024] As mentioned above, the embodiment of the present application uses the HTC VIVE Pro Eye VR device as the detection device. The VR device includes a display module, an eye movement tracking device and a head movement tracking device. After the subject correctly wears the VR device, the face is directed forward, without lowering the head, tilting the head or turning the head. The operator adjusts the position of the VR device according to the comfort of the subject, ensures that the display screen of the VR device is within the line of sight of the subject, and the image is clear. Adjust the interpupillary distance (IPD) to ensure that the image is aligned with the eyes.

[0025] After wearing the VR device, enter the calibration interface and complete the calibration through a series of visual prompts. The visual prompts include a series of dots or target points on the display module, and the subject gazes at these points for calibration.

[0026] Optionally, the display module displays a calibration point (such as a dot or a cross mark), prompting the subject to gaze at the calibration point for a certain period of time (such as a few seconds). Then the calibration point moves to different positions (such as five to nine points), and the subject gazes at each point in turn.

[0027] The eye movement tracking device (such as Tobii eye movement tracker) detects the position of the subject's eyes and the movement of the eyeball through infrared sensors and cameras, captures the line of sight direction and gaze point of the subject. The eye movement tracking device tracks the eye movement data of the subject in real time, calculates the point in the display module position or three-dimensional space that the subject is gazing at, to determine whether the line of sight of the subject is in contact with the instance object.

[0028] IGazeFocusable is an interface defined in Tobii SDK, which allows us to handle gaze events related to eye tracking in the Unity environment. Using this interface, the operator can preset a series of behaviors that will be triggered when the subject's line of sight focuses on a certain object or leaves the object. In the IGazeFocusable interface, there is a core method called GazeFocusChanged (bool hasFocus), which is called internally by Tobii SDK. When the eye movement tracking device captures the subject's line of sight entering or leaving a certain object, the GazeFocusChanged method will be triggered to execute.

[0029] The GazeFocusChanged method accepts a Boolean parameter, hasFocus, which is provided by the Tobii SDK to indicate whether the current object is being gazed at by the subject. The value of hasFocus is updated in real-time as the subject's gaze changes. Specifically, when the subject's gaze is focused on an object that implements the IGazeFocusable interface, the GazeFocusChanged method is triggered and the hasFocus parameter is set to true, indicating that the object is being gazed at. Conversely, if the subject's gaze moves away from the object, the GazeFocusChanged method is also triggered, but this time the hasFocus parameter is set to false, indicating that the object is no longer being gazed at. This mechanism allows developers to perform corresponding operations based on the focus and movement of the gaze.

[0030] Step 12: Enter the pre-detection time, display the image of the pre-detection scene on the display module, collect the eye movement data and head movement data of the subject observing the image, and perform calibration.

[0031] As shown in Figures 1A-1D The pre-detection scene is set to a white or black panoramic background to test the differences in the performance of the subject under different brightness backgrounds. The image of the detection scene is a sphere surrounding the VR camera with a white or black panoramic background attached to the sphere, and the background displays indication information (such as an arrow map), with the arrow pointing to the central detection area to guide the subject's gaze to focus on the central detection area. The arrow map attribute is double-sided shading.

[0032] This step is a pre-detection step, which is a complete eye movement experiment of the eye movement scene (black panoramic background or white panoramic background). If the drift or accuracy of the gaze point is observed, the VR device re-performs eye movement calibration on the subject.

[0033] It is worth noting that the pre-detection time is used to familiarize the subject with the detection content and equipment, and to ensure that the subject correctly understands the task requirements. Therefore, the detection scene image displayed during the pre-detection time is similar and continuous with the subsequent detection images.

[0034] The collection of eye movement and head movement data is divided into a waiting phase and a detection phase. The waiting phase refers to the time between the end of the previous detection task and the start of the next detection task. When the VR device gives a start instruction, the subject gazes at the image in front of him and gives feedback (for example, presses a switch), and the detection phase starts. When the VR device gives an end instruction, the subject gazes at the exit icon and gives feedback, and the detection phase ends. When collecting eye movement and head movement data, the program records the start and end times of the waiting phase and the detection phase to distinguish between the two phases. The start and end times of the detection are determined by the line of sight and the feedback of the subject, and the detection process can be controlled only when both are true.

[0035] Optionally, during the detection process, the Instantiate() function generates target instance objects at random positions within the specified range. If the eye tracking device detects that the subject's line of sight contacts the instance object, it triggers the timing function. If the contact duration exceeds the set threshold, it is determined that the target object has been successfully eliminated.

[0036] Optionally, the Tobii eye tracker built into the VR device captures the line of sight through the Tobii XR SDK, and the prefab of the instance object is assigned the same size of the collider attribute. When the line of sight intersects with the instance object (i.e., when the gaze is on the instance object), the relevant instance object is marked with a red flag, and the background log records the start and end times of the gaze. When the gaze reaches the required duration and there is no interruption, it is considered a successful eye movement behavior.

[0037] Since head-eye coordination movement occurs during line of sight transfer, it is necessary to exclude the interference of head movement on eye movement detection in the design of the detection scene. The specific implementation is to make the target instance objects in the detection scene the child objects of the VR camera. The world coordinates of the child object are the sum of the world coordinates of the parent object and the coordinate vector of the child object in the parent object coordinate system. In other words, when using the VR device to test eye movement and visual field, all instance objects in the virtual reality environment are made child objects of the VR camera, so that the relative position of the eye position and the specified visual field range does not change, and external head movement and body movement do not affect the eye movement detection results.

[0038] It is worth noting that the instance objects provided by the embodiments of the present application are graphics in the detection scene that need to be gazed at by the subject. Although head movement data is still important in subsequent test steps, head movement interference needs to be excluded to obtain eye movement data, so head movement data is also collected.

[0039] Step 13: Enter the detection time, display the image of the detection scene on the display module, and collect the eye movement and head movement data of the subject when observing the image.

[0040] The detection scene is set to a white or black panoramic background. This step is completed twice in a random order - black panoramic background and white panoramic background. The random order means that the data can be collected first in the white panoramic background or first in the black panoramic background.

[0041] It is worth noting that if the white panoramic background is selected in step 12, step 13 needs to detect first in the black panoramic background and then in the white panoramic background. Conversely, if the black panoramic background is selected in step 12, step 13 needs to detect first in the white panoramic background and then in the black panoramic background.

[0042] In the black panoramic background and white panoramic background experiments, the subjects complete a plurality of eye movement tasks in turn. For example, 10° fixation (30s), 24° fixation (30s), 10° smooth pursuit (30s), 24° smooth pursuit (30s), visual matching (60s) five eye movement tasks in the black panoramic background and white panoramic background respectively, a total of ten eye movement tasks, as shown in Figure 2 The black panoramic background and the white panoramic background are displayed alternately.

[0043] The virtual reality eye movement detection is guided by eye movement tasks and designed according to the measurement principle of the perimeter. For example, the instance objects appear in a certain distance and field of view in front of the screen in turn, and the subjects need to complete the tasks of fixation, saccade, visual matching, etc. on the instance objects.

[0044] Taking Unity3D as an example, a plane is specified at z = 5, and the z coordinates of all instance objects are fixed to 5, all located on the plane. The VR camera is equivalent to the human eye, the plane is equivalent to the "screen", and the randomly appearing instance objects are equivalent to the "test light points" of the perimeter.

[0045] This step includes the following sub-steps.

[0046] Sub-step 131: Enter the fixation detection time, display the fixation detection content on the display module, and the subject fixates on the detection content. The fixation detection content is the instance objects in a static state within the 10° and 24° circular field of view.

[0047] The example object is exemplified by a small cube with a size of 0.2m x 0.2m x 0.01m. The visual field range includes a 10° and 24° circular visual field range. One example object randomly appears in the visual field range. If the fixation duration on the example object exceeds a set threshold (e.g., 1s), the example object is eliminated. After a certain duration (e.g., 1s), the next example object appears. If the fixation duration on the example object does not exceed the set threshold (e.g., 3s), the example object immediately disappears. After a certain duration (e.g., 1s), the next example object appears.

[0048] Sub-step 132: Enter the tracking detection time, display the tracking detection content on the display module, and the subject fixates on the tracking detection content. The tracking detection content is an example object in a moving state within a 10° and 24° circular visual field range.

[0049] The example object is exemplified by a small cube with a size of 0.2m x 0.2m x 0.01m. The visual field range includes a 10° and 24° circular visual field range. One example object randomly appears in the visual field range and uniformly translates in the visual field range at a speed of 0.8m / s. If the fixation duration on the example object exceeds a set threshold (e.g., 2s), the example object is eliminated. After a certain duration (e.g., 2s), the next example object appears. If the fixation duration on the example object does not exceed the set threshold (e.g., 5s), the example object immediately disappears. After a certain duration (e.g., 1s), the next example object appears.

[0050] It is worth noting that the implementation object of uniform translation while maintaining fixation is referred to as smooth tracking hereinafter.

[0051] Sub-step 133: Enter the matching detection time, display the matching detection content on the display module, and the subject fixates on the matching detection content. The matching detection content is a plurality of example objects in a stationary state within the visual field range.

[0052] The example object is exemplified by a plurality of mutually different graphics, for example, 8. The plurality of example objects are regularly arranged in the visual field range, for example, in a circular arrangement and a rectangular arrangement. Each time a randomly selected example object repeatedly appears in the central region of the example object arrangement. The subject finds the example object with the same graphic as the central example object among the surrounding example objects and maintains fixation for a certain duration (e.g., 1s). The example object in the central region is eliminated. After a certain duration (e.g., 2s), the next example object appears in the central region. If the fixation duration on the example object does not exceed the set threshold (e.g., 5s), the example object immediately disappears. After a certain duration (e.g., 1s), the next example object appears.

[0053] It is worth noting that finding the surrounding instance objects with the same pattern as the central instance object, hereinafter referred to as matching.

[0054] Step 14: Merge eye movement data and head movement data based on time points.

[0055] Because of the natural coordination mechanism of the human eye and head, head movement data also contains valuable information for detection. The role of head movement data in behavior analysis mainly reflects in the following aspects.

[0056] Gaze stability: Head movement affects the stability of eye movement, and analyzing the amplitude and frequency of head movement can help understand how the subject adjusts the gaze during viewing.

[0057] Attention shift: Even in the case of head stillness, head movement data still reflects subtle changes in the subject's visual attention, such as short head micro-movements indicating attention to a specific area.

[0058] Individual differences: Different subjects have different head movement patterns when viewing the same content, and this difference can distinguish between glaucoma patients and normal people.

[0059] Environmental adaptability: Head movement data can analyze how the subject adapts to visual information in a virtual environment, providing a more comprehensive understanding of behavior.

[0060] In summary, the value of head movement data in behavior analysis lies in supplementing eye movement data and providing a more comprehensive understanding of gaze and attention.

[0061] Since the VR device collects eye movement data and head movement data from devices with different technical principles, the sampling frequencies of the data are different, and the specific time labels of the two data cannot be one-to-one corresponding. It is necessary to merge the two parts of data based on time points to create a basis for subsequent processing.

[0062] For example, the timing data collected by the embodiment of the application includes eye movement and head movement data, and records experimental data at a set sampling frequency. The timing merging algorithm down-samples the data to match the lowest sampling frequency, ensuring that the two types of data are aligned with the smallest time difference. After data merging, check for missing values or outliers to ensure data quality.

[0063] As shown in Table 1, the timing merging algorithm steps are designed according to the eye movement-head movement timing merging of the embodiment of the application. Among them, the eye movement data and head movement data are input in csv files, and the merged timing data is output in csv files.

[0064] Table 1 Timing merging algorithm Specifically, this step includes the following sub-steps.

[0065] Sub-step 141: read the eye movement data and head movement data into DataFrame table type data structure.

[0066] Read the eye movement data (time series data 1) and head movement data (time series data 2) into DataFrame table type data structure.

[0067] Sub-step 142: store the difference between the timestamps of the eye movement data and the head movement data; store the minimum timestamp difference found, with an initial value of infinity; record the index of the last merged row in the head movement data, with an initial value of 0.

[0068] Initialize the timestamp difference time_difference, local minimum min_time to infinity, and the last merged row number memo = 0.

[0069] Sub-step 143: traverse each row of the eye movement data from the 0th row to the mth row, where m is a positive integer. Sub-step 144: traverse each row of the head movement data from the index-specified row to the ith row, where i is a positive integer.

[0070] Sub-step 145: calculate the timestamp difference of the current row. If this difference is less than the currently recorded minimum timestamp difference, update the minimum timestamp difference. Otherwise, merge the mth row of the eye movement data and the ith row of the head movement data. Reset the timestamp difference and the minimum timestamp difference, and update the index to the current row's index i.

[0071] Calculate the timestamp difference time_difference = abs(time_stamp_1[m] - time_stamp_2[i]) if time_difference < min_time: update min_time = time_difference else: merge the mth row of data 1 and the ith row of data 2, reset time_difference, min_time, and memo = i.

[0072] Sub-step 146: exit sub-step 144 and continue sub-step 143 until the eye movement data is traversed.

[0073] break loop 4, continue loop 3, until all rows of data 1 are traversed.

[0074] Sub-step 147: save the merged time series data and output as a csv file.

[0075] Save the merged DataFrame and output as a csv file.

[0076] Step 15: Substitute the merged eye movement data and head movement data obtained in the above steps into the glaucoma-normal person classification model for evaluation to determine whether the subject meets the characteristics of a glaucoma patient.

[0077] In an embodiment of the present application, step 13 comprises the following sub-steps.

[0078] Sub-step 134: Evaluate the eye movement data in the detection time to obtain task indicators.

[0079] When a key event of the eye movement experiment occurs, such as the line of sight contacting the correct instance object or the line of sight contacting the incorrect instance object, an event script of Unity is triggered immediately to record the event and the time point when the event occurs.

[0080] Task indicators refer to the goals or standards set in the eye movement detection task for evaluating the performance of participants when performing the task. The task indicators provided in this step include but are not limited to accuracy, effective recognition duration, recognition duration, fixation target duration, reaction time, and reaction correct time.

[0081] This sub-step comprises the following sub-steps.

[0082] Sub-step 1341: Evaluate the eye movement data in the fixation detection time to obtain fixation task indicators.

[0083] This sub-step comprises the following task indicators.

[0084] Fixation accuracy: the number of successfully eliminated instance objects divided by the total number of instance objects.

[0085] Fixation effective recognition duration: the total duration of correctly fixating instance objects to elimination within the experiment duration (e.g., 30s).

[0086] Fixation recognition duration: the total duration of fixating instance objects within the experiment duration (e.g., 30s). This includes the cumulative duration of fixating to instance objects but not maintaining continuous fixation.

[0087] Fixation reaction time: the duration from the appearance of an instance object to the movement of the line of sight onto the instance object.

[0088] Sub-step 1342: Evaluate the eye movement data in the tracking detection time to obtain tracking task indicators.

[0089] This sub-step comprises the following task indicators.

[0090] Tracking accuracy: the number of successfully eliminated instance objects divided by the total number of instance objects.

[0091] Tracking effective recognition duration: the total duration of smooth pursuit of instance objects to extinction within the experimental duration (e.g. 30s).

[0092] Tracking recognition duration: the total duration of smooth pursuit of instance objects within the experimental duration (e.g. 30s). This includes the cumulative duration of smooth pursuit to instance objects but without sustained fixation.

[0093] Tracking reaction time: the duration from the appearance of instance objects to the movement of gaze onto instance objects.

[0094] Sub-step 1343: evaluate the eye movement data of the matching detection time to obtain matching task indicators.

[0095] This sub-step includes the following task indicators.

[0096] Matching accuracy: the number of successfully matched instance objects divided by the total number of instance objects presented.

[0097] Matching fixation target duration: the total duration of fixation on instance objects (including correct and incorrect instance objects).

[0098] Matching reaction correct time: the duration from the appearance of instance objects to the movement of gaze onto correct instance objects.

[0099] Matching reaction time: the duration from the appearance of instance objects to the movement of gaze onto instance objects (including correct and incorrect instance objects).

[0100] Sub-step 135: compare the task indicators obtained in the above steps with the data of the baseline experiment to determine whether the subject meets the characteristics of glaucoma patients.

[0101] The task indicators obtained in sub-step 134 can quantitatively evaluate the eye movement characteristics of glaucoma patients under different lighting conditions and different tasks, such as fixation times and fixation times, and analyze and explain the impact of visual field defects of glaucoma patients on their eye movement behavior, providing a basis for subsequent personalized visual training for glaucoma patients and improving their quality of life.

[0102] The fixation task indicators can analyze the following characteristics.

[0103] The eye movement event indicators of the 10° fixation task are shown in Table 2. From the results, it can be seen that the normal control group performed better in accuracy, effective recognition duration, recognition duration, and reaction time than the glaucoma group, and the accuracy and effective recognition duration had significant differences at the level of p<0.05, indicating that the efficiency and accuracy of the eye health subjects in completing the task were higher; the monocular severe group performed relatively worst in the four indicators, which may be due to the fact that the difference between the two eyes of the monocular severe patients has a more serious impact on the visual search task.

[0104] Table 2 Eye movement event indicators: 10° fixation task The results of the 24° fixation task are shown in Table 3. There were significant differences in accuracy, effective recognition duration, and recognition duration between groups (p < 0.05), and the recognition duration was significantly different (p < 0.05); there were statistically significant differences in effective recognition duration and recognition duration between the experimental group and the control group as a whole (p < 0.05), and the reaction time was significantly different (p < 0.05). And relative to the 10° fixation task, the 24° fixation task showed stronger differences between groups, because the visual field defect of glaucoma patients generally starts from the periphery and gradually damages the center, and the loss of peripheral visual field is more serious than the central visual field. The 24° fixation range is larger than the 10° range, and includes more peripheral visual field range, so the differences between groups due to the degree of visual field defect are more obvious. From the result values, the control group performed best in the four scores, and the performance gradually deteriorated as the degree of visual field defect increased, with the double-eye severe group performing worst. This result is as expected.

[0105] Table 3 Eye movement event indicators: 24° fixation task The results of the 10° smooth pursuit task are shown in Table 4. The double-eye severe group had the worst performance in accuracy, effective recognition duration, recognition duration, and reaction time, and statistical calculations showed that the double-eye severe group and the other groups had significant differences in the above two indicators (p < 0.05). However, there was no statistical difference between groups and between the experimental group and the control group as a whole, indicating that the double-eye severe group had poor ability to maintain continuous pursuit of the visual line.

[0106] Table 4 Eye movement event indicators: 10° smooth pursuit task The results of the 24° smooth pursuit task are shown in Table 5. The results trend is consistent with the 10° smooth pursuit, with the double-eye severe group performing worst and the normal control group performing best. And because the 24° experimental range is larger, the performance gap between groups is more obvious. Specifically, for the same experimental task, there is no difference between groups in the 10° range, but when the range is increased to 24°, there are differences between groups in effective recognition duration (p < 0.1), and there are differences in recognition duration and reaction time between the experimental group and the control group as a whole (p < 0.1 and p < 0.05, respectively).

[0107] Table 5 Eye movement event indicators: 24° smooth pursuit task In addition to the data comparison of the above tasks as a whole, the inventors also analyzed the experimental results of the groups in the black and white backgrounds. For the 24° fixation task, the reaction time of the binocular moderate and below group in the white background was significantly faster than that in the black background, with a statistical difference at the level of p<0.1; for the 10° smooth pursuit task, the accuracy of the binocular severe group and the binocular moderate and below group in the white background was higher (p<0.1, p<0.05), and the reaction time of the three glaucoma experimental groups in the white background was faster (all p<0.1). In the control group, there was no statistical difference in the indicators of each experimental task between the black and white backgrounds.

[0108] From the overall results of the groups in the black and white backgrounds, glaucoma patients tend to perform better in the experiment in the white background with higher brightness, and this difference in background brightness cannot be observed in the normal subject group. The white background can provide more environmental light and contrast, and sufficient light can help glaucoma patients perceive the target of the visual task. Patients are more sensitive to light differences, while normal subjects have no visual field defects, and the change in brightness during the experiment will not affect the results.

[0109] The evaluation results of the visual matching task are listed in Table 6. From the results, the accuracy of the binocular severe group is the lowest, the binocular moderate group performs relatively poorly in the correct fixation target duration, fixation target duration, and reaction time, and the monocular severe group has the slowest reaction time. There is no statistical difference in the five evaluation indicators among the groups. Compared with the visual search task, the visual matching task is basically in the central visual field, and the target size is 2.5 times that of the search task target, so the discrimination degree for glaucoma patients is not obvious, indicating that the visual matching function in the central visual field of the patient is not significantly affected by the visual field defect. Statistical analysis shows that there is no statistical difference in the five test indicators in the black and white backgrounds.

[0110] Table 6 Eye movement event indicators: visual matching task In an embodiment of the present application, the glaucoma-normal person classification model is obtained by the following steps.

[0111] Step 21: Diagnose the eye health of the subject through medical examination.

[0112] The purpose of this step is to diagnose whether the subject is a healthy person or a glaucoma patient, so as to divide the training set for the obtained detection data.

[0113] Step 22: The subject wears a VR device and calibrates the fixation point.

[0114] This step is the same as step 11.

[0115] Step 23: Enter pre-detection time, display the image of the detection scene on the display module, collect the eye movement and head movement data of the subject when observing the image, and calibrate.

[0116] This step is the same as step 12.

[0117] Step 24: Enter detection time, display the image of the detection scene on the display module, and collect the eye movement and head movement data of the subject when observing the image.

[0118] This step is the same as step 13. It includes the following sub-steps.

[0119] Sub-step 241: Enter fixation detection time, display fixation detection content on the display module, and the subject fixates on the fixation detection content. The fixation detection content is an example object in a static state within a 10° and 24° circular field of view.

[0120] This step is the same as step 131.

[0121] Sub-step 242: Enter tracking detection time, display tracking detection content on the display module, and the subject fixates on the tracking detection content. The tracking detection content is an example object in a moving state within a 10° and 24° circular field of view.

[0122] This step is the same as step 132.

[0123] Sub-step 243: Enter matching detection time, display matching detection content on the display module, and the subject fixates on the matching detection content. The matching detection content is multiple example objects in a static state within the field of view.

[0124] This step is the same as step 133.

[0125] Step 25: Merge the eye movement data and head movement data based on the time point.

[0126] This step is the same as step 14.

[0127] Step 26: Obtain sufficient data of the diagnosed subjects to form a data set, and divide the training set and test set of the data set.

[0128] For example, two sets of data of the same subject in the same scene are processed in time sequence to obtain a merged csv file. The overall sampling frequency is 60 Hz, which is down-sampled to 20 Hz, and 25 rows are selected. Each 25-row time sequence data is a group, and the 9-column eye movement and 7-column head movement data are flattened into one row, with a total of 25 x (9 + 7) = 400 as the feature quantity of the deep learning data set. The file corresponds to the category of the subject, and 0 represents a normal subject and 1 represents a glaucoma patient. If the number of rows of the merged file is not a multiple of 25, the last group will not be included in the data set. The training set and the test set are divided according to a ratio of about 3:1. Finally, after merging, splitting, data enhancement, and data division, a training set with a size of 9148 x 400 and a test set with a size of 3208 x 400 are obtained. Among them, the proportion of category 0 in the training set is 44%, the proportion of category 1 is 30%, and the total proportion in the data set is 74%. The proportion of category 0 in the test set is 15%, the proportion of category 1 is 11%, and the total proportion in the data set is 26%. The distribution ratios of the training set, the test set, and the two categories of labels are shown in Table 1 as follows. Figure 3

[0129] Step 27: Based on the 1D CNN model and the training set, a glaucoma-normal person classification model is trained to evaluate the model with the test set.

[0130] Deep learning models, especially models like Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), have highly complex internal structures. They learn complex patterns and features from large amounts of input data, but these features are not always intuitive or easily interpretable. Although the models perform well and can make accurate predictions, it is not possible to directly extract explicit human-understandable features from the models. Therefore, this step includes the following sub-steps.

[0131] Sub-step 271: Based on the 1D CNN model and the training set, LSTM or Attention layers are added to the CNN to build 1D-CNN, LSTM+1D-CNN, 1D-CNN+Attention, and LSTM+1D-CNN+Attention models for training.

[0132] The head-eye movement module has 16 input channels and 128 output channels. The convolution kernel list is [3, 4, 5], which means that 3x1, 4x1, and 5x1 convolution kernels are used to perform convolution operations on the input one-dimensional data to capture local features of different sizes. A max pooling layer is applied with a pooling kernel size of 2.

[0133] ​The LSTM network is added before the 1D CNN module. The data first passes through the respective LSTM layer, with an input dimension of 16 and an output of 128; the CNN receives data from the LSTM for encoding, with an input and output of 128, and the convolution kernel and pooling layer settings remain unchanged.

[0134] The attention mechanism is added after the output of the CNN, with one attention head for each attention layer. The two attention layers automatically weight the input data during forward propagation and output the weighted features, which helps the model capture key information in the input data. The output features are input into the fully connected network. The fully connected network consists of two linear layers and an activation layer, with a dropout operation between the two linear layers to enhance network performance and reduce the likelihood of overfitting. The number of classes for classification is set to 2, and the final output is the corresponding classification result.

[0135] During training, five-fold cross-validation is used, with a total of 100 epochs, a learning rate of 0.001, a batch size of 32, and a cross-entropy loss function as the loss function.

[0136] Sub-step 272: Obtain model evaluation indicators based on the test set.

[0137] For a binary classification model, the common evaluation method is to classify the total samples according to the actual label and the predicted label into true positive, false positive, true negative, and false negative samples. Specifically, true positive (True Positive) indicates that the model successfully detects glaucoma patients, true negative (True Negative) indicates that the model correctly identifies normal people, false positive (False Positive) indicates misdiagnosed normal people, and false negative (False Negative) indicates missed glaucoma patients.

[0138] Accuracy, precision, recall, F1 score, ROC curve and AUC, confusion matrix, and other evaluation indicators are used, and the trend of loss reduction is visualized to evaluate the deep learning model from multiple aspects such as training process and test results to determine the performance and implementation effect of the model.

[0139] Accuracy (Accuracy) refers to the proportion of correctly classified samples to the total number of samples, described by formula (1).

[0140] (1) Precision (Precision) refers to the proportion of correctly predicted positive samples to all predicted positive samples, described by formula (2).

[0141] (2) Recall refers to the proportion of samples predicted as positive classes to actual positive classes, described as formula (3).

[0142] (3) F1 score is the harmonic mean of precision and recall, which describes the comprehensive performance of the model, described as formula (4).

[0143] (4) ROC curve describes the relationship between true positive rate and false positive rate, and AUC is the area under the ROC curve. The closer the AUC is to 1, the better the model performance. Confusion matrix shows the classification results of the model through the number of true positive, false positive, true negative and false negative samples.

[0144] Sub-step 273: Select the best model according to the evaluation index.

[0145] Get the loss function curve. After 100 iterations, get the loss function curve to describe the loss function trend of the training set and the test set. As shown in Figure 4A , the 1DCNN+Attention model has the fastest decline speed in the first 10 epochs. After about 10 epochs, the loss values of the four models all decrease to 0.1, and the convergence speed is very fast. At this time, the convergence speed of the 1DCNN+Attention model is the slowest. After 70 epochs, the loss values of the models all converge to about 0.001. As shown in Figure 4B , corresponding to the effect on the test set, the final loss function values of the four models decrease to below 0.4. The 1DCNN model has the best effect and the change is relatively stable, followed by LSTM+1DCNN+Attention, LSTM+1DCNN, and 1DCNN+Attention model.

[0146] Evaluation index results. The evaluation index of the deep learning model, accuracy, precision, recall, F1 score, and AUC results are shown in Table 7. From the size of the five indexes, 1DCNN and Attention layer have a greater improvement on the model. The LSTM-1DCNN-Attention combined model reaches an accuracy of 86.6%, which has the best accuracy, precision, recall and F1 score among all models, and has better comprehensive performance. The AUC value of the 1DCNN-Attention model is the highest. As shown in Figures 5A-5D , the results of the binary confusion matrix show that the designed deep learning model has the ability to distinguish between glaucoma patients and normal people.

[0147] Table 7 Performance evaluation index of deep learning model Figures 5A-5D The following information is shown by the binary classification confusion matrix of the 1DCNN, 1DCNN-Attention, LSTM-1DCNN, and LSTM-1DCNN-Attention models: ① True Positive, the number of samples correctly predicted by the model as positive class (actual positive, predicted positive) (lower right block); ② False Positive, the number of samples incorrectly predicted by the model as positive class (actual negative, predicted positive), also called "Type I error" or "false positive" (upper right block); ③ False Negative, the number of samples incorrectly predicted by the model as negative class (actual positive, predicted negative), also called "Type II error" or "false negative" (lower left block); ④ True Negative, the number of samples correctly predicted by the model as negative class (actual negative, predicted negative) (upper left block). Through this matrix, various evaluation indexes such as accuracy, precision, recall, F1 value, etc. can be calculated to comprehensively measure the performance of the model.

[0148] The baseline experimental results of the embodiments of the present application show that the severe glaucoma group performs worse in the visual search task, and the result distinction is more obvious in the 24° visual field than in the 10° visual field. In the visual matching task, the glaucoma patients have lower evaluation than normal people.

[0149] Second embodiment On the basis of the above-mentioned first embodiment, the second embodiment of the present application provides a screening device for distinguishing early glaucoma patients, which is used to realize the screening method provided by the above-mentioned embodiments. The screening device comprises a VR device and a control device, which communicate with each other in a wireless or wired manner.

[0150] Among them, the VR device includes a display module, a head movement sensor, an eye movement sensor, an audio module and a communication module. The display module includes a head-mounted display for displaying images to the user. The head movement sensor is used to track and record the movement and rotation data of the user's head. The eye movement sensor is used to track and record the line of sight data of the pupil. The audio module is used to play audio prompts. The communication module is used to realize data transmission.

[0151] The control device includes a computing module and a storage module. The computing module includes a calculator, such as a CPU, GPU, NPU, and a special chip (such as a mobile SoC), for providing data computing capability. The storage module includes a memory and a storage, for providing data storage capability. The computing module reads a computer program in the storage module, for executing the screening method for distinguishing early glaucoma patients.

[0152] It should be noted that the above embodiments are only illustrative. The technical solutions of various embodiments can be combined, and all are within the protection scope of the present application.

[0153] The terms "first" and "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0154] The screening method for distinguishing early glaucoma patients and the screening device provided by the present application are described in detail above. Any obvious modification made by a person skilled in the art without departing from the essential content of the present application will constitute an infringement of the patent right of the present application and will bear the corresponding legal responsibility.

Claims

1. A screening method for differentiating early-stage glaucoma patients, characterized in that... Includes the following steps: Step 11: Subjects wear VR devices and calibrate their gaze points; Step 12: Enter the pre-detection time, display the image of the pre-detection scene on the display module of the VR device, collect the eye movement data and head movement data of the subject when observing the image, and perform calibration; Step 13: Enter the detection time, display the image of the detection scene on the display module, and collect the eye movement data and head movement data of the subject when observing the image; Step 14: Merge the eye movement data and the head movement data based on the time point; Step 15: Substitute the merged data obtained in the above steps into the glaucoma-normal person classification model for evaluation to determine whether the subject meets the characteristics of a glaucoma patient.

2. The screening method as described in claim 1, characterized in that: In the detection scenario shown in step 13, all instance objects in the virtual reality environment are treated as sub-objects of the VR device camera, so that the relative position of the eye position and the specified field of view does not change.

3. The screening method as described in claim 2, characterized in that... Step 13 includes the following sub-steps: Sub-step 131: Enter the gaze detection time, the gaze detection content is displayed on the display module, the gaze detection content of the subject; the gaze detection content is an instance object in a stationary state within a circular field of view of 10° and 24° in a white and black panoramic background; Sub-step 132: Enter the tracking detection time, the tracking detection content is displayed on the display module, and the subject gazes at the tracking detection content; the tracking detection content is an instance object in motion within a 10° and 24° circular field of view in a white and black panoramic background; Sub-step 133: Enter the matching detection time, the matching detection content is displayed on the display module, and the subject gazes at the matching detection content; the matching detection content is multiple instance objects in a stationary state within the field of vision in a white and black panoramic background.

4. The screening method as described in claim 3, characterized in that: The panoramic backgrounds for steps 12 and 13 are white and black, with the white and black backgrounds alternating during the step transitions.

5. The screening method as described in claim 2, characterized in that... Step 14 includes the following sub-steps: Sub-step 141: Read the eye-tracking data and the head-tracking data into a DataFrame tabular data structure; Sub-step 142: Store the timestamp difference between the eye-tracking data and the head-tracking data; store the smallest timestamp difference found, with an initial value set to infinity; record the index of the last merged row in the head-tracking data, with an initial value of 0; Sub-step 143: Starting from line 0, traverse each line of the eye-tracking data until line m, where m is a positive integer; Sub-step 144: Traverse each row of the header data starting from the row specified by the index, up to the i-th row, where i is a positive integer; Sub-step 145: Calculate the timestamp difference for the current row; If the difference is less than the minimum timestamp difference of the current record, then update the minimum timestamp difference; otherwise, merge the m-th row of the eye-tracking data and the i-th row of the head-tracking data. Reset the timestamp difference and the minimum timestamp difference, and update the index to the index i of the current row; Sub-step 146: Exit sub-step 144 and continue with sub-step 143 until the eye-tracking data has been traversed. Sub-step 147: Save the merged time series data.

6. The screening method as described in claim 3, characterized in that... Step 13 includes the following sub-steps: Sub-step 134: Evaluate eye-tracking data during the detection time to obtain task metrics; Sub-step 135: Compare the task indicators with the baseline experimental data to determine whether the subject meets the characteristics of a glaucoma patient.

7. The screening method as described in claim 1, characterized in that... The method for obtaining the glaucoma-normal person classification model includes the following steps: Step 21: Confirm the subject's eye health status through medical examination; Step 22: The subject wears the VR device and calibrates the gaze point; Step 23: Enter the pre-detection time, display the image of the detection scene on the display module, collect the eye movement and head movement data of the subject when observing the image, and perform calibration; Step 24: Enter the detection time, display the image of the detection scene on the display module, and collect the eye movement and head movement data of the subject when observing the image; Step 25: Merge the eye movement data and the head movement data based on the time point; Step 26: Obtain sufficient data from confirmed subjects to form a dataset, and divide the dataset into a training set and a test set; Step 27: Based on the 1DCNN model and the training set, train and establish the glaucoma-normal person classification model, and evaluate the model using the test set.

8. The screening method as described in claim 1, characterized in that... Step 27 includes the following sub-steps: Sub-step 271: Based on the 1DCNN model and training set, add LSTM or Attention layers to the CNN to build 1D-CNN model, LSTM+1DCNN model, 1DCNN+Attention model and LSTM+1DCNN+Attention model and carry out training. Sub-step 272: Obtain model evaluation metrics based on the test set; Sub-step 273: Select the model with the best performance based on the evaluation indicators.

9. A screening device for distinguishing early-stage glaucoma patients, comprising a VR device and a control device, characterized in that: The VR device includes a display module, a head motion sensor, an eye movement sensor, an audio module, and a communication module; the display module is used to display images; the head motion sensor is used to acquire head motion data; the eye movement sensor is used to acquire eye movement data; the audio module is used to play audio prompts; and the communication module is used to implement data transmission. The control device includes a computing module and a storage module. The computing module reads the computer program in the storage module to execute the screening method for distinguishing early glaucoma patients as described in any one of claims 1 to 8.

Images