Eye movement information extraction method, device and electronic equipment

Abstract

The present application is applicable to the field of data processing technology, and provides an eye movement information extraction method, device and electronic device. The method comprises: obtaining a frame image and an event image of the subject's eye during an eye movement test; extracting geometric features from the frame image, and determining a gaze depth prediction value based on the extracted geometric features; extracting fusion features from the frame image and the event image based on the gaze depth prediction value to obtain a fusion feature; and obtaining eye movement information based on the fusion feature. The present application obtains frame images that can provide stable visual information, and event images that are more sensitive to rapidly changing eye movement information, and extracts gaze depth prediction values. Based on this, fusion features are extracted from the frame image and the event image, and accurate and valuable eye movement information can be effectively extracted from the eye movement data, thereby improving the accuracy of individual cognitive impairment detection.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and electronic device for extracting eye movement information. Background Art

[0002] Eye movements are coordinated and controlled by multiple brain regions, including the frontal eye movement area, parietal lobe, basal ganglia, thalamus, and brainstem, and are an important external representation of brain function. Therefore, eye movement data obtained through eye movement testing can reflect specific brain functions, revealing issues in individuals' cognition, emotions, and nervous system function, and can be used to detect cognitive impairment.

[0003] Eye movement testing generates a large amount of real-time data, including eye movement trajectories, gaze points, and other data. Data processing methods used in related technologies cannot fully adapt to the complex and changing nature of eye movement data, and cannot effectively extract valuable and accurate eye movement information from this large amount of data, thus affecting the accuracy of individual cognitive impairment assessments. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide an eye movement information extraction method, device and electronic device to solve the technical problem that related methods cannot effectively extract valuable and accurate eye movement information from a large amount of eye movement data, thereby affecting the accuracy of individual cognitive impairment assessment.

[0005] In a first aspect, an embodiment of the present application provides a method for extracting eye movement information, comprising:

[0006] Acquire frame images and event images of the subject's eyes during the eye movement test;

[0007] Extracting geometric features from the frame image, and determining a gaze depth prediction value based on the extracted geometric features;

[0008] Based on the gaze depth prediction value, performing fusion feature extraction on the frame image and the event image to obtain a fusion feature;

[0009] Eye movement information is obtained according to the fusion features.

[0010] In a possible implementation of the first aspect, the extracting fusion features from the frame image and the event image based on the gaze depth prediction value to obtain the fusion features includes:

[0011] performing eye movement feature extraction on the frame image to obtain a first eye movement feature vector;

[0012] performing eye movement feature extraction on the event image based on the gaze depth prediction value to obtain a second eye movement feature vector;

[0013] A fusion feature is obtained according to the first eye movement feature vector and the second eye movement feature vector.

[0014] In a possible implementation of the first aspect, extracting eye movement features from the event image based on the gaze depth prediction value to obtain a second eye movement feature vector includes:

[0015] Extracting a positive event point set and a negative event point set from the event image, and obtaining a temporal feature of the event through recursive filtering; wherein the positive event point set is determined based on pixel positions with increased brightness in the event image, and the negative event point set is determined based on pixel positions with decreased brightness in the event image;

[0016] Constructing a three-channel event frame according to the positive event point set, the negative event point set and the time feature;

[0017] A second eye movement feature vector is extracted from the three-channel event frame based on the gaze depth prediction value.

[0018] In a possible implementation of the first aspect, extracting a second eye movement feature vector from the three-channel event frame based on the gaze depth prediction value includes:

[0019] Dividing the three-channel event frame into different gaze areas based on the gaze depth prediction value;

[0020] Extract eye movement features from each gaze area to obtain corresponding second eye movement features;

[0021] A second eye movement feature vector is obtained according to the second eye movement feature corresponding to each gaze area.

[0022] In a possible implementation manner of the first aspect, the second eye movement feature vector includes a plurality of second sub-eye movement feature vectors;

[0023] The extracting eye movement features of the event image based on the gaze depth prediction value to obtain a second eye movement feature vector includes:

[0024] Dividing the event image into a plurality of regions; the plurality of regions include a pupil region, an upper eyelid region, and a light spot region;

[0025] For each region, the sub-positive event point set and the sub-negative event point set in the region are extracted, and the sub-time features of the events in the region are obtained through recursive filtering;

[0026] Constructing a three-channel event frame corresponding to the region according to the sub-positive event point set, the sub-negative event point set, and the sub-time feature;

[0027] Based on the gaze depth prediction value, a second sub-eye movement feature vector is extracted from the three-channel event frame corresponding to the region.

[0028] In a possible implementation of the first aspect, obtaining a fusion feature according to the first eye movement feature vector and the second eye movement feature vector includes:

[0029] Performing weighted fusion on the second sub-eye movement feature vectors to obtain a weighted fused feature vector;

[0030] The weighted fused feature vector is concatenated with the first eye movement feature vector to obtain a fused feature.

[0031] In a possible implementation of the first aspect, extracting geometric features from the frame image and determining a gaze depth prediction value based on the extracted geometric features includes:

[0032] Performing geometric feature extraction on the frame image to obtain geometric features of the pupil in the frame image;

[0033] A gaze depth prediction value is determined according to the geometric features of the pupil in the frame image and the geometric features of the subject's pupil.

[0034] In a possible implementation of the first aspect, obtaining eye movement information according to the fusion feature includes:

[0035] Eye movement information is obtained according to the fusion features and a preset eye movement model; wherein the eye movement model is obtained by training the fusion features of frame images and event images of different eyes and the corresponding eye movement information.

[0036] In a second aspect, an embodiment of the present application provides an eye movement information extraction device, comprising:

[0037] The acquisition module is used to acquire the frame images and event images of the subject's eyes during the eye movement test.

[0038] The determination module is used to extract geometric features from the frame image and determine a gaze depth prediction value based on the extracted geometric features.

[0039] An extraction module is used to extract fusion features from the frame image and the event image based on the gaze depth prediction value to obtain fusion features.

[0040] The obtaining module is used to obtain eye movement information according to the fusion features.

[0041] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the eye movement information extraction method as described in any one of the first aspects is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the eye movement information extraction method as described in any one of the first aspects.

[0043] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0044] The eye movement information extraction method, device, and electronic device provided in the embodiments of the present application obtain frame images that can provide stable visual information and event images that are more sensitive to rapidly changing eye movement information when the subject is undergoing an eye movement test, and then perform fusion feature extraction on the above two images. By combining the above two images and obtaining eye movement features from different angles, it is possible to more comprehensively and accurately reflect the actual situation of eye movement and obtain effective and accurate eye movement information. At the same time, the present application extracts a gaze depth prediction value, and based on this, extracts fusion features from the frame image and the event image. The gaze depth prediction value provides an additional information dimension for feature extraction, so that the extracted fusion features can better adapt to different eye movement scenarios and individual differences, further improving the effectiveness and pertinence of the features. In this way, the present application can effectively extract accurate and valuable eye movement information from eye movement data, thereby improving the accuracy of individual cognitive impairment detection.

[0045] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0048] Figure 2 1 is a flow chart of a method for extracting eye movement information provided in one embodiment of the present application;

[0049] Figure 31 is a schematic structural diagram of an eye movement information extraction device provided in one embodiment of the present application;

[0050] Figure 4 It is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0051] The present application will be described more clearly below with reference to specific embodiments. The following embodiments will help those skilled in the art further understand the function of the present application, but are not intended to limit the present application in any form. It should be noted that those skilled in the art may make a number of modifications and improvements without departing from the concept of the present application. These all fall within the scope of protection of the present application.

[0052] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0053] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0054] In the description of this application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0055] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0056] In addition, the “plurality” mentioned in the embodiments of the present application should be interpreted as two or more.

[0057] Eye movement data can reflect specific brain functions. Subjects with different cognitive and neurological functions may have different eye movement characteristics. The eye movement data obtained through eye movement testing can be used to evaluate the cognitive impairment of the subjects, provide data support for the detection and determination of cognitive impairment, and provide a basis for the formulation of subsequent personalized rehabilitation plans.

[0058] Eye movement tests include saccades, saccades, antisaccades, and visual search tasks. These tests track the subject's eye movements, fixation, and speed, generating large amounts of real-time data. However, related technologies are unable to fully adapt to the complex and ever-changing nature of eye movement data, making it difficult to effectively extract valuable and accurate information from this vast amount of data. This, in turn, impacts the accuracy of individual cognitive impairment assessments.

[0059] Based on the above-mentioned defects, this application obtains frame images that can provide stable visual information and event images that are more sensitive to rapidly changing eye movement information when the subject is undergoing an eye movement test, and then extracts fusion features from the above two images. By combining the above two images and obtaining eye movement features from different angles, it is possible to more comprehensively and accurately reflect the actual situation of eye movement and obtain effective and accurate eye movement information. At the same time, this application extracts the gaze depth prediction value, and based on this, extracts fusion features from the frame image and event image. The gaze depth prediction value provides an additional information dimension for feature extraction, so that the extracted fusion features can better adapt to different eye movement scenarios and individual differences, further improving the effectiveness and pertinence of the features.

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0061] First reference Figure 1 , Figure 1 The application scenario of the present application is schematically shown, which may include an eye tracker 11, a display screen 12, and an electronic device 400.

[0062] When conducting an eye movement test, the subject wears an eye tracker 11, and the display screen 12 displays the corresponding stimulus point and controls the movement of the stimulus point according to the set eye movement test content, such as a saccade task, a positive saccade task, an antisaccade task, and a visual search task, to guide the subject's gaze behavior. While the subject is looking at the stimulus point as required, the eye tracker 11 collects frame images and event images of the subject's eyes at a certain frequency. Taking the positive saccade task as an example, at the beginning of the eye movement test, a fixation point is displayed in the center of the display screen 12, and the subject needs to stare at the fixation point. After that, the fixation point disappears and the stimulus point appears. The subject needs to quickly look at the stimulus point. During this process, the eye tracker 11 collects frame images and event images of the subject's eyes at a certain frequency.

[0063] The electronic device 400 obtains the above-mentioned frame images and event images collected by the eye tracker 11, extracts geometric features of the frame images, and determines a gaze depth prediction value based on the extracted geometric features. Based on the gaze depth prediction value, the electronic device 400 extracts fusion features of the frame images and the event images to obtain fusion features. Then, eye movement information is obtained based on the fusion features.

[0064] Here, the eye tracker 11 has a high sampling rate (e.g., 1000 Hz), high precision (e.g., spatial resolution less than 0.006°, spatial accuracy 0.15°), and low latency (milliseconds), enabling real-time tracking and sampling of the subject's eyes. The eye tracker 11 includes at least two cameras, which can be near-infrared enhanced cameras. The higher-resolution camera serves as the event camera, capturing event images, while the lower-resolution camera serves as the frame image capture device. The electronic device 400 can be a hardware device with data storage, processing, and analysis capabilities. The electronic device 400 can also serve as the controller within the eye tracker 11.

[0065] The following combination Figure 1 ,refer to Figure 2 To describe the eye movement information extraction method provided according to an exemplary embodiment of the present application.

[0066] Figure 2 FIG. 1 is a flow chart of an eye movement information extraction method provided by an embodiment of the present application. Figure 2 As shown, the method in the embodiment of the present application may include:

[0067] Step S201: Acquire frame images and event images of the subject's eyes during the eye movement test.

[0068] As mentioned above, the contents of the eye movement test may include a saccade task, a positive saccade task, an antisaccade task, and a visual search task, etc. In this embodiment, there is no specific limitation on which one or more test contents are performed.

[0069] Frame images can include images of the subject's eyes, providing a static view of the subject's eyes and surrounding area at a specific moment. These images contain rich information such as texture and shape, and can be used to analyze the eye's overall state and relatively stable characteristics. Event images asynchronously record brightness changes caused by eye movement, including pixel brightness changes during eye movement. Event images are more sensitive to rapid eye movement and subtle changes, capturing details that might be missed in frame images.

[0070] Step S202: extract geometric features from the frame image, and determine a gaze depth prediction value based on the extracted geometric features.

[0071] For example, the frame image contains geometric features of the eye, such as the eye outline, pupil size, and position. Geometric features can be extracted using image processing algorithms. For example, in this embodiment, an edge detection algorithm can be used to determine the eye outline. Then, based on the principle of similar triangles or other geometric models, combined with relevant dimensional information of the camera that captured the frame image and relevant dimensional information of the subject's eyes, a gaze depth prediction value can be calculated.

[0072] Here, the gaze depth prediction value can reflect the depth of the subject's eye gaze point in space, providing an additional information dimension as a reference for subsequent fusion feature extraction, which helps to extract the motion characteristics of the eyes at different depths.

[0073] In some embodiments, when determining the gaze depth prediction value based on the extracted geometric features, geometric features of the frame image can be extracted to obtain the geometric features of the pupil in the frame image. Thereafter, the gaze depth prediction value is determined based on the geometric features of the pupil in the frame image and the geometric features of the subject's pupil.

[0074] For example, in this embodiment, image segmentation algorithms, such as threshold segmentation methods and semantic segmentation methods based on deep learning, can be used to accurately separate the pupil from the frame image. The geometric features of the pupil in the frame image, such as the pupil diameter, area, ellipticity, and / or pupil center coordinates, can then be calculated. These geometric features can directly reflect the changes in the position and size of the pupil in the eye. When a subject observes objects at different distances, the pupil size will undergo adaptive changes. Therefore, the change in the pupil is closely related to the gaze depth. Extracting the above geometric features can provide key data for the subsequent prediction of the gaze depth.

[0075] In this embodiment, based on the principle of similar triangles, according to the geometric characteristics of the pupil in the frame image, the geometric characteristics of the subject's pupil, and the focal length of the camera that captures the frame image, the distance from the subject's eyes to the camera can be obtained, and then the distance from the subject's eyes to the camera is used as an approximate value of the gaze depth, that is, a predicted value of the gaze depth is obtained.

[0076] Optionally, the diameter of the pupil in the frame image is taken as the geometric feature of the pupil in the frame image, and the diameter of the subject's pupil is taken as the geometric feature of the subject's pupil. Then, the expression of the gaze depth prediction value is:

[0077] dimage / dreal=f / Z

[0078] Where dimage is the diameter of the pupil in the frame image, dreal is the diameter of the subject's pupil, f is the focal length of the camera, that is, the distance from the camera's optical center to the image plane, and Z is the distance from the subject's eyes to the camera, which is the gaze depth prediction value.

[0079] Step S203: Based on the gaze depth prediction value, extract fusion features from the frame image and the event image to obtain fusion features.

[0080] In one possible implementation, when obtaining the fusion feature, this embodiment can perform eye movement feature extraction on the frame image to obtain a first eye movement feature vector, and based on the gaze depth prediction value, perform eye movement feature extraction on the event image to obtain a second eye movement feature vector. Thereafter, the fusion feature is obtained based on the first eye movement feature vector and the second eye movement feature vector.

[0081] In this embodiment, the eye movement features extracted from the frame images include appearance features and motion features. Image processing algorithms, such as edge detection and morphological operations, can be used to extract appearance features such as the eye contour, pupil size, and position. For example, in this embodiment, the Hough transform is used to detect the circular outline of the pupil, calculate the pupil radius and center coordinates, and threshold segmentation is used to determine the eye boundary, i.e., the eye contour, and thus obtain the eye's aspect ratio. These appearance features can reflect the basic shape and state of the eye and are important for analyzing the start and end positions of eye movements and the direction of eye rotation.

[0082] Optionally, this embodiment also compares changes in the eyes between adjacent frames to calculate motion characteristics such as eyeball speed and acceleration. For example, optical flow can be used to calculate the eyeball's motion vector on the frame image plane, and then, based on the magnitude and direction of this motion vector, the eyeball's motion speed, acceleration, and direction can be calculated. Furthermore, blinking can be detected based on the frame images, and the degree of eyelid opening and closing and blink frequency can be calculated as motion characteristics.

[0083] After the appearance features and motion features are extracted from the frame image, the appearance features and motion features are normalized, and the normalized features are combined into a vector to obtain a first eye movement feature vector.

[0084] In some embodiments, when obtaining the second eye movement feature vector, a set of positive event points and a set of negative event points can be extracted based on the event image, and the time characteristics of the event can be obtained through recursive filtering. Thereafter, a three-channel event frame is constructed based on the positive event point set, the negative event point set and the time characteristics, and the second eye movement feature vector is extracted from the three-channel event frame based on the gaze depth prediction value.

[0085] The positive event point set is determined based on the pixel positions where the brightness increases in the event image, and the negative event point set is determined based on the pixel positions where the brightness decreases in the event image.

[0086] For example, the event image records the brightness change events generated when the eye moves. When the brightness of a certain pixel position increases, a positive event is generated, and when the brightness of a certain pixel position decreases, a negative event is generated. By processing the event image, the pixel positions corresponding to all positive events can be extracted to form a positive event point set, and the pixel positions corresponding to all negative events can be extracted to form a negative event point set. The above-mentioned positive event point set and negative event point set reflect the specific position information of the scene brightness change during eye movement. For example, when the subject's eyes scan quickly, a large number of positive and negative events will be generated in the corresponding area. For example, when the subject's eyes scan quickly from left to right, the brightness of the corresponding pixel position on the left side of the event image decreases, while the brightness of the corresponding pixel position on the right side increases.

[0087] Optionally, when analyzing how events change over time, recursive filtering can be used to obtain the event's temporal characteristics. For example, the temporal characteristics of events can be calculated using a recursive filtering algorithm. This algorithm is a filtering method that adaptively tracks temporal changes. Through continuous iteration, events closer to the current moment have a greater impact on the results, thereby preserving the temporal order of events and better reflecting the dynamic process of eye movements.

[0088] Next, a three-channel event frame is constructed based on the set of positive event points, the set of negative event points, and temporal features. Specifically, the positive event channel displays the distribution of positive events, using pixel values to represent the distribution of positive events. Pixel locations where positive events occur are assigned higher values, while other locations are assigned lower values or even zero. This allows for intuitive observation of areas of increased brightness within the scene. Similarly, the negative event channel displays the distribution of negative events, using pixel values to represent the distribution of negative events. Pixel locations where negative events occur are assigned lower values, while other locations are assigned higher values. This allows for intuitive observation of areas of decreased brightness within the scene. The temporal feature channel records the temporal characteristics of events, using pixel values to reflect the temporal accumulation of events at different locations. For example, pixel values are set to higher values for later occurrences. The positive, negative, and temporal feature channels then form a three-channel event frame. This three-channel event frame integrates both spatial and temporal information about events, providing a rich data foundation for the subsequent extraction of more effective and representative eye movement features.

[0089] In some embodiments, when extracting the second eye movement feature vector from the three-channel event frame, the three-channel event frame can be divided into different gaze areas based on the gaze depth prediction value, and eye movement features can be extracted for each gaze area to obtain the corresponding second eye movement features. Then, the second eye movement feature vector is obtained based on the corresponding second eye movement features of each gaze area.

[0090] For example, according to the gaze depth prediction value, the three-channel frame can be divided into gaze areas with different gaze depths. Among them, different gaze depths may correspond to different eye movement patterns and areas of interest. For example, when the subject looks at a closer object, the micro-movement of the eye and the change of focus are more obvious. In this way, different gaze areas also correspond to different eye movement patterns and areas of interest due to different gaze depths. Therefore, the gaze depth prediction value provides another information dimension for the extraction of the second eye movement feature, so that the extracted second eye movement feature can better adapt to different eye movement scenes and individual differences, thereby improving the effectiveness and pertinence of the second eye movement feature.

[0091] Alternatively, corresponding second eye movement features are extracted from each gaze region, and a second eye movement feature vector is constructed based on all the second eye movement features. In this way, the second eye movement feature vector contains feature information of the event image at different regions and time scales, and can more comprehensively describe the characteristics and patterns of eye movements.

[0092] A fused feature can then be generated based on the first and second eye movement feature vectors. For example, the first and second eye movement feature vectors can be concatenated based on their timestamps to ultimately generate a fused feature. This allows for combining frame images, which provide stable visual information, with event images, which are more sensitive to rapidly changing eye movement information, to obtain eye movement features from different perspectives, ultimately yielding effective and accurate eye movement information.

[0093] Optionally, since the eye movement features in different regions have different physiological and behavioral meanings, in order to more effectively extract the second eye movement feature, the event image may be divided into regions, and eye movement features may be extracted for each region.

[0094] In some embodiments, the second eye movement feature vector includes multiple second sub-eye movement feature vectors. When obtaining the second eye movement feature vector, the event image can be first divided into multiple regions, which may include a pupil region, an upper eyelid region, and a light spot region. Then, for each region, the sub-positive event point set and the sub-negative event point set in the region are extracted, and the sub-time features of the events in the region are obtained through recursive filtering. According to the sub-positive event point set, the sub-negative event point set, and the sub-time features, the three-channel event frame corresponding to the region is constructed. Based on the gaze depth prediction value, the second sub-eye movement feature vector is extracted from the three-channel event frame corresponding to the region.

[0095] In this embodiment, the event image is divided into multiple regions, such as the pupil region, the upper eyelid region, and the light spot region, because the eye movement features in these different regions have different physiological and behavioral significance. For example, changes in pupil size and position directly reflect the direction of gaze and the degree of attention. In the event image, events in the pupil region are primarily related to eyeball rotation and pupil dilation. Upper eyelid movement, such as blinking, is an important eye movement feature. Changes in blink frequency can reflect the subject's fatigue and mental state. In the event image, events in the upper eyelid region are primarily related to eyelid opening and closing. Light spots are typically formed by reflected light from an external light source illuminating the eye. The light spot has a relatively stable positional relationship with the pupil. Tracking the positional changes of the light spot can assist in determining the position and movement trajectory of the pupil. In the event image, events in the light spot region are primarily related to the movement and brightness changes of the light spot. Therefore, dividing the event image into multiple regions with different physiological and behavioral significance and extracting eye movement features for each region separately can more effectively extract secondary eye movement features.

[0096] Exemplarily, in this embodiment, for each area in the pupil area, upper eyelid area and light spot area, the sub-positive event point set and the sub-negative event point set in the area are extracted, and the specific implementation process and principle of obtaining the sub-time characteristics of the events in the area through recursive filtering processing can refer to the steps of extracting the positive event point set and the negative event point set according to the event image, and obtaining the time characteristics of the event through recursive filtering processing in the aforementioned embodiment, which will not be repeated here.

[0097] Similarly, in this embodiment, for each area, a three-channel event frame corresponding to the area is constructed, and the specific implementation process and principle of extracting the second sub-eye movement feature vector from the three-channel event frame corresponding to the area based on the gaze depth prediction value can refer to the steps of constructing the three-channel event frame according to the positive event point set, the negative event point set and the time feature, and extracting the second eye movement feature vector from the three-channel event frame based on the gaze depth prediction value in the aforementioned embodiment, which will not be repeated here.

[0098] Exemplarily, when obtaining the fusion feature, each second sub-eye movement feature vector may be weightedly fused to obtain a weighted fused feature vector, and the weighted fused feature vector may be concatenated with the first eye movement feature vector to obtain the fusion feature.

[0099] In this embodiment, the weights corresponding to the pupil area, upper eyelid area, and light spot area can be set as needed. For example, if the light spot area is mainly used to assist in determining the position and movement trajectory of the pupil, the weight of the light spot area can be lower than the weights of the pupil area and upper eyelid area. In other words, the weight of the second sub-eye movement feature vector corresponding to the light spot area is lower than the weight of the second sub-eye movement feature vector corresponding to the pupil area and the weight of the second sub-eye movement feature vector corresponding to the upper eyelid area. Of course, the setting of the corresponding weights for each area can also be set according to other needs, and no specific restrictions are made here. Afterwards, the weighted fused feature vector is spliced with the first eye movement feature vector to obtain a fused feature.

[0100] In this way, the event image is first divided into regions to obtain multiple regions with eye movement features of different physiological and behavioral significance, and then eye movement features are extracted for each region, which can more effectively and accurately extract the second eye movement feature.

[0101] Step S204: Obtain eye movement information based on the fusion features.

[0102] For example, eye movement information can be obtained based on the fused features and a preset eye movement model. Here, the preset eye movement model can be based on a hybrid neural network architecture, with multimodal parallel processing capabilities to achieve real-time multi-task output. The eye movement model inputs the fused features and outputs eye movement information. Furthermore, the eye movement model is trained using the fused features of frame images and event images of different eyes, as well as the corresponding eye movement information.

[0103] Optionally, the eye movement information may include saccade amplitude, saccade duration, saccade latency, and eye acceleration, etc. It should be noted that the frame image and the event image are subjected to fusion feature extraction to obtain the fusion feature, and the eye movement information is obtained based on the fusion feature. After collecting multiple frame images and multiple event images, correspondingly, multiple eye movement information can be obtained. The same type of eye movement information can be averaged to prevent the influence of accidental anomalies caused by environmental factors on the eye movement information, thereby improving the accuracy of the eye movement information. The fusion feature obtained by combining the frame image that can provide stable visual information with the event image that is more sensitive to capturing rapidly changing eye movement information can more comprehensively and accurately reflect the actual situation of the eye movement. Based on the above fusion feature and the preset eye movement model, effective and accurate eye movement information can be obtained.

[0104] As mentioned previously, subjects with impaired cognitive and neurological function often exhibit characteristic abnormalities in eye movement tests, which are reflected in eye movement information, indicating that eye movement information can represent brain function. For example, prolonged saccade latency suggests possible functional degeneration or damage in the frontal lobe eye movement area, leading to slow saccade initiation. Reduced saccade amplitude suggests possible degeneration of dopaminergic neurons in the basal ganglia, resulting in predominantly short saccades.

[0105] After obtaining the eye movement information, each eye movement information can be compared with the corresponding judgment threshold. Then, the comparison results of each eye movement information can be combined to evaluate the subject's cognitive function and neurological function to determine whether the subject has cognitive impairment. For example, the normal range of saccade latency for healthy adults is 150-250ms. The judgment threshold corresponding to the saccade latency can be set to 250ms. When the subject's saccade latency is greater than this judgment threshold, it is confirmed that the subject may have cognitive impairment. It should be noted that the normal range of saccade latency for healthy adults mentioned above is only an example.

[0106] The eye movement information extraction method provided in the embodiment of the present application obtains frame images and event images of the subject's eyes during an eye movement test, extracts geometric features from the frame images, determines a gaze depth prediction value based on the extracted geometric features, and based on the gaze depth prediction value, extracts fusion features from the frame images and event images to obtain fusion features, and then obtains eye movement information based on the fusion features.

[0107] When the subject is undergoing an eye movement test, the present application obtains a frame image that can provide stable visual information, and an event image that is more sensitive to rapidly changing eye movement information, and then extracts fusion features from the above two images. By combining the above two images and obtaining eye movement features from different angles, it is possible to more comprehensively and accurately reflect the true situation of eye movement and obtain effective and accurate eye movement information. At the same time, the present application extracts a gaze depth prediction value, and based on this, extracts fusion features from the frame image and the event image. The gaze depth prediction value provides an additional information dimension for feature extraction, so that the extracted fusion features can better adapt to different eye movement scenarios and individual differences, further improving the effectiveness and pertinence of the features. In this way, the present application can effectively extract accurate and valuable eye movement information from the eye movement data, thereby improving the accuracy of individual cognitive impairment detection.

[0108] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0109] Figure 3 Schematic diagram of the structure of the eye movement information extraction device provided by an embodiment of the present application. Figure 3 As shown, the eye movement information extraction device provided by this embodiment may include: an acquisition module 301 , a determination module 302 , an extraction module 303 and a obtaining module 304 .

[0110] The acquisition module 301 is used to acquire the frame images and event images of the subject's eyes during the eye movement test.

[0111] The determination module 302 is configured to extract geometric features from the frame image and determine a gaze depth prediction value based on the extracted geometric features.

[0112] The extraction module 303 is configured to extract fusion features from the frame image and the event image based on the gaze depth prediction value to obtain fusion features.

[0113] The obtaining module 304 is configured to obtain eye movement information based on the fusion features.

[0114] Optionally, the extraction module 303 is further configured to:

[0115] performing eye movement feature extraction on the frame image to obtain a first eye movement feature vector;

[0116] performing eye movement feature extraction on the event image based on the gaze depth prediction value to obtain a second eye movement feature vector;

[0117] A fusion feature is obtained according to the first eye movement feature vector and the second eye movement feature vector.

[0118] Optionally, the extraction module 303 is further configured to:

[0119] Extracting a positive event point set and a negative event point set from the event image, and obtaining a temporal feature of the event through recursive filtering; wherein the positive event point set is determined based on pixel positions with increased brightness in the event image, and the negative event point set is determined based on pixel positions with decreased brightness in the event image;

[0120] Constructing a three-channel event frame according to the positive event point set, the negative event point set and the time feature;

[0121] A second eye movement feature vector is extracted from the three-channel event frame based on the gaze depth prediction value.

[0122] Optionally, the extraction module 303 is further configured to:

[0123] Dividing the three-channel event frame into different gaze areas based on the gaze depth prediction value;

[0124] Extract eye movement features from each gaze area to obtain corresponding second eye movement features;

[0125] A second eye movement feature vector is obtained according to the second eye movement feature corresponding to each gaze area.

[0126] Optionally, the second eye movement feature vector includes a plurality of second sub-eye movement feature vectors; the extraction module 303 is further configured to:

[0127] Dividing the event image into a plurality of regions; the plurality of regions include a pupil region, an upper eyelid region, and a light spot region;

[0128] For each region, the sub-positive event point set and the sub-negative event point set in the region are extracted, and the sub-time features of the events in the region are obtained through recursive filtering;

[0129] Constructing a three-channel event frame corresponding to the region according to the sub-positive event point set, the sub-negative event point set, and the sub-time feature;

[0130] Based on the gaze depth prediction value, a second sub-eye movement feature vector is extracted from the three-channel event frame corresponding to the region.

[0131] Optionally, the extraction module 303 is further configured to:

[0132] Performing weighted fusion on the second sub-eye movement feature vectors to obtain a weighted fused feature vector;

[0133] The weighted fused feature vector is concatenated with the first eye movement feature vector to obtain a fused feature.

[0134] Optionally, the determining module 302 is further configured to:

[0135] Performing geometric feature extraction on the frame image to obtain geometric features of the pupil in the frame image;

[0136] A gaze depth prediction value is determined according to the geometric features of the pupil in the frame image and the geometric features of the subject's pupil.

[0137] Optionally, the obtaining module 304 is further configured to:

[0138] Eye movement information is obtained according to the fusion features and a preset eye movement model; wherein the eye movement model is obtained by training the fusion features of frame images and event images of different eyes and the corresponding eye movement information.

[0139] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0140] Figure 4 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 4 As shown, the electronic device 400 of this embodiment includes: a processor 410 and a memory 420, wherein the memory 420 stores a computer program 421 that can be run on the processor 410. When the processor 410 executes the computer program 421, the steps of any of the above-mentioned method embodiments are implemented, such as Figure 2Alternatively, when the processor 410 executes the computer program 421, the functions of the modules / units in the above-mentioned device embodiments are realized, for example Figure 3 Functions of modules 301 to 304 are shown.

[0141] For example, computer program 421 may be divided into one or more modules / units, one or more of which are stored in memory 420 and executed by processor 410 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of computer program 421 in electronic device 400.

[0142] Those skilled in the art will understand that Figure 4 These are merely examples of electronic devices and do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, buses, etc.

[0143] The processor 410 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0144] Memory 420 can be an internal storage unit of the electronic device, such as a hard drive or memory, or an external storage device, such as a plug-in hard drive, a SmartMedia Card (SMC), a Secure Digital (SD) card, or a flash memory card. Memory 420 can also include both an internal storage unit and an external storage device. Memory 420 is used to store computer programs and other programs and data required by the electronic device. Memory 420 can also be used to temporarily store data that has been output or is about to be output.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0146] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0147] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0148] In the embodiments provided by the present invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0151] If the integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0152] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for extracting eye movement information, characterized in that: include: Acquire frame images and event images of the subject's eyes during the eye movement test; Extracting geometric features from the frame image, and determining a gaze depth prediction value based on the extracted geometric features; Based on the gaze depth prediction value, performing fusion feature extraction on the frame image and the event image to obtain a fusion feature; obtaining eye movement information according to the fusion features; The step of extracting fusion features from the frame image and the event image based on the gaze depth prediction value to obtain fusion features includes: performing eye movement feature extraction on the frame image to obtain a first eye movement feature vector; performing eye movement feature extraction on the event image based on the gaze depth prediction value to obtain a second eye movement feature vector; A fusion feature is obtained according to the first eye movement feature vector and the second eye movement feature vector.

2. The eye movement information extraction method according to claim 1, characterized in that The extracting eye movement features of the event image based on the gaze depth prediction value to obtain a second eye movement feature vector includes: Extracting a positive event point set and a negative event point set from the event image, and obtaining a temporal feature of the event through recursive filtering; wherein the positive event point set is determined based on pixel positions with increased brightness in the event image, and the negative event point set is determined based on pixel positions with decreased brightness in the event image; Constructing a three-channel event frame according to the positive event point set, the negative event point set and the time feature; A second eye movement feature vector is extracted from the three-channel event frame based on the gaze depth prediction value.

3. The eye movement information extraction method according to claim 2, characterized in that: The extracting a second eye movement feature vector from the three-channel event frame based on the gaze depth prediction value includes: Dividing the three-channel event frame into different gaze areas based on the gaze depth prediction value; Extract eye movement features from each gaze area to obtain corresponding second eye movement features; A second eye movement feature vector is obtained according to the second eye movement feature corresponding to each gaze area.

4. The eye movement information extraction method according to claim 1, characterized in that The second eye movement feature vector includes a plurality of second sub-eye movement feature vectors; The extracting eye movement features of the event image based on the gaze depth prediction value to obtain a second eye movement feature vector includes: Dividing the event image into a plurality of regions; the plurality of regions include a pupil region, an upper eyelid region, and a light spot region; For each region, the sub-positive event point set and the sub-negative event point set in the region are extracted, and the sub-time features of the events in the region are obtained through recursive filtering; Constructing a three-channel event frame corresponding to the region according to the sub-positive event point set, the sub-negative event point set, and the sub-time feature; Based on the gaze depth prediction value, a second sub-eye movement feature vector is extracted from the three-channel event frame corresponding to the region.

5. The eye movement information extraction method according to claim 4, characterized in that: The obtaining of a fusion feature according to the first eye movement feature vector and the second eye movement feature vector includes: Performing weighted fusion on the second sub-eye movement feature vectors to obtain a weighted fused feature vector; The weighted fused feature vector is concatenated with the first eye movement feature vector to obtain a fused feature.

6. The eye movement information extraction method according to any one of claims 1 to 5, characterized in that: The step of extracting geometric features from the frame image and determining a gaze depth prediction value based on the extracted geometric features includes: Performing geometric feature extraction on the frame image to obtain geometric features of the pupil in the frame image; A gaze depth prediction value is determined according to the geometric features of the pupil in the frame image and the geometric features of the subject's pupil.

7. The eye movement information extraction method according to any one of claims 1 to 5, characterized in that: Obtaining eye movement information according to the fusion features includes: Eye movement information is obtained according to the fusion features and a preset eye movement model; wherein the eye movement model is obtained by training the fusion features of frame images and event images of different eyes and the corresponding eye movement information.

8. An eye movement information extraction device, characterized in that: include: An acquisition module, used for acquiring frame images and event images of the subject's eyes during the eye movement test; A determination module, configured to extract geometric features from the frame image and determine a gaze depth prediction value based on the extracted geometric features; an extraction module, configured to extract fusion features from the frame image and the event image based on the gaze depth prediction value to obtain fusion features; An obtaining module, configured to obtain eye movement information based on the fusion features; The extraction module is further used to extract eye movement features from the frame image to obtain a first eye movement feature vector; based on the gaze depth prediction value, extract eye movement features from the event image to obtain a second eye movement feature vector; and obtain a fusion feature based on the first eye movement feature vector and the second eye movement feature vector.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the eye movement information extraction method according to any one of claims 1 to 7 is implemented.

Images