Eye movement tracking method based on contact lenses and contact lenses

Abstract

The invention discloses an eye movement tracking method based on a contact lens and the contact lens, and relates to the technical field of eye movement tracking, the contact lens comprises a lens main body, a sensor integration module, an energy supply module, a wireless communication module and a packaging protection layer; the sensor integration module is arranged in a peripheral area of the lens main body or a specific layered structure; the energy supply module is connected with the sensor integration module and provides required electric energy for the sensor integration module; the wireless communication module is responsible for performing two-way data transmission with external equipment, sending the collected eye movement data to the external equipment for data processing, and receiving instruction information from the external equipment; and the packaging protection layer covers the outer layer of the lens main body. According to the invention, various motion characteristics of the eyeball can be accurately obtained, and compared with the eye movement tracking of some existing single sensors or simple equipment, the accuracy is obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of eye tracking, and more particularly to an eye tracking method based on contact lenses and a contact lens. Background Art

[0002] At present, with the continuous development of human-computer interaction, visual science research, medical diagnosis and other fields, the demand for accurate and convenient eye tracking technology has become increasingly prominent. Traditional eye tracking methods, such as desktop eye trackers and head-mounted eye tracking devices, rely on external cameras and are difficult to assemble on smart contact lenses. They are difficult to apply in scenarios where there are external objects blocking the view, and the scope of applicable scenarios has been reduced. Algorithms based on motion images have large computational complexity and low accuracy; and there are also many problems such as inconvenience in use, poor wearing comfort, and susceptibility to external environmental interference. However, there is still a lack of mature and effective technologies in this area on the market.

[0003] Therefore, how to provide an eye tracking solution that is more in line with actual usage scenarios, has high precision and good user experience is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, the present invention provides an eye tracking method based on contact lenses and contact lenses to solve the problems existing in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A contact lens, comprising: a lens body, a sensor integration module, an energy supply module, a wireless communication module, and a packaging protection layer;

[0007] The sensor integrated module is arranged in the peripheral area of ​​the lens body or in a specific layered structure;

[0008] The energy supply module is connected to the sensor integration module to provide it with the required electrical energy;

[0009] The wireless communication module is responsible for bidirectional data transmission with external devices, sending the collected eye movement data to the external devices for data processing, and also receiving command information from the external devices;

[0010] The packaging protection layer covers the outer layer of the lens body.

[0011] Optionally, the sensor integrated module includes a plurality of sensor arrays, and the sensor arrays are distributed on the lens body in a specific shape, and the specific shape is one of a ring, a square and an X shape.

[0012] Optionally, the energy supply module uses a micro thin-film battery to provide electrical energy.

[0013] Optionally, when the external device needs to recalibrate or adjust parameters of the sensor integrated module on the contact lens, a corresponding control instruction is sent to the contact lens through the wireless communication module, and the contact lens performs a corresponding operation after receiving the control instruction.

[0014] An eye tracking method based on contact lenses, comprising:

[0015] Use sensor integrated modules to obtain data related to eye movements from multiple dimensions;

[0016] After connecting with external devices through the wireless communication module, each sensor in the sensor integration module is initialized and calibrated;

[0017] When the user puts on the contact lenses, as the eyeballs move during various activities, various sensors begin to collect relevant data in real time;

[0018] The collected relevant data is periodically transmitted to an external device via a wireless communication module;

[0019] On the external device side, after receiving the data transmitted from the contact lens, it first extracts its features; then the data from different sensors are fused; finally, the fused data is recognized using a neural network model to obtain the eye movement behavior recognition results.

[0020] Optionally, a micro storage unit built into the contact lens is also included for temporarily storing data collected by the sensor. The storage unit can cope with the generation of a large amount of data in a short period of time and avoid data loss.

[0021] Optionally, the sensor integration module integrates a sensor array consisting of a micro-gyroscope and a micro-accelerometer to obtain data related to eye movement from multiple dimensions; the micro-gyroscope can accurately sense the changes in angular velocity in all directions when the eye moves; the micro-accelerometer focuses on capturing the linear acceleration during eye movement.

[0022] Optionally, the external device includes:

[0023] Establish an eye tracking recognition model, extract the overall features of the eye movement data at multiple scales from the data set transmitted by the contact lens, and use the attention mechanism to enhance the effective local features, and fuse the overall features with the local features;

[0024] Using the data set transmitted by the contact lens to train and optimize the eye tracking recognition model;

[0025] The trained eye tracking recognition model is used to identify the data transmitted by the contact lenses and predict the corresponding eye movement behavior.

[0026] Optionally, the establishing of the eye tracking recognition model specifically includes:

[0027] Establishing an eye tracking recognition model, wherein the eye tracking recognition model includes a global feature extraction network and a local feature extraction network;

[0028] The global feature extraction network is used to extract the overall features of the eye movement data in the eye movement dataset at multiple scales, and the local area data is obtained by dividing the local area according to different behavior stages of the eye movement, and the local area data is input into the local feature extraction network, and the effective local features are strengthened by using the attention mechanism;

[0029] Adding the upsampled features of the second basic convolution module of the local feature extraction network to the second basic convolution module of the global feature extraction network; adding the upsampled features of the first attention module of the local feature extraction network to the first multi-scale feature extraction module of the global feature extraction network; adding the upsampled features of the second attention module of the local feature extraction network to the second multi-scale feature extraction module of the global feature extraction network; adding the upsampled features of the second attention module of the local feature extraction network to the third multi-scale feature extraction module of the global feature extraction network;

[0030] The fused features are passed through global average pooling and a fully connected layer, and the eye movement behavior recognition results are output through softmax.

[0031] Optionally, using the data set transmitted by the contact lens to train and optimize the eye tracking recognition model includes:

[0032] Design the loss function and use Loss to express it as:

[0033] Loss = -α c (1-p c ) β log(p c )

[0034] where α c represents the weight of the corresponding eye movement behavior category, p c represents the probability of predicting the cth class, and β represents the hyperparameter;

[0035] The eye tracking recognition model is trained using the data set transmitted by the contact lens, and the parameters of the eye tracking recognition model are set, and the eye tracking recognition model is optimized using an optimizer.

[0036] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an eye tracking method based on contact lenses and contact lenses, which have the following beneficial effects:

[0037] 1. Compared with traditional head-mounted eye tracking devices, the contact lens-based design hardly brings any additional wearing burden to the user. It can be worn naturally in various scenarios such as daily activities, work, and study without affecting normal appearance and movement.

[0038] 2. High-precision eye tracking: Through the integration of multiple types of sensors and advanced data processing algorithms, various movement characteristics of the eyeball can be accurately acquired, which significantly improves the accuracy compared to the existing eye tracking with a single sensor or simple equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] Figure 1 A schematic diagram of the structure of a contact lens provided by the present invention;

[0041] Figure 2 This is a flow chart of the contact lens-based eye tracking method provided by the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The embodiment of the present invention discloses a contact lens, such as Figure 1 As shown, it includes: a lens body, a sensor integration module, an energy supply module, a wireless communication module, and a packaging protection layer;

[0044] The sensor integrated module is arranged in the peripheral area of ​​the lens body or in a specific layered structure;

[0045] The energy supply module is connected to the sensor integration module to provide it with the required electrical energy;

[0046] The wireless communication module is responsible for bidirectional data transmission with external devices, sending the collected eye movement data to external devices for data processing, and can also receive command information from external devices;

[0047] The encapsulated protective layer covers the outer layer of the lens body.

[0048] The main body of the lens is made of soft or hard materials with high oxygen permeability and good biocompatibility, ensuring that the user's eyes can breathe normally during long-term wear without causing adverse reactions such as allergies. The optical performance of the lens has been optimized, and while achieving the eye tracking function, it will not cause obvious interference to normal vision. Its parameters such as diopter can be customized according to the vision of different users.

[0049] In a specific embodiment, the sensor integration module includes a plurality of sensor arrays, and the sensor arrays are distributed in a specific shape on the lens body, and the specific shape is one of a ring, a square and an X shape. In the peripheral area of ​​the lens (such as a position close to the edge but not affecting the line of sight) or in a specific layered structure, the various micro sensors mentioned above are integrated through micro-nano manufacturing processes. Specifically, a micro gyroscope can detect the angular velocity changes of the eyeball in different directions, and a micro accelerometer can capture the linear acceleration of the eyeball during movement; it is necessary to minimize the impact on the overall thickness and flexibility of the lens to ensure that the contact lens can still fit the surface of the eyeball comfortably.

[0050] In a specific embodiment, the energy supply module uses a micro thin-film battery to provide power, which is the key to ensure that the eye tracking function can operate stably for a long time. Micro thin-film batteries have the characteristics of small size and relatively high energy density. They can store a certain amount of power in a limited space to meet the energy consumption requirements of operations such as sensor data collection and wireless communication module data transmission.

[0051] In a specific embodiment, when the external device needs to recalibrate or adjust the parameters of the sensor integrated module on the contact lens, the corresponding control instruction is sent to the contact lens through the wireless communication module, and the contact lens performs the corresponding operation after receiving the control instruction. The power management unit of the energy supply module will monitor the power status in real time, and send a reminder signal to the external device when the power is low, prompting the user to charge or replace the battery in time. This function can avoid the sudden interruption of the eye tracking system due to power exhaustion, resulting in data loss or affecting the ongoing application (such as the sudden loss of eye tracking function during human-computer interaction will affect the operating experience). Through timely reminders, users can prepare in advance and arrange the charging or battery replacement time reasonably to ensure that the eye tracking function of the contact lens is always in an available state, ensuring the reliability and continuity of the entire eye tracking process.

[0052] In order to protect the internal sensors, circuits, energy supply modules and other components from being corroded by tears, external impurities, etc., an ultra-thin encapsulation protective layer is applied to the outer layer of the entire lens with integrated functional components. Tears contain components such as salt and protein, and long-term contact may corrode circuits and damage precision components such as sensors; if impurities such as dust and microorganisms in the external environment enter the contact lens, they will also damage its function. The encapsulation protective layer is like a "protective cover" that isolates these adverse factors from the outside, ensuring that the various functional components inside can work normally in a relatively stable and clean environment, thereby extending the service life of the contact lens eye tracking system.

[0053] A contact lens-based eye tracking method, such as Figure 2 As shown, including:

[0054] Use sensor integrated modules to obtain data related to eye movements from multiple dimensions;

[0055] After connecting with external devices through the wireless communication module, each sensor in the sensor integration module is initialized and calibrated;

[0056] When the user puts on the contact lenses, as the eyeballs move during various activities, various sensors begin to collect relevant data in real time;

[0057] The collected relevant data is periodically transmitted to an external device via a wireless communication module;

[0058] On the external device side, after receiving the data transmitted from the contact lens, it first extracts its features; then the data from different sensors are fused; finally, the fused data is recognized using a neural network model to obtain the eye movement behavior recognition results.

[0059] In a specific embodiment, the contact lens also includes a micro storage unit built into the contact lens for temporarily storing the data collected by the sensor. The storage unit can cope with the large amount of data generated in a short period of time to avoid data loss. For example, when the user is in a scene with frequent rapid eye movements (such as watching fast-moving pictures), a large amount of sensor data will be generated in a short period of time. The storage unit can first save this data in order and transmit it at the right time, thereby ensuring the integrity and continuity of the eye movement data.

[0060] In a specific embodiment, the sensor integration module integrates a sensor array composed of a micro gyroscope and a micro accelerometer to obtain data related to eye movement from multiple dimensions; the micro gyroscope can accurately sense the angular velocity changes in all directions when the eyeball rotates, which is crucial for determining the speed, direction, and smoothness of the eyeball rotation. For example, when the eyeball scans to the left, the gyroscope will detect the angular velocity value changes in the corresponding direction, thereby providing basic data for the subsequent analysis of the specific movement trajectory of the eyeball. The micro accelerometer focuses on capturing the linear acceleration of the eyeball during movement. For example, when the eyeball quickly focuses on a distant object, it will produce corresponding linear acceleration movement. These subtle acceleration changes can be recorded by the accelerometer to assist in determining the start and end states of the eyeball movement and the acceleration and deceleration stages during the movement. After connecting to an external calibration device through a wireless communication module (such as Bluetooth, etc.), each sensor is initialized and calibrated to ensure that each sensor is in an accurate reference state when it starts working. Because different sensors may have certain initial deviations during the production, installation, and integration processes, the calibration operation can correct these deviations. For example, when calibrating a micro-gyroscope, its zero position and initial angular velocity reference value are set so that the subsequently measured angular velocity data are relative to this accurate reference, avoiding large errors in the entire eye movement data due to initial inaccuracies; for a micro-accelerometer, calibration can determine the gravitational acceleration reference value for each axis, so that when subsequently detecting the acceleration changes caused by eye movement, it can be accurately distinguished from the gravitational acceleration, thereby improving the accuracy of the measurement.

[0061] In a specific embodiment, the external device specifically includes:

[0062] Establish an eye tracking recognition model, extract the overall features of the eye movement data at multiple scales from the contact lens transmission data set, and use the attention mechanism to enhance the effective local features and fuse the overall features with the local features;

[0063] Use the data set transmitted by contact lenses to train and optimize the eye tracking recognition model;

[0064] The trained eye tracking recognition model is used to identify the data transmitted by the contact lenses and predict the corresponding eye movement behavior.

[0065] In a specific embodiment, establishing an eye tracking recognition model specifically includes:

[0066] Establish an eye tracking recognition model, which includes a global feature extraction network and a local feature extraction network;

[0067] Specifically, the global feature extraction network is sequentially connected by two basic convolution modules, three multi-scale feature extraction modules, global average pooling and a fully connected classification layer. Among them, the basic convolution module includes a convolution layer, a batch normalization layer (BatchNormalization, BN) and an activation function layer, and the filter size of the convolution layer is 3×3. The multi-scale feature extraction module includes a basic residual module and a multi-scale feature module. The basic residual module uses the residual module in ResNet18 and fuses 1×1 convolution features. The multi-scale feature module performs dimensionality-upgrading processing on the overall features through 1×1 convolution, and averagely groups the multi-channel features after dimensionality-upgrading. The features of each group are sequentially fused with the features extracted by the previous group using 3×3 depth-separable convolution and input into the channel attention module, and then the dimensionality reduction processing is performed through 1×1 convolution, which can enrich the extracted data features while increasing the receptive field.

[0068] Specifically, the local feature extraction network is composed of two basic convolutional modules and two attention modules connected in sequence. The basic convolutional module includes a convolutional layer, a batch normalization layer (BatchNormalization, BN) and an activation function layer. The filter size of the convolutional layer is 3×3; the attention module uses the bottleneck module as the basic module, and its filter combination is 1×1+3×3+1×1, and uses the channel space attention mechanism (ConvolutionalBlockAttentionModule, CBAM).

[0069] The global feature extraction network is used to extract the overall features of the eye movement data in the eye movement dataset at multiple scales. The local regions are divided according to the different behavioral stages of the eye movement to obtain local region data, which are then input into the local feature extraction network. The attention mechanism is used to enhance the effective local features.

[0070] In this embodiment, the overall eye movement data in the eye movement data set is used as the input of the global feature extraction network. After two basic convolution modules, the latitude of the feature map is 64×64×64. The feature channel is processed by 1×1 convolution to increase the dimension, and the BN layer and Swish activation function are added. Then, the feature channels are grouped, and the above 4 groups are directly output to the 3×3 depth separable convolution using convolution filtering. The feature inputs of the latter two groups are the fusion features of the original group features and the previous group features after 3×3 depth separable convolution. After that, the four channels are connected, and the connected features are input into the channel attention module, and then the dimension is reduced by a 1×1 convolution, and added to the original input as the output. The feature size after three multi-scale feature extraction modules is 8×8×512.

[0071] The local regions can be divided according to different behavioral stages of eye movement (such as fixation, saccade, blink, etc.). For example, the eye movement data corresponding to the fixation stage is divided into a local region, and the data of the start and end stages of saccade are divided into different local regions, etc., so as to generate multiple "local region data" that can reflect the local features of eye movement as the input of the local feature extraction network.

[0072] The upsampled features of the second basic convolution module of the local feature extraction network are added to the second basic convolution module of the global feature extraction network; the upsampled features of the first attention module of the local feature extraction network are added to the first multi-scale feature extraction module of the global feature extraction network; the upsampled features of the second attention module of the local feature extraction network are added to the second multi-scale feature extraction module of the global feature extraction network; the upsampled features of the second attention module of the local feature extraction network are added to the third multi-scale feature extraction module of the global feature extraction network; the overall features and the local features are fused. In this way, the eye tracking recognition model of this embodiment combines multi-scale features with the attention mechanism, respectively extracts the overall features and local features of the data set transmitted by contact lenses, and fuses them at different layers of the eye tracking recognition model, which can enhance the local effective features of the eye movement data set, thereby improving the recognition ability of eye movement behavior.

[0073] The fused features are passed through global average pooling and a fully connected layer, and the eye movement behavior recognition results are output through softmax.

[0074] It is expressed as:

[0075]

[0076] Among them, y i represents the output probability of the i-th eye movement behavior category, z i represents the output value of the i-th eye movement behavior category, j represents the total number of eye movement behavior categories, and the predicted eye movement behavior category is the eye movement behavior category with the highest probability.

[0077] In a specific embodiment, training and optimizing an eye tracking recognition model using a data set transmitted by a contact lens includes:

[0078] Design the loss function and use Loss to express it as:

[0079] Loss = -α c (1-p c ) β log(p c )

[0080] where α c represents the weight of the corresponding eye movement behavior category, pc represents the probability of predicting the cth class, and β represents the hyperparameter;

[0081] The eye tracking recognition model is trained using the data set transmitted by the contact lens, the parameters of the eye tracking recognition model are set, and the eye tracking recognition model is optimized using the optimizer.

[0082] Specifically, the parameters of the eye tracking recognition model include learning rate, learning rate decay strategy and the number of training iterations, where the learning rate decay strategy is an exponential decay strategy, expressed as:

[0083] LR′=LR×γ epoch

[0084] Among them, LR′ and LR represent the attenuated learning rate and the initial learning rate respectively, γ represents the attenuation exponent, and epoch represents the number of rounds of current training.

[0085] Specifically, the optimizer is the Adam optimizer.

[0086] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0087] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A contact lens, characterized in that: include: Lens body, sensor integration module, energy supply module, wireless communication module, packaging protection layer; The sensor integrated module is arranged in the peripheral area of ​​the lens body or in a specific layered structure; The energy supply module is connected to the sensor integration module to provide it with the required electrical energy; The wireless communication module is responsible for bidirectional data transmission with external devices, sending the collected eye movement data to the external devices for data processing, and also receiving command information from the external devices; The packaging protection layer covers the outer layer of the lens body.

2. A contact lens according to claim 1, characterized in that: The sensor integrated module includes a plurality of sensor arrays, and the sensor arrays are distributed on the lens body in a specific shape, and the specific shape is one of a ring, a square and an X shape.

3. A contact lens according to claim 1, characterized in that: The energy supply module adopts micro thin film batteries to provide electrical energy.

4. A contact lens according to claim 1, characterized in that: When the external device needs to recalibrate or adjust parameters of the sensor integrated module on the contact lens, the corresponding control instruction is sent to the contact lens through the wireless communication module, and the contact lens performs the corresponding operation after receiving the control instruction.

5. An eye tracking method based on contact lenses, applied to a contact lens as claimed in any one of claims 1 to 4, characterized in that: include: Use sensor integrated modules to obtain data related to eye movements from multiple dimensions; After connecting with external devices through the wireless communication module, each sensor in the sensor integration module is initialized and calibrated; When the user puts on the contact lenses, as the eyeballs move during various activities, various sensors begin to collect relevant data in real time; The collected relevant data is periodically transmitted to an external device via a wireless communication module; On the external device side, after receiving the data transmitted from the contact lens, it first extracts its features; then the data from different sensors are fused; finally, the fused data is recognized using a neural network model to obtain the eye movement behavior recognition results.

6. The contact lens-based eye tracking method according to claim 5, characterized in that: It also includes a micro storage unit built into the contact lens for temporarily storing data collected by the sensor. The storage unit can cope with the generation of a large amount of data in a short period of time and avoid data loss.

7. The contact lens-based eye tracking method according to claim 5, characterized in that: The sensor integration module integrates a sensor array consisting of micro-gyroscopes and micro-accelerometers to obtain data related to eye movement from multiple dimensions; the micro-gyroscope can accurately sense the changes in angular velocity in all directions when the eye moves; the micro-accelerometer focuses on capturing the linear acceleration during eye movement.

8. The contact lens-based eye tracking method according to claim 5, characterized in that: The external device side specifically includes: Establish an eye tracking recognition model, extract the overall features of the eye movement data at multiple scales from the data set transmitted by the contact lens, and use the attention mechanism to enhance the effective local features, and fuse the overall features with the local features; Using the data set transmitted by the contact lens to train and optimize the eye tracking recognition model; The trained eye tracking recognition model is used to identify the data transmitted by the contact lenses and predict the corresponding eye movement behavior.

9. The contact lens-based eye tracking method according to claim 8, characterized in that: The establishment of the eye tracking recognition model specifically includes: Establishing an eye tracking recognition model, wherein the eye tracking recognition model includes a global feature extraction network and a local feature extraction network; The global feature extraction network is used to extract the overall features of the eye movement data in the eye movement dataset at multiple scales, and the local area data is obtained by dividing the local area according to different behavior stages of the eye movement, and the local area data is input into the local feature extraction network, and the effective local features are strengthened by using the attention mechanism; Adding the upsampled features of the second basic convolution module of the local feature extraction network to the second basic convolution module of the global feature extraction network; adding the upsampled features of the first attention module of the local feature extraction network to the first multi-scale feature extraction module of the global feature extraction network; adding the upsampled features of the second attention module of the local feature extraction network to the second multi-scale feature extraction module of the global feature extraction network; adding the upsampled features of the second attention module of the local feature extraction network to the third multi-scale feature extraction module of the global feature extraction network; The fused features are passed through global average pooling and a fully connected layer, and the eye movement behavior recognition results are output through softmax.

10. The contact lens-based eye tracking method according to claim 5, characterized in that: Training and optimizing the eye tracking recognition model using the data set transmitted by the contact lens includes: Design the loss function and use Loss to express it as: Loss=-α c (1-p c ) β log(p c ) where α c represents the weight of the corresponding eye movement behavior category, p c represents the probability of predicting the cth class, and β represents the hyperparameter; The eye tracking recognition model is trained using the data set transmitted by the contact lens, and the parameters of the eye tracking recognition model are set, and the eye tracking recognition model is optimized using an optimizer.

Images