An eye detection method, device, storage medium and program product

Abstract

Embodiments of the present disclosure provide a method and device for eye detection, a storage medium and a program product, and relate to the technical field of human-computer interaction, which can improve the accuracy of detecting the activity change of the eyeball of a user by smart glasses. The method is applied to smart glasses and includes: obtaining an eye detection instruction, the eye detection instruction being used to indicate the motion state of the eyeball; obtaining distance data based on the eye detection instruction, the distance data including distance values between the smart glasses and at least one eyelid region when the eyeball moves to different positions; and determining a target motion state of the eyeball according to the distance data.

Description

Technical Field

[0001] This disclosure relates to the field of human-computer interaction technology, and in particular to an eye detection method, device, storage medium, and program product. Background Technology

[0002] In recent years, with the continuous development of technology, wearable devices have become increasingly diverse, expanding the ways users can interact with computers in various scenarios. For example, users can control screen operations through smart glasses.

[0003] Currently, smart glasses can use image processing to determine changes in the user's eye movements when responding to user control operations on the screen, thereby reading the user's control commands and replacing manual input.

[0004] However, in the above technical solutions, changes in user scenarios and light source factors (such as light intensity and light uniformity) in different scenarios may affect the image information on which image processing is based, reducing the accuracy of smart glasses in detecting changes in the user's eye movements. Summary of the Invention

[0005] This disclosure provides an eye detection method, apparatus, storage medium, and program product, which can at least improve the accuracy of smart glasses in detecting changes in user eye movement.

[0006] On the one hand, an eye detection method is provided, which is applied to smart glasses, including: acquiring an eye detection command, the eye detection command being used to indicate the movement state of the detected eye; acquiring distance data based on the eye detection command, the distance data including: the distance value between the smart glasses and at least one eyelid region when the eye moves to different positions; and determining the target movement state of the eye based on the distance data.

[0007] On the other hand, an eye detection device is provided for use in smart glasses, the device comprising: an acquisition module and a processing module.

[0008] The acquisition module is used to acquire eye detection commands, which indicate the movement state of the detected eyeballs. The acquisition module is also used to acquire distance data based on the eye detection commands. The distance data includes the distance values ​​between the smart glasses and at least one eyelid area when the eyeball moves to different positions. The processing module is used to determine the target movement state of the eyeballs based on the distance data.

[0009] In another aspect, an electronic device is provided, comprising: a memory and a processor. The memory and the processor are coupled. The memory is used to store a computer program. When the processor executes the computer program, it implements the eye detection method of any of the above embodiments.

[0010] In another aspect, a computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the eye detection method of any of the above embodiments.

[0011] In another aspect, a computer program product is provided, which includes computer program instructions that, when executed, implement the eye detection method of any of the above embodiments.

[0012] This disclosure describes a method to determine the deformation process of a user's eyelids by acquiring changes in the distance between the smart glasses and the user's eyelids, thereby identifying the eye movement state that triggers the eyelid deformation. By detecting eye movements based on changes in the distance between the smart glasses and the user's eyelids, the influence of light source factors in different usage scenarios on parameter data can be avoided, improving the accuracy of the smart glasses in detecting changes in user eye movement. Furthermore, detecting eye movements by collecting and analyzing distance data between the smart glasses and the user's eyelids reduces energy consumption and deployment space requirements compared to image processing methods, making the smart glasses lighter, smaller, and more energy-efficient. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in this disclosure, the accompanying drawings used in some embodiments of this disclosure will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.

[0014] Figure 1 This is a schematic diagram of the structure of an eye detection system provided in some embodiments of the present disclosure;

[0015] Figure 2 This is a schematic flowchart illustrating an eyeball detection method provided in some embodiments of this disclosure;

[0016] Figure 3 This is a schematic diagram illustrating an example of detecting vertical eye movement provided in some embodiments of this disclosure;

[0017] Figure 4 This is a schematic diagram illustrating another example of detecting vertical eye movement provided by some embodiments of this disclosure;

[0018] Figure 5 This is a schematic diagram illustrating another example of detecting vertical eye movement provided by some embodiments of this disclosure;

[0019] Figure 6 A schematic diagram illustrating an example of the deployment location of multiple distance sensors on smart glasses, provided for some embodiments of this disclosure;

[0020] Figure 7 This is a schematic diagram illustrating another example of detecting vertical eye movement provided by some embodiments of this disclosure;

[0021] Figure 8 This is a schematic diagram illustrating an example of detecting left and right eye movement, provided by some embodiments of this disclosure.

[0022] Figure 9 This is a schematic diagram illustrating another example of detecting left and right eye movement provided by some embodiments of this disclosure;

[0023] Figure 10 This is a schematic diagram illustrating another example of detecting left and right eye movement provided by some embodiments of this disclosure;

[0024] Figure 11 This is a schematic diagram illustrating the trend of distance value changing over time, provided for some embodiments of this disclosure;

[0025] Figure 12 This is a schematic diagram illustrating an example of detecting eye movement provided in some embodiments of this disclosure;

[0026] Figure 13 This is a schematic diagram illustrating another example of detecting eye movement provided by some embodiments of this disclosure;

[0027] Figure 14 This is a schematic diagram illustrating another example of detecting eye movement provided by some embodiments of this disclosure;

[0028] Figure 15 This is a schematic diagram illustrating another trend of distance value changing over time, provided for some embodiments of this disclosure;

[0029] Figure 16 A schematic diagram of a virtual screen for smart glasses provided in some embodiments of this disclosure;

[0030] Figure 17 A schematic flowchart illustrating another eyeball detection method provided in some embodiments of this disclosure;

[0031] Figure 18 This is a schematic diagram of the structure of an eyeball detection device provided in some embodiments of the present disclosure;

[0032] Figure 19 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this disclosure. Detailed Implementation

[0033] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0034] It should be noted that, in this disclosure, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this disclosure should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0035] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0036] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "more than one" means two or more.

[0037] With the continuous development of technology, smart glasses are experiencing explosive growth. As devices that users carry at all times, smart glasses can connect and interconnect with peripheral devices, and have a wide range of applications.

[0038] In human-computer interaction, detecting the direction of the user's gaze and determining the target on the virtual screen that the user is currently looking at is an effective method for manipulating the target.

[0039] However, the current eye trackers have a drawback: they cannot save power while tracking accurately. Most devices use cameras to recognize the eyes, which requires intensive image processing, thus significantly increasing power consumption and requiring large-capacity batteries, which in turn increases the overall weight.

[0040] Therefore, how to make eye-tracking devices for smart glasses lightweight, small in size, and low in power consumption, and improve the accuracy of smart glasses in detecting changes in the user's eye movements, is an urgent problem that needs to be solved.

[0041] Based on this, this disclosure provides an eyeball detection method applied to scenarios where smart glasses detect a user's eyeballs. By acquiring the distance change between the smart glasses and the user's eyelids, the deformation process of the user's eyelids can be determined, thereby identifying the eyeball movement state that triggers the eyelid deformation. In this way, detecting the eyeball by referencing the distance change between the smart glasses and the user's eyelids avoids the influence of light source factors in different usage scenarios on parameter data, improving the accuracy of the smart glasses in detecting changes in the user's eyeball activity. Furthermore, detecting the eyeball by collecting and analyzing distance data between the smart glasses and the user's eyelids reduces energy consumption and deployment space requirements compared to image processing methods, making the smart glasses lighter, smaller, and more energy-efficient.

[0042] For example, such as Figure 1 The diagram shown is a structural schematic of an eye detection system provided in an embodiment of this disclosure. The eye detection system may include: at least one distance acquisition module (S1-Sn), a data storage module, a baseband module, a power management module, a central processing unit (CPU), and a program module.

[0043] Among them, the distance acquisition module is used to collect the distance value between the user's eyelid area and the smart glasses. Multiple distance acquisition modules can collect the distance value between one eyelid area and the smart glasses at the same time, or different distance acquisition modules can collect the distance values ​​between different eyelid areas and the smart glasses.

[0044] It should be noted that the distance acquisition module is a miniature distance detection sensor. For example, a distance acquisition module can be a laser rangefinder. Another example is an ultrasonic rangefinder. Yet another example is an infrared rangefinder.

[0045] The data storage module stores the distance data collected by all distance acquisition modules, providing a data foundation for the CPU to determine the movement state of the eyeball.

[0046] The baseband module is used to process the signals from the distance acquisition module to obtain distance data between the user's eyelid area and the smart glasses.

[0047] The power management module is used to provide a stable power supply for the normal operation of other modules and to manage energy consumption based on the operating conditions of each module.

[0048] The CPU is used to analyze and process the distance data between the smart glasses and the eyelid area to determine the movement state of the user's eyeballs.

[0049] Program modules provide functional logic for the operation of other modules.

[0050] It should be noted that, Figure 1 This is just an example framework diagram. Figure 1 The number of modules included, and the names of each module, are unlimited, except for... Figure 1 In addition to the modules shown, the eye detection system may also include other modules, such as an antenna module.

[0051] The application scenarios of the embodiments disclosed herein are not limited. The system architecture and business scenarios described in the embodiments of this disclosure are for the purpose of more clearly illustrating the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of this disclosure. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this disclosure are also applicable to similar technical problems.

[0052] Figure 2 A flowchart of an eyeball detection method is shown, as follows: Figure 2 As shown, the eyeball detection method includes:

[0053] S201, Obtain eye detection command.

[0054] Among them, the eyeball detection command is used to indicate the movement state of the detected eyeball.

[0055] As one possible implementation, smart glasses can receive eye detection commands sent by other terminal devices.

[0056] For example, a user wearing smart glasses can send eye detection commands to the smart glasses via their mobile phone.

[0057] Alternatively, users wearing smart glasses can receive eye-detection commands sent from other users' mobile phones.

[0058] As another possible implementation, smart glasses can receive eye detection commands manually input by the user wearing the smart glasses.

[0059] For example, a user wearing smart glasses can trigger the eye detection function by clicking the control button on the smart glasses to obtain eye detection instructions.

[0060] As another possible implementation, smart glasses can obtain eye detection commands by detecting preset movements of the user's eyeballs.

[0061] For example, smart glasses can trigger an eye detection function to obtain an eye detection command when they detect that the user's eyeballs have moved in a circular trajectory for 3 revolutions.

[0062] In other words, by expanding the sources of stimulation for eye detection in smart glasses, we can enrich the human-computer interaction methods of smart glasses and improve the user experience.

[0063] Optionally, the smart glasses can obtain eye detection instructions from within the system after a preset mode (such as eye detection mode) is activated.

[0064] It should be noted that the surface of the human eye consists of the eyeball and the upper and lower eyelids. The cornea is the most prominent part of the eyeball. When the line of sight moves up and down, the cornea on the eyeball moves along with it. The eyelids also move up and down accordingly as the line of sight moves up and down.

[0065] In other words, as the user's eyeballs move, the user's eyelids, which are connected to the eyeballs, also deform accordingly. That is, eyeball movement and eyelid deformation change synchronously, and there is a correlation between eyeball movement and eyelid deformation. Therefore, eyeball movement can be indirectly detected by detecting eyelid deformation, and the detection of eyelid deformation can be achieved through the change in distance between the smart glasses and the eyelid area in the following embodiment (S202).

[0066] S202. Obtain distance data based on eye detection instructions.

[0067] As one possible implementation, smart glasses can be equipped with at least one distance sensor. The smart glasses can then respond to eye-detection commands based on this at least one deployed distance sensor to acquire distance data.

[0068] The distance data may include the distance values ​​between the smart glasses and at least one eyelid region when the eyeball moves to different positions.

[0069] Optionally, the smart glasses may be equipped with only one distance sensor. Based on this single distance sensor, the smart glasses can acquire distance data by collecting distance values ​​between at least one eyelid region and a reference point on the smart glasses as the eyeball moves to different positions.

[0070] The reference point on the smart glasses can be the deployment location of the distance sensor on the smart glasses.

[0071] Alternatively, this reference point on the smart glasses can be a preset position on the smart glasses. As the user's wearing posture changes, the distance between this preset position on the smart glasses and the user's eyelids is not easily changed (i.e., it always remains within a stable distance range).

[0072] For example, the preset position on smart glasses can be the bridge of the nose. No matter how the user changes the wearing posture of the smart glasses, the distance between the bridge of the nose and the user's eyelids will always remain at a stable value (or remain at a stable distance range).

[0073] Similarly, the number of distance sensors deployed in smart glasses can be the same as the number of eyelid regions. Each distance sensor collects the distance value between an eyelid region and the sensor to obtain the distance value between the smart glasses and at least one eyelid region.

[0074] The following example illustrates how smart glasses use a distance sensor to detect eyelid deformation caused by the vertical movement of the eyeball.

[0075] For example, taking a distance sensor S1 as an example, when the user looks forward (i.e., the direction of eye gaze is directly forward), such as Figure 3 As shown, the distance sensor S1 detects the distance between S1 and the user's upper eyelid as D11;

[0076] In combination with the above Figure 3 When the user looks upwards (i.e., the direction of eye gaze is directly upwards), the upward movement of the eyeball causes the protruding cornea to push the upper eyelid towards the distance sensor S1. Simultaneously, the upper eyelid flips upwards, increasing its thickness. This is similar to... Figure 4 As shown, the distance sensor S1 detects the distance between the distance sensor S1 and the user's upper eyelid as D12, and D11 > D12.

[0077] In combination with the above Figure 3 When the user looks upwards (i.e., at the extreme position where the eyeball is directly upwards), the upper eyelid flips upwards and moves out of the detection area of ​​the distance sensor S1 due to the upward rotation of the eyeball. At this time, the distance sensor S1 detects the distance between itself and the eyeball. Figure 5 As shown, the distance sensor S1 detects the distance between the distance sensor S1 and the user's upper eyelid as D13, and D13 > D11.

[0078] For example, taking multiple distance sensors (such as distance sensor S1 and distance sensor Sn) as an example, Figure 6 As shown, this illustrates the deployment locations of multiple distance sensors on the smart glasses. Distance sensor S1 is deployed in the upper half of the smart glasses to detect the distance between the sensor and the user's upper eyelid, while distance sensor Sn is deployed in the lower half of the smart glasses to detect the distance between the sensor and the user's lower eyelid.

[0079] When the user is looking forward (i.e., the direction of their gaze is directly forward), such as Figure 7As shown, distance sensor S1 detects the distance between distance sensor S1 and the user's upper eyelid as D11, and distance sensor Sn detects the distance between distance sensor Sn and the user's lower eyelid as Dn1.

[0080] The following example illustrates how smart glasses use a distance sensor to detect eyelid deformation caused by left and right eye movements.

[0081] For example, when the glasses are looking straight ahead, such as Figure 8 As shown, the diagram illustrates the detection points of various distance sensors on the eyelid, including S1D representing the detection point of distance sensor S1, S2D representing the detection point of distance sensor S2, S3D representing the detection point of distance sensor S3, and SnD representing the detection point of distance sensor Sn. The dashed circles in the diagram represent the cornea. Distance sensor S2 can detect a distance of D21 (e.g., 8.2 mm) from S2D.

[0082] In combination with the above Figure 8 When the eyes turn to the right (as with the left eye, turning towards the nose), as... Figure 9 As shown, because the cornea of ​​the eyeball bulges and pushes up the upper eyelid of the S2D detection point, the distance between the distance sensor S2 and S2D will decrease, so that the smart glasses can determine that the left eye is looking to the right when the distance sensor S2 detects that the distance to S2D is D22 (e.g., 6.5mm) less than D21.

[0083] In combination with the above Figure 8 When the eyes turn to the left (e.g., the left eye, turning in the opposite direction of the nose), as... Figure 10 As shown, because the cornea of ​​the eyeball bulges and pushes up the upper eyelid of the S3D detection point, the distance between the distance sensor S3 and S3D will decrease, so that the smart glasses can detect the distance to S3D through the distance sensor S3 and determine that the left eye is looking to the left.

[0084] S203. Determine the target movement state of the eyeball based on the distance data.

[0085] Understandably, by acquiring changes in the distance between the smart glasses and the user's eyelids, the deformation process of the user's eyelids can be determined, thereby identifying the eye movement state that triggers the eyelid deformation. In this way, detecting eye movements by referencing changes in the distance between the smart glasses and the user's eyelids can avoid the influence of light source factors in different usage scenarios on parameter data, improving the accuracy of the smart glasses in detecting changes in the user's eye movement. Furthermore, detecting eye movements by collecting and analyzing distance data between the smart glasses and the user's eyelids can reduce energy consumption and deployment space requirements compared to image processing methods, making smart glasses lighter, smaller, and more energy-efficient.

[0086] It should be noted that, in the embodiments of this disclosure, the movement state of the eyeball may include the gaze direction and / or movement trajectory of the eyeball (i.e., the target movement state determined above may include the target gaze direction and / or the target movement trajectory).

[0087] In some embodiments, during the process of the smart glasses determining the target motion state of the eyeball based on distance data (i.e., S203), the smart glasses can determine the target gaze direction of the eyeball based on the first set of distance values ​​in the distance data.

[0088] The first set of distance values ​​includes the distance values ​​between the smart glasses and at least one eyelid region when the eyeball moves to its last position.

[0089] In other words, after responding to an eye detection command, smart glasses can obtain the distance values ​​between the user's eyeball and each eyelid region when the user's eyeball is finally at rest, and then determine the deformation state of the user's eyelid at this time, so as to determine the direction of the user's gaze.

[0090] As one possible implementation, smart glasses can store multiple preset conditions, with each preset condition corresponding to a preset gaze direction. During the process of determining the target gaze direction based on a first set of distance values, the smart glasses can match the first set of distance values ​​against multiple preset conditions. If the first set of distance values ​​satisfies the first preset condition among the multiple preset conditions, the target gaze direction is determined to be the preset gaze direction corresponding to the first preset condition.

[0091] The first preset condition may include a preset distance range corresponding to each eyelid area.

[0092] For example, multiple preset conditions may include: preset condition A and preset condition B. In preset condition A, the preset distance range corresponding to eyelid region A can be 0.2mm to 0.25mm, and the preset distance range corresponding to eyelid region B can be 1.2mm to 1.3mm. In preset condition B, the preset distance range corresponding to eyelid region A can be 0.71mm to 0.89mm, and the preset distance range corresponding to eyelid region B can be 0.5mm to 0.61mm. If the distance value corresponding to eyelid region A in the first distance value set is 0.22mm and the distance value corresponding to eyelid region B is 1.21mm, then preset condition A is determined to be the first preset condition satisfied by the first distance value set.

[0093] It should be noted that the preset conditions stored in the smart glasses can be sensor distance data of a certain eye angle (e.g., angle C) obtained when the smart glasses initialize and calibrate the distance sensors S1-Sn, so that when the sensor data matches the calibration data during detection, it can be determined that the eye angle at this time is angle C.

[0094] In other words, smart glasses can match the distance values ​​of each eyelid region with their respective multiple distance ranges, and then combine the preset gaze directions corresponding to the different distance ranges corresponding to different eyelid regions to determine the preset gaze direction corresponding to the distance range matched by each eyelid region, thereby determining the target gaze direction of the user's eyes.

[0095] Optionally, the distance data may further include: an initial set of distance values ​​between the smart glasses and at least one eyelid region when the eyeball is looking straight ahead. During the process of the smart glasses determining the target gaze direction based on the first set of distance values, the smart glasses can calculate the difference between the distance value corresponding to each eyelid region in the first set of distance values ​​and the corresponding distance value in the initial set of distance values, obtaining a first set of distance difference values. Then, the smart glasses can match the first set of distance difference values ​​based on multiple preset conditions, and if the first set of distance difference values ​​satisfies a second preset condition among the multiple preset conditions, the target gaze direction is determined to be the preset gaze direction corresponding to the second preset condition.

[0096] The second preset condition may include the preset distance variation range corresponding to each eyelid area.

[0097] For example, multiple preset conditions may include: preset condition A and preset condition B. In preset condition A, the preset distance variation range corresponding to eyelid region A can be 0.3mm to 0.35mm, and the preset distance variation range corresponding to eyelid region B can be 1.1mm to 1.2mm. In preset condition B, the preset distance variation range corresponding to eyelid region A can be 0.72mm to 0.8mm, and the preset distance variation range corresponding to eyelid region B can be 0.4mm to 0.51mm. If, in the first distance difference set, the difference between the distance value corresponding to eyelid region A in the first distance value set and the distance value corresponding to it in the initial distance value set is 0.32mm, and the difference between the distance value corresponding to eyelid region B in the first distance value set and the distance value corresponding to it in the initial distance value set is 1.11mm, then preset condition A is determined to be the second preset condition satisfied by the first distance difference set.

[0098] In other words, smart glasses can match the distance changes of each eyelid region relative to the initial gaze direction with their respective multiple distance change ranges, and then combine the preset gaze directions corresponding to the different distance change ranges corresponding to different eyelid regions to determine the preset gaze direction corresponding to the distance change range matched by each eyelid region, thereby determining the target gaze direction of the user's eyes.

[0099] It should be noted that the initial distance value set can be the data parameters set when the smart glasses are first worn by the user; or, the initial distance value set can be the data parameters that are corrected in real time each time the smart glasses are worn by the user.

[0100] Taking the initial distance value set as an example, which consists of data parameters that are corrected in real time each time the smart glasses are worn by a user, the smart glasses can acquire a second distance value set when worn. This second distance value set includes the distance values ​​between the smart glasses and at least one eyelid region when the eyeball moves to different positions within a first preset time period. Next, the smart glasses can filter a third distance value set from the second distance value set. In the third distance value set, the difference between any two distance values ​​corresponding to each eyelid region is less than a first preset distance difference threshold, and the number of distance values ​​corresponding to each eyelid region in the third distance value set is greater than a preset number threshold. Then, for each eyelid region, the smart glasses can calculate the average of all distance values ​​corresponding to that eyelid region in the third distance value set to obtain the initial distance value set.

[0101] In other words, the distance between the smart glasses and the user's eyes may change over time, or the position of the smart glasses may vary slightly each time they are worn. To eliminate this error, the distance data needs to be corrected after it is acquired.

[0102] Furthermore, since the time the eye spends looking straight ahead is usually much longer than the time spent looking in a certain direction, the most frequent distance data obtained by a distance sensor is the data from when the eye is looking straight ahead.

[0103] For example, distance sensor S1 collects data 30 times in 3 seconds, excludes extreme data (maximum value, minimum value), selects the 20 closest distance data, and calculates the average of these 20 data. This average is the distance value D1s between distance sensor S1 and detection point S1D when the eye is looking straight ahead.

[0104] Similarly, other distance sensors can also use this method to obtain distance data when looking straight ahead.

[0105] This subroutine for obtaining D1s is executed once every time a user wears smart glasses, allowing D1s to be updated dynamically.

[0106] As another possible implementation, the distance data may further include: a set of historical distance values ​​between the smart glasses and at least one eyelid region when the eyes were in a historical position before acquiring the eye detection command; and the smart glasses may store multiple preset conditions, each preset condition corresponding to a preset eye movement angle. During the process of the smart glasses determining the target gaze direction based on the first set of distance values, the smart glasses can calculate the difference between the distance value corresponding to each eyelid region in the first set of distance values ​​and the corresponding distance value in the historical set of distance values, obtaining a second set of distance differences. Then, the smart glasses can match the second set of distance differences based on multiple preset conditions, and if the second set of distance differences satisfies a third preset condition, the target gaze direction is determined based on the historical gaze direction when the eyes were in the historical position, combined with the preset eye movement angle corresponding to the third preset condition.

[0107] The third preset condition may include the range of distance variation corresponding to each eyelid region.

[0108] It should be noted that the description of the third preset condition can be found in the example above describing the second preset condition, and will not be repeated here.

[0109] For example, if the historical gaze direction when the eye is in its historical position is the upper left direction (45 degrees upward and 30 degrees to the left), and the preset eye movement angles corresponding to the third preset condition include: 60 degrees downward and 15 degrees to the right, then the smart glasses can determine the target gaze direction as the lower left direction (15 degrees downward and 15 degrees to the left) based on the upper left direction and the eye movement angles of 60 degrees downward and 15 degrees to the right.

[0110] In other words, smart glasses can match the distance changes of each eyelid region relative to the historical gaze direction with its corresponding range of multiple distance changes, and then combine the preset eye movement angles corresponding to the different distance change ranges corresponding to different eyelid regions to determine the target gaze direction of the user's eyes based on the historical gaze direction.

[0111] In some embodiments, during the process of the smart glasses determining the target motion state of the eyeball based on distance data (i.e., S203), the smart glasses can determine at least one first distance sequence from the distance data, and determine the target motion trajectory of the eyeball based on the trend of the distance value in each first distance sequence changing over time.

[0112] In this system, a first distance sequence corresponds to an eyelid region. The first distance sequence includes the distance values ​​between the corresponding eyelid region and the smart glasses at multiple measurement times, and the distance values ​​in the first distance sequence are arranged in chronological order.

[0113] For example, the first distance sequence corresponding to eyelid region A can be 1.21mm-1.1mm-0.85mm. Here, 1.21mm is the distance between eyelid region A and the smart glasses at 8:12:11, 1.1mm is the distance between eyelid region A and the smart glasses at 8:12:12, and 0.85mm is the distance between eyelid region A and the smart glasses at 8:12:13.

[0114] In combination with the above Figures 3-5 The distance sensor S1 in the middle, such as Figure 11 As shown, the vertical axis represents the distance data (D) of the eyelid region detected by the distance sensor S1, and the horizontal axis represents time (t). Based on the distance data detected by the distance sensor S1 as it changes over time (i.e., D11, D12, and D13), the smart glasses can determine the trajectory of the user's eyes as they shift from looking forward to looking upward.

[0115] Similarly, combining the above Figures 8-10 Multiple distance sensors work together to detect changes in distance between distance sensors S1 and S1D, and between distance sensors Sn and SnD. This allows the system to determine the direction in which the user's eyes are turning. The CPU analyzes the distance changes detected by multiple sensors to determine the current direction of the user's gaze.

[0116] At time T1, such as Figure 12 As shown, when the user's eyes are looking straight ahead, distance sensor S1 detects the detection point S1D on the eyelid, and the detection distance between distance sensor S1 and S1D is D11; distance sensor S2 detects the detection point S2D on the eyelid, and the detection distance between distance sensor S2 and S2D is D21.

[0117] At time T2, such as Figure 13 As shown, the user's line of sight is in direction A. The detection distance between distance sensors S1 and S1D is D12, and the detection distance between distance sensors S2 and S2D is D22.

[0118] At time T3, such as Figure 14 As shown, the user's line of sight is in direction B. The detection distance between distance sensors S1 and S1D is D13, and the detection distance between distance sensors S2 and S2D is D23.

[0119] The distance data corresponding to D11, D12 and D13 detected by distance sensor S1 at times T1, T2 and T3, and the distance data corresponding to D21, D22 and D23 detected by distance sensor S2 at times T1, T2 and T3, can be shown in Table 1.

[0120] Table 1 shows the distance data detected by distance sensors S1 and S2 at times T1, T2, and T3, respectively.

[0121]

[0122] Based on Table 1 above, such as Figure 15 As shown, it illustrates the trend of distance values ​​corresponding to S1D and S2D changing over time.

[0123] At time T1, during D11 and D21, the eyelids were not pushed out by the cornea, indicating that the eyes were looking straight ahead.

[0124] At time T2, the distance between D12 and D22 is less than that at time T1, and the eyelid is pushed out by the cornea, indicating that the eye's line of sight is in direction A.

[0125] At time T3, the distance at D13 is similar to that at T1, indicating that the eyelid has not been pushed out by the cornea. At time T3, the distance at D23 is similar to that at D12, indicating that the eyelid has been pushed out by the cornea, meaning that the eye's line of sight is in direction B at this time.

[0126] In other words, smart glasses can determine the deformation process of the user's eyelids based on the changes in distance between each eyelid area and the smart glasses, and thus determine the movement trajectory of the user's eyeballs.

[0127] As one possible implementation, in the process of smart glasses determining the target movement trajectory of the eyeball based on the trend of distance values ​​changing over time in each first distance sequence, smart glasses can filter the distance values ​​in each first distance sequence to obtain a second distance sequence corresponding to each first distance sequence, and determine the target movement trajectory based on the trend of distance values ​​changing over time in each second distance sequence.

[0128] Among them, the difference between any two adjacent distance values ​​in the second distance sequence is greater than the second preset distance difference threshold.

[0129] For example, when a user's eyeball is focused in a certain direction, the protruding cornea pushes up the eyelid, and the data detected by the distance sensor in that direction will decrease. When the decrease exceeds a certain value (e.g., Es = 1.1 mm), it is determined that the focus in that direction is valid.

[0130] In combination with the above Figures 3-5 In the distance sensor S1, at time T1, the distance sensor S1 detects D11 = 8.1 mm, and at time T2, D12 = 6.8 mm. The change range E = D11 - D12 = 1.3 mm, and E > Es, which can be used to determine that the eyeball is moving upward.

[0131] In other words, by filtering the distance values ​​of each eyelid region during the deformation process, invalid data with small changes in distance values ​​can be eliminated, thereby improving the accuracy of subsequent eyeball detection.

[0132] As another possible implementation, each distance value in the first distance sequence corresponds to a holding duration, which is the duration for which the eye remains at the position corresponding to that distance value. During the process of the smart glasses determining the target movement trajectory of the eye based on the trend of distance values ​​changing over time in each first distance sequence, the smart glasses can delete first-type distance values ​​from each first distance sequence to obtain a third distance sequence corresponding to each first distance sequence, and determine the target movement trajectory based on the trend of distance values ​​changing over time in each third distance sequence.

[0133] Among them, the holding time corresponding to the distance value of the first type is less than the first preset time threshold, and the difference between the distance values ​​at the two positions before and after the distance value of the first type in the first distance sequence is less than the third preset distance difference threshold.

[0134] For example, the first distance sequence can be 1.21mm-0.45mm-1.2mm-0.43mm. Here, 1.21mm corresponds to a holding time of 1 second (s), 0.45mm corresponds to a holding time of 3 seconds, 1.2mm corresponds to a holding time of 1 second, and 0.43mm corresponds to a holding time of 1 second. If the first preset duration threshold is 2 seconds and the third preset distance difference threshold is 0.03mm, then the smart glasses can determine that 1.2mm is the first type of distance value, and the third distance sequence is 1.21mm-0.45mm-0.43mm.

[0135] It should be noted that the smart glasses can collect distance values ​​at 1 second intervals. Therefore, based on a 1-second collection interval, the first distance sequence after unfolding is 1.21mm-0.45mm--0.45mm-0.45mm-1.2mm-0.43mm, and the third distance sequence after unfolding is 1.21mm-0.45mm--0.45mm-0.45mm-0.43mm.

[0136] Optionally, in the process of determining the third distance sequence from the first distance sequence, in order to ensure that the number of distance values ​​in the two distance sequences is consistent, the smart glasses can replace the first type of distance value with the average of the distance values ​​at the two positions before and after the first type of distance value.

[0137] Based on the above example, the expanded third distance sequence can be 1.21mm-0.45mm--0.45mm-0.45mm-0.44mm-0.43mm.

[0138] In other words, by filtering the distance values ​​of each eyelid region during the deformation process, invalid data with small changes in distance value and interfering data with distance changes in a very short time but quick recovery can be eliminated, thereby improving the accuracy of subsequent eyeball detection.

[0139] In some embodiments, smart glasses can perform corresponding functions by combining eye detection results (i.e., target motion state).

[0140] In other words, smart glasses detect the direction of the user's gaze, map the operation instructions and target of the smart terminal, and thus achieve the relevant functions.

[0141] For example, such as Figure 16 As shown, it illustrates a virtual screen diagram of smart glasses, which represents the user's field of vision when looking straight ahead, with multiple application (APP) icons around the screen.

[0142] The distance sensor in the smart glasses detects and determines that the user's eyes are looking upwards. The program in the smart glasses analyzes the direction of the user's gaze and finds it matches the position of APP1 on the screen. The program in the smart glasses then issues a notification that APP1 is locked (using zoom, color change, or shaking). The smart glasses receive the user's next action to launch APP1 (e.g., blinking, 1-second gaze, or gesture).

[0143] Alternatively, if the user wearing smart glasses looks upwards, the smart glasses detect this movement and display a drop-down notification bar on their virtual screen. When the eyes look straight ahead, the notification bar disappears. This way, menus can be accessed by detecting eye movements.

[0144] The eyeball detection method provided in this disclosure will be described below with reference to specific embodiments, such as... Figure 17 As shown, it includes:

[0145] Step 1: Initialization.

[0146] Step 2: The smart glasses notify the user to change their gaze direction from the initial position to the specific position.

[0147] Step 3: During the process of the smart glasses turning in a specific direction, the data from each distance sensor is collected and used as comparison data for that specific direction to complete the reference data entry.

[0148] In other words, smart glasses can calibrate data from various distance sensors. By issuing commands, smart glasses guide the user's gaze direction and collect data from various distance sensors to complete the input of reference data.

[0149] Step 4: The smart glasses collect data from each distance sensor and compare and analyze the data with the reference data.

[0150] Step 5: The smart glasses determine whether the current distance sensor data matches the reference data of each distance sensor in the direction of eye gaze.

[0151] In some embodiments, if the conditions are met, step six is ​​performed.

[0152] Step 6: Output the direction of eye gaze.

[0153] The eye gaze direction that is matched with the current distance sensor data will be used as the current eye gaze direction.

[0154] In other instances, if the condition is not met, step four is repeated.

[0155] In practical applications of smart glasses, the distance sensor can transmit the detected data to the CPU in a timely and accurate manner. The CPU then processes the data and eliminates momentary interference. For example, if the distance sensor S1 detects a change in distance within 0.2 seconds that quickly recovers, it can determine that this was a casual glance by the user and not looking upwards, thus filtering out this interference.

[0156] In other words, during human-computer interaction, the user wears smart glasses and looks at a target on a virtual screen. A distance sensor detects the distance and analyzes the user's gaze direction. When the user's gaze aligns with the target on the virtual screen, the target is activated. This is further enhanced by blinking, gestures, or nodding, triggering the target's action. This allows for human-computer interaction through eye movements, greatly simplifying smart glasses operation. Compared to voice, touch, and gesture controls, eye-based operation is significantly more convenient.

[0157] In summary, by installing several distance sensors on the smart glasses frame to detect changes in the distance between the target area and the sensor, the current direction of eye movement is obtained. Through calculation, the eye's gaze direction is determined, and the smart glasses map relevant operation commands based on the user's current gaze direction to achieve the corresponding functions. In this way, by using low-power distance sensors to detect eye movement, the drawbacks of using high-power cameras and image processing solutions are overcome, resulting in a more practical and lightweight smart glasses solution.

[0158] It is understood that, in order to achieve the above-mentioned functions, the eye detection device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the algorithm steps of the various examples described in conjunction with the embodiments of this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0159] This disclosure embodiment can divide the eye detection device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one functional module. The integrated module can be implemented in hardware or software. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the example of dividing each functional module according to each function.

[0160] Figure 18 This is a schematic diagram of an eye detection device provided in an embodiment of this disclosure. The eye detection device can be applied to smart glasses and perform the above-described functions. Figure 2 The method for detecting the eyeball is shown. For example... Figure 18 As shown, the eye detection device 1800 includes: an acquisition module 1801 and a processing module 1802.

[0161] The acquisition module 1801 is used to acquire eye detection instructions, which are used to indicate the movement state of the detected eyeball; the acquisition module 1801 is also used to acquire distance data based on the eye detection instructions, the distance data including: the distance value between the smart glasses and at least one eyelid area when the eyeball moves to different positions; the processing module 1802 is used to determine the target movement state of the eyeball based on the distance data.

[0162] In some embodiments, the target motion state includes the target gaze direction of the eyeball; the processing module 1802 is specifically configured to determine the target gaze direction based on a first set of distance values ​​in the distance data, the first set of distance values ​​including the distance values ​​between the smart glasses and at least one eyelid region when the eyeball moves to its last position.

[0163] In some embodiments, the processing module 1802 is specifically used to determine the target gaze direction as the preset gaze direction corresponding to the first preset condition when the first distance value set satisfies the first preset condition.

[0164] In some embodiments, the distance data further includes: an initial set of distance values ​​between the smart glasses and at least one eyelid region when the eyeball is looking straight ahead; the processing module 1802 is specifically used to calculate the difference between the distance value corresponding to each eyelid region in the first set of distance values ​​and the distance value corresponding to the initial set of distance values, to obtain a first set of distance difference values; the processing module 1802 is also used to determine the target gaze direction as the preset gaze direction corresponding to the second preset condition when the first set of distance difference values ​​satisfies the second preset condition.

[0165] In some embodiments, the acquisition module 1801 is further configured to acquire a second distance value set when the smart glasses are worn, the second distance value set including distance values ​​between the smart glasses and at least one eyelid region when the eyeball moves to different positions within a first preset time period; the processing module 1802 is further configured to filter out a third distance value set from the second distance value set, wherein the difference between any two distance values ​​corresponding to each eyelid region in the third distance value set is less than a first preset distance difference threshold, and the number of distance values ​​corresponding to each eyelid region in the third distance value set is greater than a preset number threshold; the processing module 1802 is further configured to calculate the average value of all distance values ​​corresponding to the eyelid region in the third distance value set for each eyelid region, to obtain an initial distance value set.

[0166] In some embodiments, the distance data further includes: a set of historical distance values ​​between the smart glasses and at least one eyelid region when the eyeball was in a historical position before obtaining the eyeball detection command; the processing module 1802 is specifically used to calculate the difference between the distance value corresponding to each eyelid region in the first distance value set and the distance value corresponding to the historical distance value set, to obtain a second distance difference set; the processing module 1802 is also used to determine the target gaze direction based on the historical gaze direction when the eyeball was in a historical position, combined with the preset eyeball change angle corresponding to the third preset condition, when the second distance difference set satisfies a third preset condition.

[0167] In some embodiments, the target motion state includes the target motion trajectory of the eyeball; the processing module 1802 is specifically configured to determine at least one first distance sequence from the distance data, one first distance sequence corresponding to one eyelid region, the first distance sequence including the distance values ​​between the corresponding eyelid region and the smart glasses at multiple measurement times, and the distance values ​​in the first distance sequence are arranged in chronological order; the processing module 1802 is further configured to determine the target motion trajectory based on the trend of the distance values ​​in each first distance sequence changing over time.

[0168] In some embodiments, the processing module 1802 is specifically used to filter the distance values ​​in each first distance sequence to obtain a second distance sequence corresponding to each first distance sequence, wherein the difference between any two adjacent distance values ​​in the second distance sequence is greater than a second preset distance difference threshold; the processing module 1802 is also used to determine the target motion trajectory based on the trend of the distance values ​​in each second distance sequence changing over time.

[0169] In some embodiments, a distance value in each first distance sequence corresponds to a holding duration, which is the duration for which the eye remains at the position corresponding to the distance value; the processing module 1802 is specifically used to delete the first type of distance value in each first distance sequence to obtain a third distance sequence corresponding to each first distance sequence, wherein the holding duration corresponding to the first type of distance value is less than a first preset duration threshold, and the difference between the distance values ​​at the two positions before and after the first type of distance value in the first distance sequence is less than a third preset distance difference threshold; the processing module 1802 is also used to determine the target motion trajectory based on the trend of the distance value in each third distance sequence changing over time.

[0170] When the functions of the integrated modules described above are implemented in hardware, this disclosure provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 19 As shown, the electronic device 1900 includes: a processor 1902 and a bus 1904. Optionally, the electronic device may also include a memory 1901; optionally, the electronic device may also include a communication interface 1903.

[0171] Processor 1902 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with embodiments of this disclosure. Processor 1902 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with embodiments of this disclosure. Processor 1902 may also be a combination of functions implementing computation, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0172] Communication interface 1903 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0173] The memory 1901 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0174] As one possible implementation, the memory 1901 can exist independently of the processor 1902. The memory 1901 can be connected to the processor 1902 via a bus 1904 and is used to store instructions or program code. When the processor 1902 calls and executes the instructions or program code stored in the memory 1901, it can implement the eye detection method provided in this embodiment of the present disclosure.

[0175] In another possible implementation, the memory 1901 can also be integrated with the processor 1902.

[0176] The 1904 bus can be an extended industry standard architecture (EISA) bus, etc. The 1904 bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 19 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0177] Some embodiments of this disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions that, when executed on a computer, cause the computer to perform the eye detection method as described in any of the above embodiments.

[0178] Exemplary examples of computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0179] This disclosure provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the eye detection method described in any of the above embodiments.

[0180] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions within the technical scope disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for detecting an eyeball, characterized in that, Applied to smart glasses, the method includes: Obtain an eyeball detection command, wherein the eyeball detection command is used to indicate the movement state of the detected eyeball; Based on the eyeball detection command, distance data is acquired, including: the distance values ​​between the smart glasses and at least one eyelid region when the eyeball moves to different positions; Based on the distance data, the target motion state of the eyeball is determined.

2. The method according to claim 1, characterized in that, The target motion state includes the target gaze direction of the eyeball; determining the target motion state of the eyeball based on the distance data includes: The target gaze direction is determined based on a first set of distance values ​​in the distance data, wherein the first set of distance values ​​includes the distance values ​​between the smart glasses and the at least one eyelid region when the eyeball moves to its last position.

3. The method according to claim 2, characterized in that, Determining the target gaze direction based on the first set of distance values ​​in the distance data includes: If the first set of distance values ​​satisfies the first preset condition, the target gaze direction is determined to be the preset gaze direction corresponding to the first preset condition.

4. The method according to claim 2, characterized in that, The distance data further includes: an initial set of distance values ​​between the smart glasses and the at least one eyelid region when the eyeball is looking straight ahead; determining the target gaze direction based on the first set of distance values ​​in the distance data includes: Calculate the difference between the distance value corresponding to each eyelid region in the first distance value set and the distance value corresponding to the initial distance value set to obtain the first distance difference set; If the first set of distance differences satisfies the second preset condition, the target gaze direction is determined to be the preset gaze direction corresponding to the second preset condition.

5. The method according to claim 4, characterized in that, The method further includes: When the smart glasses are worn, a second set of distance values ​​is obtained, the second set of distance values ​​including the distance values ​​between the smart glasses and the at least one eyelid area when the eyeball moves to different positions within a first preset time period; A third distance value set is selected from the second distance value set, wherein the difference between any two distance values ​​corresponding to each eyelid region in the third distance value set is less than a first preset distance difference threshold, and the number of distance values ​​corresponding to each eyelid region in the third distance value set is greater than a preset number threshold; For each eyelid region, the average of all distance values ​​corresponding to the eyelid region in the third distance value set is calculated to obtain the initial distance value set.

6. The method according to claim 2, characterized in that, The distance data further includes: a set of historical distance values ​​between the smart glasses and the at least one eyelid region when the eyeball was in a historical position before the eyeball detection command was acquired; determining the target gaze direction based on the first set of distance values ​​in the distance data includes: Calculate the difference between the distance value corresponding to each eyelid region in the first distance value set and the distance value corresponding to the historical distance value set to obtain a second distance difference set; When the second distance difference set satisfies the third preset condition, the target gaze direction is determined based on the historical gaze direction when the eyeball is in the historical position, combined with the preset eyeball change angle corresponding to the third preset condition.

7. The method according to claim 1, characterized in that, The target motion state includes the target motion trajectory of the eyeball; determining the target motion state of the eyeball based on the distance data includes: At least one first distance sequence is determined from the distance data, each first distance sequence corresponding to one eyelid region. The first distance sequence includes the distance values ​​between the corresponding eyelid region and the smart glasses at multiple measurement times, and the distance values ​​in the first distance sequence are arranged in chronological order. The target trajectory is determined based on the trend of distance values ​​changing over time in each of the first distance sequences.

8. The method according to claim 7, characterized in that, Determining the target trajectory based on the trend of distance values ​​changing over time in each of the first distance sequences includes: The distance values ​​in each of the first distance sequences are filtered to obtain the second distance sequence corresponding to each of the first distance sequences. The difference between any two adjacent distance values ​​in the second distance sequence is greater than a second preset distance difference threshold. The target trajectory is determined based on the trend of distance values ​​changing over time in each of the second distance sequences.

9. The method according to claim 7, characterized in that, Each distance value in the first distance sequence corresponds to a holding duration, which is the duration for which the eyeball remains at the position corresponding to the distance value; Determining the target trajectory based on the trend of distance values ​​changing over time in each of the first distance sequences includes: Delete the first type of distance value in each of the first distance sequences to obtain a third distance sequence corresponding to each of the first distance sequences. The retention time corresponding to the first type of distance value is less than a first preset time threshold, and the difference between the distance values ​​located before and after the first type of distance value in the first distance sequence is less than a third preset distance difference threshold. The target trajectory is determined based on the trend of distance values ​​changing over time in each of the third distance sequences.

10. An electronic device, characterized in that, include: Memory and processor; Memory and processor are coupled; The memory is used to store instructions that can be executed by the processor; When the processor executes the instructions, it performs the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-9.

12. A computer program product, characterized in that, The computer program product includes computer program instructions that, when executed, implement the method as described in any one of claims 1-9.

Images