Strabismus detection method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a strabismus detection method and device, electronic equipment and storage medium, the method comprising: acquiring a left eye image and a right eye image of a user when the user's left eye and right eye respectively fixate on a first set visual marker located at the center of a display screen; determining a first left eye gaze direction vector based on the left eye image and a first right eye gaze direction vector based on the right eye image; determining a first reference vector pointing from a reference point to the first set visual marker; determining the absolute values of the angles between the first left eye gaze direction vector and the first reference vector and between the first right eye gaze direction vector and the first reference vector; determining a strabismus detection compensation angle of the user's eyes in the left-right movement direction based on the two absolute values; and determining a final strabismus angle of the user's eyes in the left-right movement direction based on the strabismus detection compensation angle and the strabismus angle of the user's eyes in the left-right movement direction.

Description

Technical Field

[0001] This disclosure relates to the field of user eye detection technology, and more specifically, to a strabismus detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Head-mounted displays combined with eye-tracking technology are widely used in virtual reality, augmented reality, and medical diagnostics. In ophthalmological testing, using head-mounted displays for strabismus detection offers advantages in convenience and repeatability, providing users with a preliminary assessment of visual function.

[0003] In practical applications, strabismus detection schemes in related technologies do not take into account the differences in the wearing position of the head-mounted device each time, which causes changes in the relative position relationship between the camera and the eyeball, thus affecting the accuracy of strabismus detection. Summary of the Invention

[0004] One objective of this invention is to provide a new technical solution for strabismus detection, taking into account the impact of differences in the wearing position of the head-mounted device on strabismus detection each time.

[0005] According to a first aspect of the present invention, a strabismus detection method is provided, comprising: Acquire images of the user's left and right eyes when the user's left and right eyes are focused on a first set visual marker, respectively, wherein the first set visual marker is located at the center of the display screen; A first left-eye gaze direction vector is determined based on the user's left-eye image, and a first right-eye gaze direction vector is determined based on the user's right-eye image; A first reference vector is determined from the reference point to the first set visual mark, wherein the reference point is the center of the line connecting the centers of the user's left and right eyeballs; Determine the absolute value of the angle between the first left eye gaze direction vector and the first reference vector, and the absolute value of the angle between the first right eye gaze direction vector and the first reference vector, respectively. Based on the absolute values ​​of the two included angles, determine the strabismus detection compensation angles for the user's eyes in the left and right movement directions; Based on the strabismus detection compensation angle and the strabismus angle of the user's eyes in the left and right movement direction, the final strabismus angle of the user's eyes in the left and right movement direction is determined.

[0006] Optionally, determining the first left-eye gaze direction vector based on the user's left-eye image and the first right-eye gaze direction vector based on the user's right-eye image includes: Based on the user's left eye image, determine the left eye rotation information when the user's left eye is focused on the first set visual mark, and based on the user's right eye image, determine the right eye rotation information when the user's right eye is focused on the first set visual mark. Obtain a set gaze direction vector, wherein the set gaze direction vector is the gaze vector when an eye without strabismus views a set visual mark displayed in the display area directly in front of it; Based on the left eye rotation information and the set gaze direction vector, a first left eye gaze direction vector is determined, and based on the right eye rotation information and the set gaze direction vector, a first right eye gaze direction vector is determined.

[0007] Optionally, determining the first reference vector pointing from the reference point to the first defined visual marker includes: The spatial location information of the center of the left eyeball is determined based on the user's left eye image, and the spatial location information of the center of the right eyeball is determined based on the user's right eye image. Based on the spatial location information of the center of the left eyeball and the spatial location information of the center of the right eyeball, the spatial location information of the reference point is determined; The first reference vector is determined based on the spatial location information of the reference point and the spatial location information of the first set visual marker.

[0008] Optionally, the oblique angle of the user's eyes in the left-right movement direction is the oblique angle determined when the user gazes at the second set visual marker displayed at the set position on the display screen.

[0009] Optionally, the method further includes: Acquire the user's left eye image and right eye image when the user is gazing at the second set visual marker; A second left-eye gaze direction vector is determined based on the user's left-eye image when the user gazes at the second set visual mark, and a second right-eye gaze direction vector is determined based on the user's right-eye image when the user gazes at the second set visual mark; Determine a second reference vector pointing from the reference point to the second set visual mark; The first projection angle of the angle between the second left eye gaze direction vector and the second reference vector on the first reference plane is determined, and the second projection angle of the angle between the second right eye gaze direction vector and the second reference vector on the first reference plane is determined. Based on the first projection angle and the second projection angle, determine the oblique angle of the user's eyes in the left and right movement direction of the user's eyes.

[0010] Optionally, the method further includes: Acquire the user's left eye image and right eye image when the user is gazing at the second set visual marker; A second left-eye gaze direction vector is determined based on the user's left-eye image when the user gazes at the second set visual mark, and a second right-eye gaze direction vector is determined based on the user's right-eye image when the user gazes at the second set visual mark; Determine a second reference vector pointing from the reference point to the second set visual mark; The angle between the second left eye gaze direction vector and the second reference vector is determined as a third projection angle on the second reference plane, and the angle between the second right eye gaze direction vector and the second reference vector is determined as a fourth projection angle on the second reference plane. Based on the third projection angle and the fourth projection angle, the oblique angle of the user's eyes in the vertical movement direction is determined.

[0011] Optionally, the method further includes: Each time the head-mounted device is re-worn onto the user's head, the first left-eye gaze direction vector, the first right-eye gaze direction vector, and the first reference vector are re-determined, and the absolute values ​​of the two included angles are recalculated to update the strabismus detection compensation angle.

[0012] According to a second aspect of the present invention, a strabismus detection device is provided, comprising: The acquisition module is used to acquire the user's left eye image and the user's right eye image when the user's left eye and right eye are respectively looking at the first set visual mark, wherein the first set visual mark is located at the center of the display screen; A gaze direction vector determination module is used to determine a first left eye gaze direction vector based on the user's left eye image, and to determine a first right eye gaze direction vector based on the user's right eye image. A reference vector determination module is used to determine a first reference vector pointing from a reference point to the first set visual mark, wherein the reference point is the center of the line connecting the centers of the user's left and right eyeballs; An absolute angle determination module is used to determine the absolute angle between the first left eye gaze direction vector and the first reference vector, and the absolute angle between the first right eye gaze direction vector and the first reference vector, respectively. The compensation angle determination module is used to determine the strabismus detection compensation angle of the user's eyes in the left and right movement direction based on the absolute value of the two included angles; The compensation module is used to determine the final angle of the user's eyes in the left and right movement direction based on the strabismus detection compensation angle and the angle of the user's eyes in the left and right movement direction.

[0013] According to a third aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program for controlling the processor to operate in order to perform the method according to any one of the first aspects.

[0014] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the method according to any one of the first aspects.

[0015] This disclosure provides a strabismus detection method. After each use of a head-mounted device, the strabismus detection compensation angle is automatically determined by the user's gaze at the visual marker at the center of the display screen. This compensates for differences in wearing position in subsequent strabismus detections and improves detection accuracy.

[0016] The features and advantages of the embodiments of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of these embodiments.

[0018] Figure 1 This is a schematic diagram of the structure of an electronic device to which the methods of the embodiments of this disclosure can be applied.

[0019] Figure 2 This is a flowchart illustrating a strabismus detection method according to some embodiments.

[0020] Figure 3 This is a flowchart illustrating the determination of the strabismus detection compensation angle of a user's eyes in the left-right movement direction, according to some embodiments.

[0021] Figure 4 This is a schematic block diagram of a strabismus detection device according to some embodiments.

[0022] Figure 5 This is a block schematic diagram of an electronic device according to one embodiment. Detailed Implementation

[0023] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0024] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the scope of this disclosure or its application or use.

[0025] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0026] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0027] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0028] This disclosure relates to a strabismus detection scheme. Figure 1 This is a schematic diagram of the structure of an electronic device capable of applying the strabismus detection method of the present disclosure. The electronic device 1000 is a device capable of running a computer program, which may be a local application installed on the head-mounted device, or a web application, lightweight application, or applet, etc., without limitation herein. The electronic device 1000 may be a VR head-mounted device, an AR head-mounted device, or a terminal device (e.g., a mobile phone, a tablet computer), without limitation herein.

[0029] like Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display screen 1500, a camera 1600, etc. Figure 1 The hardware configuration shown is illustrative only and is not intended to limit this disclosure, its application, or its use.

[0030] The processor 1100 is used to execute computer programs, which can be written using instruction sets of architectures such as x86, Arm, RISC, MIPS, and SSE. The memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. The interface device 1300 includes, for example, a USB interface, a network cable interface, and a headphone jack. The communication device 1400 is capable of wired or wireless communication. The communication device 1400 may include at least one short-range communication module, such as any module for short-range wireless communication based on short-range wireless communication protocols such as Hilink, WiFi (IEEE 802.11), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, and LiFi. The communication device 1400 may also include a long-range communication module, such as any module for WLAN, GPRS, or 2G / 3G / 4G / 5G long-range communication.

[0031] When this electronic device is used for strabismus detection, the uncertainty of the wearing position of the head-mounted device will introduce systematic errors in the strabismus angle measurement, reducing the reliability of the detection results.

[0032] To address the aforementioned technical issues, this disclosure provides a strabismus detection method. After each use of the head-mounted device, the strabismus detection compensation angle is automatically determined by the user's gaze at the visual marker at the center of the display screen. This compensates for differences in wearing position during subsequent strabismus detection, thereby improving detection accuracy.

[0033] The following combination Figure 1 The illustrated electronic device illustrates the strabismus detection method provided in this disclosure.

[0034] <Example 1> Figure 2 This is a flowchart illustrating a strabismus detection method according to some embodiments. The strabismus detection method consists of... Figure 1 The illustrated electronic device 1000 is implemented.

[0035] like Figure 2 As shown, the strabismus detection method of this embodiment may include the following steps S210 to S260.

[0036] Step S210: Obtain the user's left eye image and right eye image when the user's left eye and right eye are respectively looking at the first set visual mark, wherein the first set visual mark is located at the center of the display screen.

[0037] When the electronic device is a terminal device, the terminal device establishes a communication connection with the head-mounted device. The head-mounted device sends the user's left eye image and right eye image that it has collected to the terminal device so that the terminal device can perform subsequent processing.

[0038] When the first set visual marker is displayed in the center of the control display screen, the user's right eye is covered, and the camera captures an image of the user's left eye. When the user's left eye is covered, the camera captures an image of the user's right eye.

[0039] The first set of visual markers can be graphic markers such as crosses, dots, squares, and cartoon icons.

[0040] The display screen and camera are both components of the head-mounted device.

[0041] The camera consists of two cameras: a left-eye camera and a right-eye camera. The left-eye camera captures images of the user's left eye. The right-eye camera captures images of the user's right eye.

[0042] Step S220: Determine the first left eye gaze direction vector based on the user's left eye image, and determine the first right eye gaze direction vector based on the user's right eye image.

[0043] In some embodiments, step S220 may include steps S2211 to S2213.

[0044] Step S2211: Based on the user's left eye image, determine the left eye rotation information when the user's left eye is focused on the first set visual mark, and based on the user's right eye image, determine the right eye rotation information when the user's right eye is focused on the first set visual mark.

[0045] Based on eye-tracking technology, the system analyzes the user's left eye image to capture left eye movement data and determine left eye rotation information. Similarly, it analyzes the user's right eye image to capture right eye movement data and determine right eye rotation information. Eye rotation information can be represented using quaternions, rotation matrices, or Euler angles to describe the eye's rotational state in three-dimensional space.

[0046] Step S2212: Obtain the set gaze direction vector, wherein the set gaze direction vector is the gaze vector when the eye without strabismus looks at the set visual mark displayed in the display area directly in front.

[0047] The gaze direction vector is set to a preset vector, for example, calibrated when the head-mounted device leaves the factory. This preset gaze direction vector represents the gaze direction vector when the eye is viewing a visual marker directly in front of it in a normal state (without strabismus and when the head-mounted device is worn in an ideal position).

[0048] Step S2213: Determine the first left eye gaze direction vector based on the left eye rotation information and the set gaze direction vector, and determine the first right eye gaze direction vector based on the right eye rotation information and the set gaze direction vector.

[0049] The embodiments disclosed herein can accurately calculate the actual gaze direction vector when a user gazes at a visual marker based on eye rotation information and a preset gaze direction vector, providing a reliable data foundation for subsequent strabismus detection.

[0050] Step S230: Determine the first reference vector pointing from the reference point to the first set visual mark, wherein the reference point is the center of the line connecting the centers of the user's left and right eyeballs.

[0051] In some embodiments, step S230 may include steps S2311 to S2313.

[0052] Step S2311: Determine the spatial location information of the center of the left eyeball based on the user's left eye image, and determine the spatial location information of the center of the right eyeball based on the user's right eye image.

[0053] Based on the user's left eye image, the outline of the left eyeball is extracted, and the coordinates of the center of the left eyeball in three-dimensional space are calculated by fitting an eyeball model. Based on the user's right eye image, the outline of the right eyeball is extracted, and the coordinates of the center of the right eyeball in three-dimensional space are calculated by fitting an eyeball model.

[0054] Step S2312: Determine the spatial position information of the reference point based on the spatial position information of the center of the left eyeball and the spatial position information of the center of the right eyeball.

[0055] The midpoint of the line connecting the center of the left eyeball and the center of the right eyeball is used as the reference point. The spatial position information of the reference point is calculated using the spatial position information of the center of the left eyeball and the center of the right eyeball.

[0056] Step S2313: Determine the first reference vector based on the spatial location information of the reference point and the spatial location information of the first set visual marker.

[0057] The spatial location information of the first visual marker is pre-stored and can be obtained directly.

[0058] The embodiments disclosed herein can accurately determine the reference point and reference vector, providing a precise reference for calculating the angle between the gaze direction vector and the reference vector, thereby improving the accuracy of the calculation of the strabismus detection compensation angle.

[0059] Step S240: Determine the absolute value of the angle between the first left eye gaze direction vector and the first reference vector, and the absolute value of the angle between the first right eye gaze direction vector and the first reference vector.

[0060] The angle between the first left eye gaze direction vector and the first reference vector, and the angle between the first right eye gaze direction vector and the first reference vector are calculated using the vector dot product formula.

[0061] Step S250: Determine the strabismus detection compensation angle of the user's eyes in the left and right movement direction based on the absolute values ​​of the two included angles.

[0062] The strabismus detection compensation angle for the user's eyes in the left-right movement direction is the sum of the absolute values ​​of the two included angles. The strabismus detection compensation angle is determined based on the following formula:

[0063] in, For strabismus detection compensation angle, The absolute value of the angle between the first left eye gaze direction vector and the first reference vector. It is the absolute value of the angle between the first right eye gaze direction vector and the first reference vector.

[0064] For users with strabismus, their eyes already deviate from the normal binocular position in their natural state. Therefore, an alternating occlusion method is chosen to allow each eye to return to its standard position (the position the eyes should be in when not strabismus, i.e., the position normal eyes should be in when focusing on an object directly in front). The alternating occlusion method obscures the visual field of one eye, forcing the unobscured eye to focus on the target. The strabismic eye of the user will be forced to return to the correct eye position to focus on the target, thus obtaining the standard eye position data of the user's eyes. This ensures that the subsequent strabismus detection compensation angle only reflects the influence of the head-mounted device wearing position on the strabismus angle measurement, and is effective, without including the user's strabismus information.

[0065] Ideally, when the user's eyes are focused on the visual marker at the center of the display screen, assuming the user does not have strabismus, the first left-eye gaze direction vector and the first right-eye gaze direction vector are symmetrical about a plane determined by the vertical and horizontal directions of the head-mounted device. Due to differences in the wearing position of the head-mounted device, the sum of the absolute values ​​of the two included angles is used as the strabismus detection compensation angle for the user's eyes in the left-right movement direction, which is used to correct subsequent strabismus angle measurements.

[0066] Figure 3 A flowchart illustrating the determination of the strabismus detection compensation angle of a user's eyes in the left-right movement direction according to an embodiment of the present invention is shown. Figure 3 As shown, steps S210 to S250 are repeated three times to obtain the strabismus detection compensation angles of the three users' eyes in the left-right movement direction. The average of these three strabismus detection compensation angles in the left-right movement direction is then calculated to obtain the final strabismus detection compensation angle, thereby improving the detection accuracy.

[0067] Step S260: Determine the final angle of the user's eyes in the left and right movement direction based on the strabismus detection compensation angle and the angle of the user's eyes in the left and right movement direction.

[0068] The angle of view of the user's eyes in the left and right movement direction is the angle of view determined when the user is looking at the second set visual marker displayed on the screen at the set position.

[0069] When the user gazes at the second set visual marker, the eye images of the user's two eyes are simultaneously captured, and the oblique angle of the user's eyes in the left and right movement direction is determined based on the eye images of the user's two eyes.

[0070] After obtaining the strabismus angle of the user's eyes in the left-right movement direction, this angle is subtracted from the strabismus detection compensation angle to obtain the final strabismus angle of the user's eyes in the left-right movement direction. This final strabismus angle of the user's eyes in the left-right movement direction eliminates the influence of differences in the wearing position of the head-mounted device and more accurately reflects the user's true degree of strabismus. The final horizontal strabismus angle is determined based on the following formula.

[0071] in, For the final horizontal oblique perspective, The oblique angle of the eyes in the left and right movement direction. This is the compensation angle for strabismus detection.

[0072] The final angle of view of the user's eyes in the left-right movement direction is the angle of view when the user is looking at the second set visual mark. When the display position of the second set visual mark on the display screen changes, the final angle of view of the user's eyes in the left-right movement direction will also change.

[0073] The larger the absolute value of the final angle of strabismus in the left-right movement direction of a user's eyes, the greater the degree of strabismus in that direction.

[0074] The embodiments of this disclosure can automatically determine the strabismus detection compensation angle by observing the visual marker at the center of the display screen after each time the head-mounted device is worn, thereby compensating for differences in wearing position in subsequent strabismus detection and improving detection accuracy.

[0075] In some embodiments, each time the head-mounted device is re-worn onto the user's head, the first left-eye gaze direction vector, the first right-eye gaze direction vector, and the first reference vector are re-determined, and the absolute values ​​of the two included angles are recalculated to update the strabismus detection compensation angle.

[0076] When a user removes and then puts on the head-mounted device again, the device's position may change, altering the relative position of the camera and the eyeball. To ensure accurate strabismus detection, upon detecting a re-wearing event (e.g., device movement detected by sensors or user interaction), the aforementioned strabismus detection compensation angle update operation is automatically executed. This ensures that strabismus detection is always based on the latest wearing position, improving the reliability and accuracy of the detection.

[0077] <Example 2> In some embodiments of this disclosure, determining the oblique angle of the user's eyes in the left-right movement direction includes steps S310 to S350.

[0078] Step S310: Obtain the user's left eye image and right eye image when the user is looking at the second set visual marker.

[0079] The first and second visual markers can be the same or different markers. The second visual marker can be a graphic marker such as a cross, dot, square, or cartoon icon.

[0080] The second set visual mark can be displayed at any set position on the display screen, such as the center, top, bottom, left, right, upper left, lower left, upper right, lower right, etc.

[0081] Step S320: Determine the second left eye gaze direction vector based on the user's left eye image when the user gazes at the second set visual mark, and determine the second right eye gaze direction vector based on the user's right eye image when the user gazes at the second set visual mark.

[0082] The determination methods for the second left-eye gaze direction vector and the second right-eye gaze direction vector can refer to the determination methods for the first left-eye gaze direction vector and the first right-eye gaze direction vector, and will not be elaborated on here.

[0083] Step S330: Determine a second reference vector pointing from the reference point to the second set visual mark.

[0084] The method for determining the second reference vector can refer to the method for determining the first reference vector described above, and will not be elaborated further here.

[0085] Step S340: Determine the first projection angle of the angle between the second left eye gaze direction vector and the second reference vector on the first reference plane, and the second projection angle of the angle between the second right eye gaze direction vector and the second reference vector on the first reference plane.

[0086] The first reference plane is a horizontal plane that passes through a reference point and is jointly determined by the left-right and front-back directions of the head-mounted device.

[0087] Step S350: Determine the oblique viewing angle of the user's eyes in the left and right movement direction based on the first projection angle and the second projection angle.

[0088] Subtracting the first and second projection angles yields the oblique angle of the user's eyes in the left-right movement direction. The oblique angle of the user's eyes in the left-right movement direction is then determined based on the following formula.

[0089] in, The oblique viewing angle of the user's eyes in the left and right movement directions. The first projection angle, This is the included angle of the second projection.

[0090] The embodiments disclosed herein can project the gaze direction vector and reference vector in three-dimensional space onto a horizontal reference plane using a projection method, thereby accurately calculating the oblique angle in the horizontal direction and providing reliable measurement results for strabismus detection.

[0091] In some embodiments of this disclosure, determining the oblique angle of the user's eyes in the vertical movement direction of the user's eyes includes steps S410 to S450.

[0092] Step S410: Obtain the user's left eye image and right eye image when the user is looking at the second set visual marker.

[0093] The first and second visual markers can be the same or different markers. The second visual marker can be a graphic marker such as a cross, dot, square, or cartoon icon.

[0094] The second set visual mark can be displayed at any set position on the display screen, such as the center, top, bottom, left, right, upper left, lower left, upper right, lower right, etc.

[0095] Step S420: Determine the second left eye gaze direction vector based on the user's left eye image when the user gazes at the second set visual mark, and determine the second right eye gaze direction vector based on the user's right eye image when the user gazes at the second set visual mark.

[0096] The determination methods for the second left-eye gaze direction vector and the second right-eye gaze direction vector can refer to the determination methods for the first left-eye gaze direction vector and the first right-eye gaze direction vector, and will not be elaborated on here.

[0097] Step S430: Determine a second reference vector pointing from the reference point to the second set visual mark.

[0098] The method for determining the second reference vector can refer to the method for determining the first reference vector described above, and will not be elaborated further here.

[0099] Step S440: Determine the third projection angle of the angle between the second left eye gaze direction vector and the second reference vector on the second reference plane, and the fourth projection angle of the angle between the second right eye gaze direction vector and the second reference vector on the second reference plane.

[0100] The second reference plane is a vertical plane that passes through the reference point and is perpendicular to the first reference plane, and is determined by the up-down and front-back directions of the head-mounted device.

[0101] Step S450: Determine the oblique viewing angle of the user's eyes in the vertical movement direction based on the third projection angle and the fourth projection angle.

[0102] Subtracting the third projection angle from the first projection angle yields the oblique angle of the user's eyes in the vertical movement direction. The oblique angle of the user's eyes in the vertical movement direction is then determined based on the following formula.

[0103] in, The oblique viewing angle of the user's eyes in the vertical movement direction. The included angle of the third projection. This is the included angle of the fourth projection.

[0104] The angle of view of the user's eyes in the vertical direction is the angle of view when the user is looking at the second set visual mark. When the display position of the second set visual mark on the display screen changes, the angle of view of the user's eyes in the vertical direction will also change.

[0105] The larger the absolute value of the angle of strabismus in the vertical movement direction of a user's eyes, the greater the degree of strabismus in that direction.

[0106] It should be noted that the user's skew angle in the vertical movement direction does not need to be calibrated.

[0107] The embodiments disclosed herein can project the gaze direction vector and reference vector in three-dimensional space onto a vertical reference plane using a projection method, thereby accurately calculating the oblique angle in the vertical direction and providing reliable measurement results for strabismus detection.

[0108] In some embodiments, when there are multiple second-defined visual markers at multiple defined positions on the display screen, the oblique viewing angles of the user's eyes in the left-right and up-down directions are determined for each defined position, and the oblique viewing angles in the left-right and up-down directions corresponding to different defined positions are displayed. For example, the defined positions include the center, top, bottom, left, right, upper left, lower left, upper right, and lower right of the display screen. The oblique viewing angles in the left-right and up-down directions determined in these nine positions are filled into a nine-square grid and displayed intuitively to the user.

[0109] <Example 3> One embodiment of this disclosure provides a strabismus detection device. According to... Figure 4 As shown, the strabismus detection device includes an acquisition module 410, a gaze direction vector determination module 420, a reference vector determination module 430, an absolute angle determination module 440, a compensation angle determination module 450, and a compensation module 460.

[0110] The acquisition module 410 is used to acquire the user's left eye image and the user's right eye image when the user's left eye and right eye are respectively looking at the first set visual mark, wherein the first set visual mark is located at the center of the display screen.

[0111] The gaze direction vector determination module 420 is used to determine a first left eye gaze direction vector based on the user's left eye image and a first right eye gaze direction vector based on the user's right eye image.

[0112] The reference vector determination module 430 is used to determine the first reference vector pointing from the reference point to the first set visual mark, wherein the reference point is the center of the line connecting the centers of the user's left and right eyeballs.

[0113] The absolute angle determination module 440 is used to determine the absolute angle between the first left eye gaze direction vector and the first reference vector, and the absolute angle between the first right eye gaze direction vector and the first reference vector, respectively.

[0114] The compensation angle determination module 450 is used to determine the strabismus detection compensation angle of the user's eyes in the left and right movement direction based on the absolute values ​​of the two included angles.

[0115] The compensation module 460 is used to determine the final angle of the user's eyes in the left and right movement direction based on the strabismus detection compensation angle and the angle of the user's eyes in the left and right movement direction.

[0116] In some embodiments, the gaze direction vector determination module 420 is configured to determine, based on the user's left eye image, the left eye rotation information when the user's left eye is gazing at a first set visual marker, and based on the user's right eye image, the right eye rotation information when the user's right eye is gazing at a second set visual marker; obtain a set gaze direction vector, wherein the set gaze direction vector is the gaze vector when an eye without strabismus views a set visual marker displayed in the display area directly in front; determine a first left eye gaze direction vector based on the left eye rotation information and the set gaze direction vector, and determine a first right eye gaze direction vector based on the right eye rotation information and the set gaze direction vector.

[0117] In some embodiments, the reference vector determination module 430 is used to determine the spatial position information of the center of the left eyeball based on the user's left eye image, and to determine the spatial position information of the center of the right eyeball based on the user's right eye image; to determine the spatial position information of a reference point based on the spatial position information of the center of the left eyeball and the spatial position information of the center of the right eyeball; and to determine a first reference vector based on the spatial position information of the reference point and the spatial position information of the first set visual marker.

[0118] In some embodiments, the angle of view of the user's eyes in the left-right movement direction is the angle of view determined when the user gazes at the second set visual marker displayed at the set position on the display screen.

[0119] In some embodiments, the device further includes a slant angle determination module. The slant angle determination module is configured to acquire an image of the user's left eye and an image of the user's right eye when the user is gazing at the second set visual mark; determine a second left-eye gaze direction vector based on the user's left-eye image when the user is gazing at the second set visual mark, and determine a second right-eye gaze direction vector based on the user's right-eye image when the user is gazing at the second set visual mark; determine a second reference vector pointing from a reference point to the second set visual mark; determine a first projection angle on a first reference plane of the angle between the second left-eye gaze direction vector and the second reference vector, and a second projection angle on a first reference plane of the angle between the second right-eye gaze direction vector and the second reference vector; and determine the slant angle of the user's eyes in the left-right movement direction based on the first projection angle and the second projection angle.

[0120] In some embodiments, the oblique angle determination module is further configured to acquire an image of the user's left eye and an image of the user's right eye when the user is looking at the second set visual mark; determine a second left eye gaze direction vector based on the image of the user's left eye when the user is looking at the second set visual mark, and determine a second right eye gaze direction vector based on the image of the user's right eye when the user is looking at the second set visual mark; determine a second reference vector pointing from a reference point to the second set visual mark; determine a third projection angle on the second reference plane of the angle between the second left eye gaze direction vector and the second reference vector, and a fourth projection angle on the second reference plane of the angle between the second right eye gaze direction vector and the second reference vector; and determine the oblique angle of the user's eyes in the vertical movement direction based on the third projection angle and the fourth projection angle.

[0121] In some embodiments, each time the head-mounted device is re-worn onto the user's head, the first left-eye gaze direction vector, the first right-eye gaze direction vector, and the first reference vector are re-determined, and the absolute values ​​of the two included angles are recalculated to update the strabismus detection compensation angle.

[0122] <Example 4> This embodiment provides an electronic device. Figure 5 A schematic diagram of the hardware structure of the electronic device is shown.

[0123] like Figure 5As shown, the electronic device 1000 includes a processor 1100 and a memory 1200. The memory can be used to store computer programs, and the processor can be used to retrieve the computer programs from the memory to execute any method embodiment of this disclosure. The processor can be one or more, and these processors can execute instructions individually or jointly. Similarly, the memory can be one or more, and these memories can store the aforementioned computer programs individually or jointly.

[0124] The processor 1100 is used to execute computer instructions, which can be written using instruction sets of architectures such as x86, Arm, RISC, MIPS, and SSE. The memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as hard disks, etc., and is not limited here.

[0125] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the audio storage method in any embodiment of this disclosure. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited thereto; it may also be a temporary storage medium.

[0126] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0127] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0128] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0129] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of the present invention.

[0130] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0131] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0132] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0134] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for detecting strabismus, characterized in that, include: The system acquires an image of the user's left eye when the user's right eye is covered and the user's right eye when the user's left eye is covered and the user's right eye when the user's right eye is covered and ... is covered and the user's right eye is covered and the user's right eye is covered and the user's right eye is covered, wherein the Determining a first left-eye gaze direction vector based on the user's left-eye image and a first right-eye gaze direction vector based on the user's right-eye image includes: determining left-eye rotation information when the user's left eye gazes at the first set visual marker based on the user's left-eye image, and determining right-eye rotation information when the user's right eye gazes at the first set visual marker based on the user's right-eye image; obtaining a set gaze direction vector, wherein the set gaze direction vector is the gaze vector when an eye without strabismus views the set visual marker displayed in the display area directly in front; determining a first left-eye gaze direction vector based on the left-eye rotation information and the set gaze direction vector, and determining a first right-eye gaze direction vector based on the right-eye rotation information and the set gaze direction vector. A first reference vector is determined from the reference point to the first set visual mark, wherein the reference point is the center of the line connecting the centers of the user's left and right eyeballs; Determine the absolute value of the angle between the first left eye gaze direction vector and the first reference vector, and the absolute value of the angle between the first right eye gaze direction vector and the first reference vector, respectively. The sum of the absolute values ​​of the two included angles is determined as the strabismus detection compensation angle for the user's eyes in the left and right movement directions; Based on the strabismus detection compensation angle and the strabismus angle of the user's eyes in the left and right movement direction, the final strabismus angle of the user's eyes in the left and right movement direction is determined.

2. The method according to claim 1, characterized in that, Determining the first reference vector pointing from the reference point to the first set visual mark includes: The spatial location information of the center of the left eyeball is determined based on the user's left eye image, and the spatial location information of the center of the right eyeball is determined based on the user's right eye image. Based on the spatial location information of the center of the left eyeball and the spatial location information of the center of the right eyeball, the spatial location information of the reference point is determined; The first reference vector is determined based on the spatial location information of the reference point and the spatial location information of the first set visual marker.

3. The method according to claim 1, characterized in that, The angle of view of the user's eyes in the left and right movement direction is the angle of view determined when the user looks at the second set visual marker displayed at the set position on the display screen.

4. The method according to claim 3, characterized in that, The method further includes: Acquire the user's left eye image and right eye image when the user is gazing at the second set visual marker; A second left-eye gaze direction vector is determined based on the user's left-eye image when the user gazes at the second set visual mark, and a second right-eye gaze direction vector is determined based on the user's right-eye image when the user gazes at the second set visual mark; Determine a second reference vector pointing from the reference point to the second set visual mark; The first projection angle of the angle between the second left eye gaze direction vector and the second reference vector on the first reference plane is determined, and the second projection angle of the angle between the second right eye gaze direction vector and the second reference vector on the first reference plane is determined. Based on the first projection angle and the second projection angle, the oblique viewing angle of the user's eyes in the left and right movement direction is determined.

5. The method according to claim 3, characterized in that, The method further includes: Acquire the user's left eye image and right eye image when the user is gazing at the second set visual marker; A second left-eye gaze direction vector is determined based on the user's left-eye image when the user gazes at the second set visual mark, and a second right-eye gaze direction vector is determined based on the user's right-eye image when the user gazes at the second set visual mark; Determine a second reference vector pointing from the reference point to the second set visual mark; The angle between the second left eye gaze direction vector and the second reference vector is determined as a third projection angle on the second reference plane, and the angle between the second right eye gaze direction vector and the second reference vector is determined as a fourth projection angle on the second reference plane. Based on the third projection angle and the fourth projection angle, the oblique angle of the user's eyes in the vertical movement direction is determined.

6. The method according to claim 1, characterized in that, The method further includes: Each time the head-mounted device is re-worn onto the user's head, the first left-eye gaze direction vector, the first right-eye gaze direction vector, and the first reference vector are re-determined, and the absolute values ​​of the two included angles are recalculated to update the strabismus detection compensation angle.

7. A strabismus detection device, characterized in that, include: The acquisition module is used to acquire the user's left eye image when the user's right eye is covered and the user's right eye image when the user's left eye is looking at the first set visual mark, and the user's right eye image when the user's left eye is covered and the user's right eye is looking at the first set visual mark, wherein the first set visual mark is located at the center of the display screen. A gaze direction vector determination module is used to determine a first left-eye gaze direction vector based on the user's left-eye image and a first right-eye gaze direction vector based on the user's right-eye image, including: determining left-eye rotation information when the user's left eye gazes at the first set visual mark based on the user's left-eye image, and determining right-eye rotation information when the user's right eye gazes at the first set visual mark based on the user's right-eye image; obtaining a set gaze direction vector, wherein the set gaze direction vector is the gaze vector when an eye without strabismus views a set visual mark displayed in a display area directly in front; determining a first left-eye gaze direction vector based on the left-eye rotation information and the set gaze direction vector, and determining a first right-eye gaze direction vector based on the right-eye rotation information and the set gaze direction vector; A reference vector determination module is used to determine a first reference vector pointing from a reference point to the first set visual mark, wherein the reference point is the center of the line connecting the centers of the user's left and right eyeballs; An absolute angle determination module is used to determine the absolute angle between the first left eye gaze direction vector and the first reference vector, and the absolute angle between the first right eye gaze direction vector and the first reference vector, respectively. The compensation angle determination module is used to determine the sum of the absolute values ​​of the two included angles as the strabismus detection compensation angle of the user's eyes in the left and right movement directions; The compensation module is used to determine the final angle of the user's eyes in the left and right movement direction based on the strabismus detection compensation angle and the angle of the user's eyes in the left and right movement direction.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program for controlling the processor to operate in order to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images