Eye movement tracking calibration system, method and device

Abstract

The invention discloses an eye movement tracking calibration system, method and device, the eye movement tracking calibration system comprises a head-mounted device, a display device and a calibration device, and the head-mounted device at least comprises a near-eye camera and a scene camera; the display device is used for displaying the head calibration point on a screen of the display device according to the position information of the head calibration point under the condition that the user wears the head-mounted device; the calibration device is used for controlling the display device to output a plurality of picture calibration points through the screen in response to a confirmation instruction for the head calibration point; the near-to-eye camera is used for shooting human eyes of a user wearing the head-mounted device when the display device outputs a plurality of picture calibration points through a screen to obtain a human eye shooting image of each picture calibration point; and the calibration device is also used for obtaining a calibration result of the eye movement tracking calibration system according to the human eye shooting image of each picture calibration point. According to the method and the device, the head of the user is aligned with the target area through head calibration, so that the calibration accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and more specifically, to an eye tracking calibration system, method and device. Background Art

[0002] Eye tracking technology is generally used to estimate the optical axis of the human eye so as to output images that are directly facing the human eye based on the optical axis of the human eye. However, where the human eye actually looks is determined by the visual axis, which generally has a fixed deviation from the optical axis, and the deviation is different for each person, so eye tracking calibration is required.

[0003] Existing eye tracking calibration methods generally use calibration images or videos with features, requiring users to look at the appearance or change of features while watching the displayed content, calculate the visual axis based on the location of the features, and calculate the optical axis based on the pupil position of the human eye. The calibration result is obtained based on the visual axis and the optical axis.

[0004] However, using this method, the accuracy of eye tracking calibration is low and the user experience is poor. Summary of the invention

[0005] The present application proposes an eye tracking calibration system, method and device to improve the above-mentioned defects.

[0006] In a first aspect, an embodiment of the present application provides an eye tracking calibration system, which includes at least: a head-mounted device, a display device, and a calibration device, wherein the head-mounted device includes at least a near-eye camera and a scene camera; wherein:

[0007] The display device is used to display the head calibration points on the screen of the display device according to the position information of the head calibration points when the user wears the head mounted device, wherein the position information of the head calibration points is obtained by converting the intersection between the main optical axis of the scene camera and the screen into the screen coordinate system where the screen is located;

[0008] The calibration device is used to control the display device to sequentially output a plurality of picture calibration points through a screen in response to a confirmation instruction for the head calibration point; the confirmation instruction is generated when the head calibration point is located in a target area for a target time or when a trigger operation for the head calibration point is received by a user wearing the head mounted device;

[0009] The near-eye camera is used to photograph the human eyes of the user wearing the head-mounted device when the display device sequentially outputs a plurality of picture calibration points through the screen, so as to obtain a human eye photographed image corresponding to each of the picture calibration points;

[0010] The calibration device is also used to obtain the calibration result of the eye tracking calibration system according to the human eye captured image corresponding to each of the picture calibration points.

[0011] In a second aspect, an embodiment of the present application also provides an eye tracking calibration method, the method comprising: in response to a confirmation instruction for a head calibration point, controlling a display device to sequentially output a plurality of picture calibration points through a screen; the confirmation instruction is generated when the length of time for which the head calibration point has been located in a target area reaches a target length of time or when a trigger operation for the head calibration point is received from a user wearing the head mounted device; the position information of the head calibration point is obtained by converting the intersection between the main optical axis of a scene camera and the screen to a screen coordinate system where the screen is located; when the display device sequentially outputs a plurality of picture calibration points through the screen, obtaining a human eye captured image corresponding to each of the picture calibration points; and obtaining a calibration result of the eye tracking calibration system based on the human eye captured image corresponding to each of the picture calibration points.

[0012] In a third aspect, an embodiment of the present application further provides a calibration device, including:

[0013] A control module, configured to control the display device to sequentially output a plurality of picture calibration points through a screen in response to a confirmation instruction for a head calibration point; the confirmation instruction is generated when the head calibration point is located in a target area for a target time or when a trigger operation for the head calibration point is received from a user wearing a head mounted device; the position information of the head calibration point is obtained by converting an intersection point between a main optical axis of a scene camera and a screen into a screen coordinate system where the screen is located;

[0014] An acquisition module, used to acquire a human eye captured image corresponding to each picture calibration point when the display device sequentially outputs multiple picture calibration points through the screen;

[0015] The output module is used to obtain the calibration result of the eye tracking calibration system according to the human eye captured image corresponding to each picture calibration point.

[0016] In a fourth aspect, the present application also provides another calibration device, including:

[0017] one or more processors;

[0018] Memory;

[0019] One or more application programs, wherein the one or more application programs are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to execute the above method.

[0020] The present application provides an eye tracking calibration system, method and device. In the present application, first, when the head calibration point is located in the target area for a target time or a trigger operation of the user wearing the head-mounted device is received, a confirmation instruction is generated to confirm that the head calibration is completed, so that the user's head is aligned with the target area, thereby improving the accuracy of the calibration. Then, the display device is controlled to output the picture calibration points at multiple target positions in sequence through the screen, and the human eye images corresponding to the picture calibration points at the multiple target positions are captured, so as to calibrate multiple directions. Finally, according to the human eye images corresponding to the picture calibration points at the multiple target positions, the calibration result of the eye tracking calibration system is obtained, thereby ensuring all-round calibration, improving the accuracy of the calibration result, and enhancing the user experience.

[0021] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A schematic diagram of an eye tracking calibration system provided in an embodiment of the present application is shown.

[0024] Figure 2 A schematic diagram of a target display feature and a head calibration point according to the present embodiment of the application is shown.

[0025] Figure 3 A schematic diagram of a picture calibration point in an embodiment of the present application is shown.

[0026] Figure 4 A schematic diagram of a visual axis vector and an optical axis vector in an embodiment of the present application is shown.

[0027] Figure 5 A flow chart of an eye tracking calibration method provided in an embodiment of the present application is shown.

[0028] Figure 6 A structural block diagram of a calibration device provided in an embodiment of the present application is shown.

[0029] Figure 7A structural block diagram of a calibration device according to another embodiment of the present application is shown.

[0030] Figure 8 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] Please refer to Figure 1 , Figure 1 A schematic diagram of an eye tracking calibration system provided by an embodiment of the present application is shown. A head mounted device 101, a display device 102 and a calibration device 103, wherein the head mounted device 101 at least includes a near-eye camera 104 and a scene camera 105;

[0034] The display device 102 is used to display the head calibration points on the screen of the display device according to the position information of the head calibration points when the user wears the head-mounted device. The position information of the head calibration points is obtained by converting the intersection between the main optical axis of the scene camera and the screen into the screen coordinate system where the screen is located.

[0035] The head mounted device 101 may be glasses, a helmet, or other device that requires eye tracking calibration.

[0036] The scene camera 105 may be a camera mounted on the head mounted device 101. When the user wears the head mounted device to watch the screen of the display device, the scene camera 105 may capture the screen displayed on the display device. The capture area is 106 and the size of the capture area 106 is not less than the size of the screen of the display device.

[0037] There may be two near-eye cameras 104, which are respectively installed at the areas corresponding to two human eyes, so as to capture images of human eyes. For example, the near-eye cameras 104 may be symmetrically installed at the positions of two eyes on the head-mounted device 101, and the shooting area of ​​the near-eye camera 104 is 107, and the size of the shooting area 107 must ensure that the entire user's eyes can be captured.

[0038] The display device 102 may be a device with a screen, such as a computer, a mobile phone, or a tablet.

[0039] The calibration device 103 includes a processor and a signal transmission and receiving system. The processor is used to call the images taken by the near-eye camera and the scene camera, process the images and feedback the results. The signal transmission and receiving system is used to transmit information between the processor and the camera. The information transmission between the processor and the camera can be wired or wireless. The calibration device 103 can be set on the head-mounted device 101 or not.

[0040] The head calibration point is the point displayed on the screen where the intersection of the main optical axis of the scene camera and the screen is located. The position of the head calibration point on the screen will move as the user wearing the head-mounted device moves his head. The main optical axis is a straight line passing through the optical center of the camera lens and parallel to the incident light.

[0041] Before displaying the head calibration point on the screen, the position information of the head calibration point needs to be determined. The position information here refers to the position information of the head calibration point in the screen coordinate system, which can be the coordinates in the screen coordinate system.

[0042] In some embodiments, the calibration device 103 is also used to determine the relative position relationship between the screen and the scene camera, and according to the relative position relationship between the screen and the scene camera, convert the intersection between the main optical axis of the scene camera and the screen to the screen coordinate system where the screen is located to obtain the position information of the head calibration point.

[0043] The position information of the intersection between the main optical axis of the scene camera 105 and the screen in the scene camera coordinate system is known. It is only necessary to determine the relative position relationship between the screen and the scene camera to convert the position information in the scene camera coordinate system to the screen coordinate system to obtain the position information of the head calibration point.

[0044] In some embodiments, the display device 102 is further used to output a picture including a target display feature through the screen; the scene camera 105 is further used to photograph the screen when the display device outputs the picture including the target display feature through the screen to obtain a feature captured image, wherein the feature captured image includes a target captured feature corresponding to the target display feature; the calibration device 103 is further used to determine the relative position relationship between the screen and the scene camera based on the position information of the target captured feature in the feature captured image and the position information of the target display feature in the screen.

[0045] During calibration, a screen including the target display feature is played on the screen of the display device 102. The screen may be one of multiple images or a video frame of a video. The multiple images and the video may be fixed images or videos or randomly generated images or videos. The target display feature may be an Aruco code. The Aruco code is a synthetic square mark consisting of a wide black border and an internal ID (binary matrix). Figure 2 201 in the figure is a target display feature, where the target display feature is an ARUCO code, which can be located at the center of the picture or around the picture.

[0046] The position information of the target display feature on the screen is known. After capturing the feature shooting image, the position information of the target display feature on the feature shooting image can be calculated. The target display feature on the feature shooting image and the target display feature on the screen have a one-to-one correspondence. Based on the position information of the two, the relative position relationship between the screen and the scene camera can be determined.

[0047] After determining the relative position relationship between the screen and the scene camera 105, the position information of the head calibration point in the screen coordinate system is obtained based on the relative position relationship and the position information of the intersection between the main optical axis of the scene camera and the screen in the scene camera coordinate system, and the head calibration point is displayed on the screen according to the position information.

[0048] The calibration device 103 is used to control the display device to output multiple screen calibration points in sequence through the screen in response to a confirmation instruction for the head calibration point; the confirmation instruction is generated when the head calibration point is located in the target area for a target time or when a trigger operation for the head calibration point is received by a user wearing the head-mounted device.

[0049] like Figure 2 As shown, Figure 2 201 is the target display feature. Figure 2 202 in the figure is the target area. Figure 2 203 is the head calibration point.

[0050] When the position of the head calibration point moves to the target area as the head of the user wearing the head-mounted device moves, a prompt such as "Do not move your head" may be displayed on the screen to remind the user not to move his head. At the same time, the user can perform a trigger operation to generate a confirmation instruction. The trigger operation may be, for example, clicking a button on the display device, etc. The trigger operation must be performed within a specified time period. If the user does not perform a trigger operation, a confirmation instruction is automatically generated after the head calibration point is located in the target area for a target time period. The target time period may be the same as or different from the specified time period, and may be set according to needs.

[0051] If the user does not perform a trigger operation within the specified time and the head calibration point does not stay in the target area for the target time, the user wearing the head mounted device moves his head again so that the head calibration point moves to the target area.

[0052] Through head calibration, the user can align the target area of ​​the screen. Subsequent screen calibration is performed based on the user aligning the target area of ​​the screen, thereby improving the accuracy of calibration.

[0053] The calibration device 103 receives the confirmation instruction for the head calibration point, indicating that the head calibration has been completed. The screen calibration needs to be performed. During the subsequent screen calibration, the head needs to be kept as still as possible. Then, a prompt appears on the screen that the user needs to continue to look at the screen calibration point. After the prompt, the screen calibration point will be displayed. Figure 3 As shown, Figure 3 401 in FIG. 4 is a picture calibration point.

[0054] The picture calibration point can be a movable point, which moves with the change of the picture output on the screen. The picture calibration point can move in any trajectory, but the moving trajectory must ensure that the picture calibration point can move around the screen at least once and stay at multiple target positions on the screen for a fixed time. The multiple target positions include at least the four corner points, the midpoints of the four sides and the center point of the screen. When the picture calibration point stays at each target position for a fixed time, color change, scaling and other characteristic changes will occur. The multiple picture calibration points here refer to a picture calibration point at multiple target positions.

[0055] The picture calibration points can also be multiple immovable points. The screen of the display device displays one picture calibration point at a time. Each picture calibration point is located at a different target position on the screen. The style of each picture calibration point can be the same or different. When each picture calibration point stays at its respective target position for a fixed period of time, characteristic changes such as color change or scaling will occur.

[0056] The near-eye camera 104 is used to photograph the human eyes of the user wearing the head-mounted device when the display device sequentially outputs multiple picture calibration points through the screen, so as to obtain the human eye photographed image corresponding to each picture calibration point.

[0057] In some embodiments, the near-eye camera 104 receives a trigger operation from a user wearing a head-mounted device in response to the screen outputting a plurality of image calibration points at target positions, and photographs the eyes of the user wearing the head-mounted device to obtain an eye-captured image of the target position. If the user does not perform a trigger operation within a fixed time period when the image calibration point stays at the target position, the eye-captured image is not captured for the image calibration point at that position. The trigger operation may be a user clicking the left mouse button or a space on the keyboard.

[0058] The calibration device 103 is also used to obtain the calibration result of the eye tracking calibration system according to the human eye captured image corresponding to each picture calibration point.

[0059] The human eye images corresponding to the image calibration points at multiple target positions are obtained, and the human eye images are processed to obtain the calibration results of the eye tracking calibration system.

[0060] In some embodiments, the calibration device 103 is further used to determine the visual axis vector and the optical axis vector corresponding to each picture calibration point based on the human eye captured image corresponding to each picture calibration point; and determine the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point; and obtain the calibration result of the eye tracking calibration system based on the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point.

[0061] Among them, the visual axis refers to the line connecting the real user's viewpoint through the node and the center of the fovea, and the optical axis is the user's gaze direction estimated based on two eye images taken by two near-eye cameras. The gaze direction is the line connecting the center of the cornea and the center of the pupil.

[0062] like Figure 4 As shown, Figure 4 a in the figure is the user's left eye. Figure 4 The b in the figure is the user's right eye. Figure 4 The c in is the user's pupil center, Figure 4 The 401 in the code is the screen. Figure 4 402 in is the optical axis vector, Figure 4 403 in is the visual axis vector, Figure 4 The d in is the user’s real viewpoint after calibration according to the calibration result. Figure 4 The e in is the viewpoint before calibration.

[0063] To determine the angle between the viewing axis vector and the optical axis vector corresponding to the image calibration point, the viewing axis vector and the optical axis vector need to be determined respectively.

[0064] In some embodiments, the calibration device 103 is further used to determine the target pupil position information of the pupil center of the user wearing the head-mounted device in the image captured by the human eye corresponding to the picture calibration point based on the image captured by the human eye corresponding to the picture calibration point; and determine the visual axis vector corresponding to the picture calibration point based on the target pupil position information of the pupil center in the image captured by the human eye corresponding to the picture calibration point and the position information of the picture calibration point on the screen.

[0065] In some embodiments, after determining the target pupil position information in the human eye captured image corresponding to the pupil center of the user wearing the head-mounted device at the picture calibration point, the target pupil position information is converted from the near-eye camera coordinate system to the target coordinate system according to the target pupil position information and the conversion relationship between the near-eye camera coordinate system and the target coordinate system; according to the position information of the picture calibration point in the screen coordinate system and the conversion relationship between the screen coordinate system and the target coordinate system, the position information of the picture calibration point in the screen coordinate system is converted to the target coordinate system, the pupil center in the target coordinate system and the picture calibration point are connected, and the vector from the pupil center to the picture calibration point is obtained, which is the visual axis vector.

[0066] The target coordinate system may be any coordinate system, such as a scene camera coordinate system.

[0067] In some embodiments, the calibration device 103 is further used to determine, based on the human eye captured image corresponding to the picture calibration point, target corneal position information of the cornea center of the user wearing the head-mounted device in the human eye captured image corresponding to the picture calibration point and target pupil position information of the pupil center in the human eye captured image corresponding to the picture calibration point; and determine the optical axis vector corresponding to the picture calibration point based on the target corneal position information of the cornea center in the human eye captured image corresponding to the picture calibration point and target pupil position information of the pupil center in the human eye captured image corresponding to the picture calibration point.

[0068] In some embodiments, after determining the target corneal position information of the corneal center and the target pupil position information of the pupil center of the user wearing the head-mounted device, the target pupil position information is converted from the near-eye camera coordinate system to the target coordinate system based on the target corneal position information and the conversion relationship between the near-eye camera coordinate system and the target coordinate system, and the pupil center and corneal center in the target coordinate system are connected to obtain a vector from the pupil center to the corneal center, which is the optical axis vector.

[0069] After obtaining the visual axis vector and the optical axis vector, the angle between the visual axis vector and the optical axis vector can be calculated. The angle can be calculated by, for example, the formula cosθ=(A·B) / (|A|·|B|), where A and B are two vectors, A·B is the dot product of the two vectors, and |A| and |B| are the moduli of the two vectors.

[0070] As an implementation mode, the calibration device 103 is also used to divide the multiple picture calibration points into calibration point groups corresponding to multiple orientations according to the position information of the multiple picture calibration points on the screen; and determine the orientation calibration result corresponding to each orientation according to the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point in the calibration point group corresponding to each orientation; and obtain the calibration result of the eye tracking calibration system according to the orientation calibration results corresponding to the multiple orientations.

[0071] Among them, multiple directions can include four directions: upper left, lower left, upper right, and lower right, or six directions: upper left, upper middle, upper right, lower left, lower middle, and lower right. The specific directions can be set according to needs, and the calibration point group corresponding to each direction includes at least one picture calibration point.

[0072] For each orientation, after multiple angles are calculated, an average value, median or mode of the multiple angles may be taken to obtain a calibration result for each orientation.

[0073] As another embodiment, the calibration device 103 is further used to perform target calculation on the angles between the visual axis vector and the optical axis vector corresponding to each of the plurality of image calibration points to determine the calibration result of the eye tracking calibration system.

[0074] The target operation may be an operation for calculating statistical quantities such as an average value, a median, and a mode of the angles between the visual axis vectors and the optical axis vectors corresponding to the plurality of picture calibration points.

[0075] In some embodiments, the calibration device 103 is also used to obtain a calibration result of the eye tracking calibration system based on the human eye captured image corresponding to each screen calibration point when the number of effective interactions reaches a preset threshold. Effective interaction refers to receiving a confirmation operation for the screen calibration point from the user wearing the head-mounted device when the screen outputs the screen calibration point.

[0076] The user's trigger operation is a trigger operation performed when the picture calibration point is at a target position, and the number of times the user can perform the trigger operation when the screen outputs the picture calibration point is not greater than the number of the plurality of picture calibration points.

[0077] If the number of valid interactions does not reach the preset threshold, it means that the calibration result lacks reliability, the current calibration is invalid, and recalibration is required.

[0078] In this embodiment, first, a confirmation instruction is generated when the head calibration point is located in the target area for a target time or when a trigger operation by the user wearing the head-mounted device is received, confirming that the head calibration is completed, so that the user's head is aligned with the target area, thereby improving the accuracy of the calibration. Then, the display device is controlled to output the picture calibration points at multiple target positions through the screen in sequence, and the human eye images corresponding to the picture calibration points at the multiple target positions are captured, so as to calibrate multiple directions. Finally, the calibration result of the eye tracking calibration system is obtained according to the human eye images corresponding to the picture calibration points at the multiple target positions, thereby ensuring all-round calibration, improving the accuracy of the calibration result, and enhancing the user experience.

[0079] Please refer to Figure 5 , Figure 5 A flow chart of an eye tracking calibration method provided in an embodiment of the present application is shown. The method can be used to calibrate the device 103. The method includes:

[0080] S501, in response to a confirmation instruction for a head calibration point, controlling a display device to sequentially output a plurality of picture calibration points through a screen; the confirmation instruction is generated when the head calibration point is located in a target area for a target time or when a trigger operation for the head calibration point is received from a user wearing a head mounted device; the position information of the head calibration point is obtained by converting the intersection between the main optical axis of the scene camera and the screen into a screen coordinate system where the screen is located;

[0081] S502, when the display device sequentially outputs a plurality of picture calibration points through the screen, obtaining a human eye captured image corresponding to each picture calibration point;

[0082] S503: Obtain a calibration result of the eye tracking calibration system according to the human eye captured image corresponding to each picture calibration point.

[0083] Optionally, the method for acquiring the position information of the head calibration point includes:

[0084] Determine the relative position relationship between the screen and the scene camera, and based on the relative position relationship between the screen and the scene camera, convert the intersection between the main optical axis of the scene camera and the screen to the screen coordinate system where the screen is located to obtain the position information of the head calibration point.

[0085] Optionally, determining a relative position relationship between the screen and the scene camera includes:

[0086] When the display device outputs a picture of the target display feature through the screen, a feature captured image of the screen is obtained, and the feature captured image includes a target capture feature corresponding to the target display feature; based on the position information of the target capture feature in the feature captured image and the position information of the target display feature in the screen, the relative position relationship between the screen and the scene camera is determined.

[0087] Optionally, a calibration result of the eye tracking calibration system is obtained according to the human eye captured image corresponding to each picture calibration point, including:

[0088] According to the human eye captured image corresponding to each picture calibration point, the visual axis vector and the optical axis vector corresponding to each picture calibration point are determined; and the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point is determined; and according to the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point, the calibration result of the eye tracking calibration system is obtained.

[0089] Optionally, determining the visual axis vector corresponding to each picture calibration point according to the human eye captured image corresponding to each picture calibration point includes:

[0090] According to the human eye captured image corresponding to the picture calibration point, determine the target pupil position information of the pupil center of the user wearing the head-mounted device in the human eye captured image corresponding to the picture calibration point; and according to the target pupil position information in the human eye captured image corresponding to the picture calibration point and the position information of the picture calibration point on the screen, determine the visual axis vector corresponding to the picture calibration point.

[0091] Optionally, determining the optical axis vector corresponding to each picture calibration point according to the human eye captured image corresponding to each picture calibration point includes:

[0092] According to the human eye captured image corresponding to the picture calibration point, determine the target cornea position information of the cornea center of the user wearing the head-mounted device in the human eye captured image corresponding to the picture calibration point and the target pupil position information of the pupil center in the human eye captured image corresponding to the picture calibration point; and determine the optical axis vector corresponding to the picture calibration point according to the target cornea position information in the human eye captured image corresponding to the picture calibration point and the target pupil position information in the human eye captured image corresponding to the picture calibration point.

[0093] Optionally, according to the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point, a calibration result of the eye tracking calibration system is obtained, including:

[0094] According to the position information of the multiple picture calibration points on the screen, the multiple picture calibration points are divided into calibration point groups corresponding to multiple orientations; and according to the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point in the calibration point group corresponding to each orientation, the orientation calibration result corresponding to each orientation is determined; according to the orientation calibration results corresponding to the multiple orientations, the calibration result of the eye tracking calibration system is obtained.

[0095] Optionally, according to the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point, a calibration result of the eye tracking calibration system is obtained, including:

[0096] A target operation is performed on the angles between the visual axis vector and the optical axis vector corresponding to each of the multiple screen calibration points to determine the calibration result of the eye tracking calibration system.

[0097] Optionally, according to the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point, a calibration result of the eye tracking calibration system is obtained, including:

[0098] When the number of valid interactions reaches a preset threshold, the calibration result of the eye tracking calibration system is obtained according to the human eye captured image corresponding to each screen calibration point. The valid interaction refers to the confirmation operation of the user wearing the head-mounted device for the screen calibration point when the screen outputs the screen calibration point.

[0099] Please refer to Figure 6 , Figure 6 The structure block diagram of a calibration device 103 provided in an embodiment of the present application is shown. The calibration device 600 includes:

[0100] The control module 601 is used to control the display device to sequentially output multiple picture calibration points through the screen in response to a confirmation instruction for the head calibration point; the confirmation instruction is generated when the head calibration point is located in the target area for a target time or when a trigger operation for the head calibration point is received from a user wearing the head mounted device; the position information of the head calibration point is obtained by converting the intersection between the main optical axis of the scene camera and the screen into a screen coordinate system where the screen is located;

[0101] An acquisition module 602 is used to acquire a human eye captured image corresponding to each picture calibration point when the display device sequentially outputs multiple picture calibration points through the screen;

[0102] The output module 603 is used to obtain the calibration result of the eye tracking calibration system according to the human eye captured image corresponding to each picture calibration point.

[0103] Optionally, the device also includes a first determination module, which is used to determine the relative position relationship between the screen and the scene camera, and according to the relative position relationship between the screen and the scene camera, convert the intersection between the main optical axis of the scene camera and the screen to the screen coordinate system where the screen is located to obtain the position information of the head calibration point.

[0104] Optionally, the device also includes a second determination module, which is used to obtain a feature capture image of the screen when the display device outputs a picture of the target display feature through the screen, the feature capture image including a target capture feature corresponding to the target display feature; and determine the relative position relationship between the screen and the scene camera based on the position information of the target capture feature in the feature capture image and the position information of the target display feature in the screen.

[0105] Optionally, the output module 603 is also used to determine the visual axis vector and the optical axis vector corresponding to each picture calibration point according to the human eye captured image corresponding to each picture calibration point; and determine the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point; and obtain the calibration result of the eye tracking calibration system according to the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point.

[0106] Optionally, the output module 603 is also used to determine the target pupil position information of the pupil center of the user wearing the head-mounted device in the image captured by the human eye corresponding to the picture calibration point based on the image captured by the human eye corresponding to the picture calibration point; and determine the visual axis vector corresponding to the picture calibration point based on the target pupil position information of the pupil center in the image captured by the human eye corresponding to the picture calibration point and the position information of the picture calibration point on the screen.

[0107] Optionally, the output module 603 is also used to determine the target cornea position information of the user wearing the head-mounted device in the human eye captured image corresponding to the picture calibration point and the target pupil position information of the pupil center in the human eye captured image corresponding to the picture calibration point according to the human eye captured image corresponding to the picture calibration point; and determine the optical axis vector corresponding to the picture calibration point according to the target cornea position information of the cornea center in the human eye captured image corresponding to the picture calibration point and the target pupil position information of the pupil center in the human eye captured image corresponding to the picture calibration point.

[0108] Optionally, the output module 603 is also used to divide the multiple picture calibration points into calibration point groups corresponding to multiple orientations according to the position information of the multiple picture calibration points on the screen; and determine the orientation calibration result corresponding to each orientation according to the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point in the calibration point group corresponding to each orientation; and obtain the calibration result of the eye tracking calibration system according to the orientation calibration results corresponding to the multiple orientations.

[0109] Optionally, the output module 603 is further used to perform target calculation on the angles between the visual axis vector and the optical axis vector corresponding to each of the plurality of picture calibration points to determine a calibration result of the eye tracking calibration system.

[0110] Optionally, the output module 603 is also used to obtain a calibration result of the eye tracking calibration system based on the human eye captured image corresponding to each screen calibration point when the number of effective interactions reaches a preset threshold. Effective interaction refers to receiving a confirmation operation for the screen calibration point from the user wearing the head-mounted device when the screen outputs the screen calibration point.

[0111] Please refer to Figure 7 , Figure 7 The structural block diagram of a calibration device of another embodiment of the present application is shown. The calibration device 700 can be a calibration device capable of running applications such as a smart phone, a tablet computer, and an e-book. The calibration device 700 in the present application may include one or more of the following components: a processor 710, a memory 720, and one or more applications. One or more applications may be stored in the memory 720 and configured to be executed by one or more processors 710, and one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0112] The processor 710 may include one or more processing cores. The processor 710 uses various interfaces and lines to connect the various parts of the entire calibration device 700, and executes various functions and processes data of the calibration device 700 by running or executing instructions, programs, code sets or instruction sets stored in the memory 720, and calling data stored in the memory 720. Optionally, the processor 710 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 710 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 710, but may be implemented separately through a communication chip.

[0113] The memory 720 may include a random access memory (RAM) or a read-only memory (ROM). The memory 720 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the calibration device 700 during use.

[0114] Please refer to Figure 8 , Figure 8 The structure block diagram of a computer-readable storage medium in an embodiment of the present application is shown. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0115] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An eye tracking calibration system, characterized in that: The eye tracking calibration system comprises at least: a head-mounted device, a display device and a calibration device, wherein the head-mounted device comprises at least a near-eye camera and a scene camera; wherein, The display device is used to display the head calibration points on the screen of the display device according to the position information of the head calibration points when the user wears the head mounted device, wherein the position information of the head calibration points is obtained by converting the intersection between the main optical axis of the scene camera and the screen into the screen coordinate system where the screen is located; The calibration device is used to control the display device to sequentially output a plurality of picture calibration points through a screen in response to a confirmation instruction for the head calibration point; the confirmation instruction is generated when the head calibration point is located in a target area for a target time or when a trigger operation for the head calibration point is received by a user wearing the head mounted device; The near-eye camera is used to photograph the human eyes of the user wearing the head-mounted device when the display device sequentially outputs a plurality of picture calibration points through the screen, so as to obtain a human eye photographed image corresponding to each of the picture calibration points; The calibration device is also used to obtain the calibration result of the eye tracking calibration system according to the human eye captured image corresponding to each of the picture calibration points.

2. The system according to claim 1, characterized in that The calibration device is also used to determine the relative position relationship between the screen and the scene camera, and according to the relative position relationship between the screen and the scene camera, convert the intersection between the main optical axis of the scene camera and the screen to the screen coordinate system where the screen is located, so as to obtain the position information of the head calibration point.

3. The system according to claim 2, characterized in that The display device is further used to output a picture including the target display feature through a screen; The scene camera is further configured to capture the screen when the display device outputs a picture of the target display feature through the screen, so as to obtain a feature capture image, wherein the feature capture image includes a target capture feature corresponding to the target display feature; The calibration device is also used to determine the relative position relationship between the screen and the scene camera based on the position information of the target shooting feature in the feature shooting image and the position information of the target display feature in the screen.

4. The system according to claim 1, characterized in that The calibration device is also used to determine the visual axis vector and the optical axis vector corresponding to each of the picture calibration points based on the human eye captured image corresponding to each of the picture calibration points; and determine the angle between the visual axis vector and the optical axis vector corresponding to each of the picture calibration points; and obtain the calibration result of the eye tracking calibration system based on the angle between the visual axis vector and the optical axis vector corresponding to each of the picture calibration points.

5. The system according to claim 4, characterized in that The calibration device is further used to determine target pupil position information of the pupil center of the user wearing the head mounted device in the human eye captured image corresponding to the picture calibration point according to the human eye captured image corresponding to the picture calibration point; And according to the target pupil position information of the pupil center in the human eye captured image corresponding to the picture calibration point and the position information of the picture calibration point in the screen, the visual axis vector corresponding to the picture calibration point is determined.

6. The system according to claim 4, characterized in that The calibration device is further used to determine, based on the human eye captured image corresponding to the picture calibration point, target cornea position information of the cornea center of the user wearing the head mounted device in the human eye captured image corresponding to the picture calibration point and target pupil position information of the pupil center in the human eye captured image corresponding to the picture calibration point; The optical axis vector corresponding to the picture calibration point is determined according to the target cornea position information of the cornea center in the human eye captured image corresponding to the picture calibration point and the target pupil position information of the pupil center in the human eye captured image corresponding to the picture calibration point.

7. The system according to claim 4, characterized in that The calibration device is further used to divide the multiple picture calibration points into multiple calibration point groups corresponding to respective directions according to the position information of the multiple picture calibration points on the screen; And according to the angle between the visual axis vector and the optical axis vector corresponding to each picture calibration point in the calibration point group corresponding to each orientation, the orientation calibration result corresponding to each orientation is determined; according to the orientation calibration results corresponding to multiple orientations, the calibration result of the eye tracking calibration system is obtained.

8. The system according to claim 4, characterized in that The calibration device is also used to perform target calculation on the angles between the visual axis vector and the optical axis vector corresponding to each of the multiple picture calibration points to determine the calibration result of the eye tracking calibration system.

9. The system according to claim 4, characterized in that The calibration device is also used to obtain the calibration result of the eye tracking calibration system according to the human eye captured image corresponding to each of the picture calibration points when the number of effective interactions reaches a preset threshold, and the effective interaction refers to receiving a confirmation operation for the picture calibration point by the user wearing the head-mounted device when the screen outputs the picture calibration point.

10. An eye tracking calibration method, characterized in that: The method comprises: In response to a confirmation instruction for the head calibration point, controlling the display device to sequentially output a plurality of picture calibration points through the screen; the confirmation instruction is generated when the head calibration point is located in the target area for a target time or when a trigger operation for the head calibration point is received from a user wearing the head mounted device; the position information of the head calibration point is obtained by converting the intersection between the main optical axis of the scene camera and the screen into a screen coordinate system where the screen is located; When the display device sequentially outputs a plurality of picture calibration points through the screen, obtaining a human eye captured image corresponding to each of the picture calibration points; The calibration result of the eye tracking calibration system is obtained according to the human eye captured image corresponding to each of the picture calibration points.

11. A calibration device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to execute the method of claim 10.

Images