Eye tracking calibration method, head-mounted device, and computer-readable storage medium

Abstract

The application discloses an eye movement tracking calibration method, a head-mounted device and a computer readable storage medium, and relates to the technical field of head-mounted devices. The method comprises the following steps: in response to an input operation of a wearer, collecting a field of view image of the wearer and a corresponding eye image; determining a gaze area of the wearer based on an operation object targeted by the input operation, and cutting the field of view image according to the gaze area to obtain a gaze area image; and after a plurality of pairs of gaze area images and eye images are collected, performing eye movement tracking calibration processing based on the plurality of pairs of gaze area images and eye images. The application realizes a non-intrusive continuous calibration without the active participation of a user, and improves the problems of a time-consuming and lengthy calibration process and a complex and tedious calibration process.

Description

Technical Field

[0001] This application relates to the field of head-mounted device technology, and more particularly to an eye-tracking calibration method, a head-mounted device, and a computer-readable storage medium. Background Technology

[0002] Eye-tracking technology, as an important means of interaction, has been widely used in recent years in various smart head-mounted devices such as augmented reality (AR) glasses and virtual reality (VR) glasses to achieve precise control and human-computer interaction based on eye movements.

[0003] However, due to individual differences in each user's ocular physiological structure (such as corneal curvature and pupil size), eye-tracking systems cannot be used universally. Personalized calibration is required for each user to eliminate system errors and ensure tracking accuracy. In addition, positional shifts and changes in fit during device wear can introduce additional errors. Therefore, in practical applications, users often need to recalibrate the device each time they wear it again.

[0004] However, traditional eye-tracking calibration methods primarily rely on user participation in a standardized calibration process. This involves requiring the user to sequentially gaze at multiple pre-defined calibration points (typically 5 to 9 points) on a display screen or chart. The system then builds a personalized model by collecting eye images or data from the user's gaze at these points. However, this calibration method requires the user to maintain full focus and complete the gaze task in a fixed order, during which no other operations can be performed. The calibration process is time-consuming and complex. This not only reduces the smoothness and efficiency of the user experience but may also lead to user fatigue or distraction affecting the calibration quality. Summary of the Invention

[0005] The main purpose of this application is to provide an eye-tracking calibration method, a head-mounted device, and a computer-readable storage medium, which aims to solve the technical problems of traditional eye-tracking calibration methods that rely on users to actively participate in the standard calibration process, resulting in a long calibration process and a complex calibration process.

[0006] To achieve the above objectives, this application provides an eye-tracking calibration method, which is applied to a head-mounted device and includes the following steps: In response to the wearer's input operation, the system acquires the wearer's visual field image and corresponding eye image; Based on the target of the input operation, the wearer's gaze area is determined, and the visual field image is cropped according to the gaze area to obtain the gaze area image; After collecting multiple pairs of the gaze region images and the eye images, eye-tracking calibration processing is performed based on the multiple pairs of the gaze region images and the eye images.

[0007] In one embodiment, prior to the step of determining the wearer's gaze region based on the target of the input operation, the method further includes: After the input operation is completed, the intention of the input operation is identified; If the operation intent indicates to operate on an object in the wearer's field of vision, then the step of determining the wearer's gaze area based on the target of the operation based on the input operation is performed; If the operation intent does not indicate operation on an object in the wearer's field of vision, the acquired field of vision image and the corresponding eye image are discarded.

[0008] In one embodiment, the step of performing eye-tracking calibration processing based on multiple pairs of gaze region images and eye images includes: All the gaze region images are divided into at least one set of gaze region images, wherein the eye images corresponding to the gaze region images in each set of gaze region images have a consistent eye pose; For each set of gaze region images, the gaze point position corresponding to the calibrated eye image is determined based on the gaze region images in the set of gaze region images, wherein the calibrated eye image is the eye image corresponding to each gaze region image in the set of gaze region images; A calibration dataset is constructed based on each of the eye images and each of the gaze points. Eye tracking calibration is performed based on the calibration dataset. The calibration dataset includes multiple sets of calibration data, each set of calibration data including one of the eye images and the gaze point position corresponding to the eye image.

[0009] In one embodiment, the step of determining the fixation point position corresponding to the calibrated eye image based on the fixation region image in the fixation region image set includes: The gaze region image in the gaze region image set is determined as the calibration gaze region image; If there is a common intersection among all the calibrated gaze region images, the center point of the common intersection is determined as the gaze point position corresponding to the calibrated eye image; If there is no common intersection among all the calibrated gaze region images, and there is an intersection between at least two of the calibrated gaze region images, then the target intersection with the most intersections is determined based on the polygon spatial indexing algorithm, and the center point of the target intersection is determined as the gaze point position corresponding to the calibrated eye image.

[0010] In one embodiment, the step of determining the target intersection with the most intersections based on the polygon spatial indexing algorithm includes: For each calibrated gaze region image, an image confidence level is set based on the object contour size within the calibrated gaze region image, wherein the image confidence level and the object contour size have a negative correlation trend; Based on the polygon spatial indexing algorithm and the credibility of each image, the target intersection with the largest number of weighted intersections is determined, wherein the image credibility is used to weight the number of intersections corresponding to the calibrated gaze region image.

[0011] In one embodiment, the step of performing eye-tracking calibration processing based on the calibration dataset includes: The calibration dataset is divided into a calibration dataset and a validation dataset. The calibration dataset includes multiple fixation region images corresponding to the eye images, while the validation dataset includes a single fixation region image corresponding to the eye images. Eye-tracking calibration is performed using the calibration dataset, and after eye-tracking calibration is completed, calibration verification is performed using the verification dataset to obtain the verification result. If the verification result indicates that the verification is successful, the eye-tracking calibration process ends.

[0012] In one embodiment, the step of acquiring the wearer's visual field image and eye image includes: At the start of the input operation, a frame of the wearer's visual field image and the corresponding eye image are acquired, or; During the time period between the start and end of the input operation, multiple frames of the wearer's visual field images and corresponding eye images are acquired.

[0013] In one embodiment, after the step of acquiring multiple frames of the wearer's visual field images and corresponding eye images, the method further includes: Multiple frames of eye images are grouped to obtain multiple sets of eye images, wherein the eye images in each set of eye images have a consistent eye pose; Determine the target eye image set that contains the most eye images in each of the aforementioned eye image sets; The target eye image is obtained by averaging the eye pose values ​​of all the eye images in the target eye image set. The target eye image is determined as the eye image corresponding to each frame of the visual field image.

[0014] In addition, to achieve the above objectives, this application also provides a head-mounted device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the eye-tracking calibration method as described above.

[0015] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of the eye-tracking calibration method as described above.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the eye-tracking calibration method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: This application's embodiments trigger data acquisition by "responding to the wearer's input operation," transforming eye-tracking calibration from an independent operation process into an accompanying process embedded in natural interaction. When the user performs a routine input operation, the system utilizes the user's natural gaze at the object being operated on to acquire visual field images and corresponding eye images. This integrates calibration data acquisition into the user's daily operating scenario, eliminating the intrusiveness of traditional calibration processes. Users do not need to interrupt their current task or perform specific calibration actions, thus significantly improving the smoothness of the interaction experience and effectively avoiding user fatigue and attention distraction caused by forced focus. Then, by analyzing the object targeted by the input operation, the user's gaze area is determined, and a gaze area image is generated accordingly. This mechanism transforms the target object of the user's actual operation into a "natural calibration point," replacing the traditional fixed predefined calibration pattern. This calibration method based on real interaction scenarios continuously accumulates "gaze area image-eye image" sample pairs without the user's awareness, and performs eye-tracking calibration processing using the accumulated "gaze area image-eye image" samples. In this way, by collecting sample pairs formed by the wearer during natural interaction, a seamless and continuous calibration without the need for active user participation is achieved, thereby effectively improving the problems of time-consuming and cumbersome calibration processes. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the eye-tracking calibration method of this application; Figure 2 This is a flowchart illustrating the third embodiment of the eye-tracking calibration method of this application; Figure 3 This is a schematic diagram of the fixation point location determination process in an embodiment of the eye-tracking calibration method of this application; Figure 4 This is the intended implementation of a tracking calibration verification process in one embodiment of the eye-tracking calibration method of this application; Figure 5 This is a schematic diagram of the hardware operating environment of the eye-tracking calibration method device in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Eye-tracking technology is becoming increasingly mature as an important interaction method and is being applied to various head-mounted devices. However, since each user's eyes are different, individual calibration is required for each user. The calibration process requires the user to focus on multiple calibration points on the chart or display screen, which is lengthy (usually exceeding 1 minute) and complex (requiring the user to sequentially focus on multiple fixation points as required). During this process, the user cannot perform other operations, resulting in a significant time consumption.

[0024] In addition, since the wearing position and fit are different each time, the tracking results will be deviated to a certain extent. Therefore, in actual use, users need to recalibrate when wearing it again, which further complicates the calibration process.

[0025] Based on this, the main solution of this application is: in response to the wearer's input operation, to acquire the wearer's visual field image and the corresponding eye image; to determine the wearer's gaze area based on the operation object targeted by the input operation, and to crop the visual field image according to the gaze area to obtain the gaze area image; after collecting multiple pairs of the gaze area images and the eye images, to perform eye-tracking calibration processing based on the multiple pairs of the gaze area images and the eye images.

[0026] This application triggers data acquisition by responding to the wearer's input, transforming eye-tracking calibration from an independent process into an embedded process within natural interaction. When a user performs a routine input operation, the system captures visual field images and corresponding eye images by utilizing the user's natural gaze at the object being operated on. This integrates calibration data acquisition into the user's daily usage scenario, eliminating the intrusiveness of traditional calibration processes. Users do not need to interrupt their current task or perform specific calibration actions, significantly improving the smoothness of the interaction experience and effectively avoiding user fatigue and distraction caused by forced focus. Then, by analyzing the object targeted by the input operation, the user's gaze area is determined, and a gaze area image is generated accordingly. This mechanism transforms the target object of the user's actual operation into a "natural calibration point," replacing the traditional fixed predefined calibration pattern. This calibration method, based on real-world interaction scenarios, continuously accumulates "gaze area image-eye image" sample pairs without the user's awareness, and performs eye-tracking calibration processing using these accumulated samples. In this way, by collecting sample pairs formed by the wearer during natural interaction, a seamless and continuous calibration without the need for active user participation is achieved, thereby effectively improving the problems of time-consuming and cumbersome calibration processes.

[0027] It should be noted that the execution subject of the various embodiments of the eye-tracking calibration method of this application can be a head-mounted device capable of realizing the above functions, such as AI (Artificial Intelligence) glasses, XR (Extended Reality) glasses, etc., and the various embodiments of the eye-tracking calibration method of this application do not impose specific limitations on this.

[0028] Based on this, this application proposes an eye-tracking calibration method according to a first embodiment, wherein the eye-tracking calibration method is applied to a head-mounted device, with reference to... Figure 1 As shown, the eye-tracking calibration method includes the following steps S10~S30: Step S10: In response to the wearer's input operation, acquire the wearer's visual field image and corresponding eye image; Input operations can be interactive commands issued by the wearer through handheld controllers, gesture recognition, voice commands, or physical buttons on the device. Upon receiving such input, the head-mounted device's built-in forward-facing camera or other forward-facing image acquisition device begins to work, capturing images within the wearer's current field of vision as a visual field image, which represents the real or virtual scene seen by the user; and a rear-facing camera or other rear-facing image acquisition device is used to capture images of the wearer's eyes to obtain the wearer's eye state at the current moment.

[0029] Furthermore, it should be noted that the visual field image and the eye image are acquired synchronously, that is, the eye image and the visual field image are acquired at almost the same time. This synchronous acquisition mechanism ensures that each "eye image" corresponds to the scene content ("visual field image") that the user is looking at when the input is triggered, thus establishing a reliable data foundation for subsequent calibration.

[0030] Step S20: Determine the wearer's gaze region based on the operation target of the input operation, and crop the visual field image according to the gaze region to obtain the gaze region image; The target of operation refers to the interactive target that the wearer points to through input operations. Specifically, by parsing the semantic content or interactive target of the input operations, the specific object that the wearer intends to operate can be identified. The target of operation can be represented as a virtual button, icon or object pointed to in a virtual environment, or a physical object that is of interest in a real scene.

[0031] After identifying the object to be manipulated, the spatial boundary of the object in the visual field image can be defined using computer vision algorithms (such as deep learning-based object detection and instance segmentation models) to form a gaze region. This gaze region at least includes the main body of the object. In a preferred embodiment, the gaze region includes the main body of the object and a surrounding area within a predetermined range of the main body to fully preserve the visual attention area. Subsequently, the visual field image is cropped according to the defined gaze region to generate a gaze region image centered on the object to be manipulated.

[0032] For example, when a wearer inputs the command "turn on the desk lamp" via voice, the "desk lamp" is recognized as the object of the operation, and the image area of ​​the desk lamp in the visual field is then cropped to obtain the corresponding gaze region image. Similarly, if a wearer faces a flower and inputs the voice question "what kind of flower is this?", the "flower" is identified as the object of the operation, and the image area of ​​the flower in the visual field is cropped to obtain the gaze region image. As another example, in a scenario requiring a payment task, when the wearer inputs the command "scan to pay," the "QR code" in the visual field is identified as the object of the operation, and the image area of ​​the QR code in the visual field is cropped to obtain the gaze region image. Furthermore, in a virtual interface interaction scenario, when the wearer uses a gesture or a controller to click the close button in the upper right corner of a dialog box, the "close button" is recognized as the object of the operation, and the image area of ​​the close button in the visual field is cropped to obtain the gaze region image.

[0033] Step S30: After collecting multiple pairs of the gaze region images and the eye images, eye-tracking calibration processing is performed based on the multiple pairs of the gaze region images and the eye images.

[0034] As the wearer performs a series of input operations, the device continuously repeats steps S10 and S20 to collect multiple pairs of gaze area images and corresponding eye images. These multiple pairs of image data contain information about the wearer's eye features and visual field under different gaze targets.

[0035] It should be noted that when the number of accumulated "gaze region image-eye image" sample pairs reaches a preset threshold (this threshold can be set according to actual needs and is reached in a dispersed manner during natural use), calibration processing can be initiated.

[0036] Furthermore, after cropping to the fixation area image, the fixation point position in the image is determined. The fixation point position refers to the specific location in the fixation area image where the wearer is actually focusing, such as the center point of the image, a feature point of a virtual button or object, or the center position of a QR code. Thus, a "fixation point position - eye image" calibration data pair is constructed based on the eye image and the fixation point position, and eye tracking calibration is performed through the calibration data pair.

[0037] For eye-tracking calibration, mature eye-tracking calibration methods can be used, such as machine learning-based model optimization algorithms and multi-point calibration algorithms. This embodiment does not impose specific limitations on this.

[0038] During the calibration process, the collected sample pairs are used to establish or optimize the wearer's eye-tracking model, enabling the device to more accurately predict the wearer's gaze point position based on their eye movements, thereby achieving precise eye-tracking functionality and improving the performance and user experience of the head-mounted device during interaction.

[0039] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, before the step of determining the wearer's gaze area based on the target of the input operation, the method further includes: Step A10: After the input operation ends, identify the intention of the input operation; After the wearer completes an input operation, the head-mounted device can analyze the input through its built-in voice recognition module, gesture recognition module, or other interaction modules to determine the wearer's true intention. For example, if the input is a voice command "turn on the desk lamp," the device uses voice recognition technology to convert the voice command into text and further analyzes the text content to identify the intention as "turn on the desk lamp." If the input is a gesture, the device uses a gesture recognition algorithm to analyze the shape, direction, and movement of the gesture to identify the intention. The purpose of this step is to determine whether the input operation is related to the operation of an object in the field of vision, so as to decide whether to proceed with the fixation area determination step.

[0040] Step A20: If the operation intention indicates to operate on an object in the wearer's field of vision, then the step of determining the wearer's gaze area based on the operation object targeted by the input operation is performed. If the device recognizes that the intended action is directed at an object within the field of vision, such as "turn on the desk lamp" or "scan a QR code to pay," the device proceeds to the next step: determining the wearer's gaze area. The device further analyzes the specific content of the input action, identifies the object being manipulated (such as "desk lamp" or "QR code"), and extracts the corresponding gaze area image from the visual field image based on the object's location and extent. This ensures that only input actions related to object manipulation trigger the determination of the gaze area and subsequent calibration processes, improving the targeting and efficiency of the calibration.

[0041] Step A30: If the operation intention does not indicate operation on an object in the wearer's field of vision, then the acquired field of vision image and the corresponding eye image are discarded.

[0042] If the intended operation does not specify that the wearer should interact with an object in their field of vision, the acquired visual field image and corresponding eye image are discarded. If the device recognizes that the intended operation is unrelated to interacting with an object in the field of vision—for example, if the wearer simply issues an irrelevant voice command (such as "play music") or makes a non-operational gesture (such as waving)—the device considers these input operations irrelevant to eye-tracking calibration. Therefore, the device discards the visual field image and eye image acquired during that input operation to avoid invalid data interfering with the calibration process, while also saving storage space and computing resources.

[0043] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. On this basis, refer to Figure 2 As shown, the step of performing eye-tracking calibration processing based on multiple pairs of gaze region images and eye images includes: Step B10: Divide all the gaze region images into at least one set of gaze region images, wherein the eye images corresponding to the gaze region images in each set of gaze region images have a consistent eye pose. The eye pose refers to the orientation of the eye in three-dimensional space, which may include the angle of the eye (such as the angle of deflection and pitch) and the position of the eye (such as the relative position in the eye socket).

[0044] By analyzing the acquired eye images, the eye pose is identified, and images of gaze regions with consistent eye poses are grouped together. It should be noted that in this embodiment, the consistency principle for eye pose includes two cases: first, the eye poses are completely identical, meaning all feature parameters are completely consistent within the measurement error range; second, the similarity between eye poses is greater than a preset similarity threshold, which can be quantitatively evaluated by calculating the Euclidean distance or cosine similarity of the eye pose vectors.

[0045] For example, suppose the collected "fixation region image-eye image" pairs are [(fixation region image 1-eye image 1), (fixation region image 2-eye image 2), (fixation region image 3-eye image 3), (fixation region image 4-eye image 4), (fixation region image 5-eye image 5)]. Among them, eye image 2 and eye image 4 have the same eye pose, eye image 3 and eye image 5 have the same eye pose, and eye image 1 does not have the same eye pose as any eye image. Then, they can be divided into 3 sets of fixation region images, namely [(fixation region image 1)], [(fixation region image 2, fixation region image 4)], and [(fixation region image 3, fixation region image 5)].

[0046] Step B20: For each set of gaze region images, determine the gaze point position corresponding to the calibrated eye image based on the gaze region images in the set of gaze region images, wherein the calibrated eye image is the eye image corresponding to each gaze region image in the set of gaze region images; For each set of fixation region images, the coordinates of the fixation point can be determined by analyzing the visual content of all fixation region images in the set, and the determined coordinates of the fixation point can be used as the fixation point position of the corresponding eye image (i.e., the calibrated eye image) of the fixation region images in that set.

[0047] For example, suppose there are 3 sets of fixation region images, namely [(fixation region image 1)], [(fixation region image 2, fixation region image 4)], and [(fixation region image 3, fixation region image 5)]. The fixation point position calculated based on fixation region image 1 is fixation point position A, the fixation point position calculated based on fixation region image 2 and fixation region image 4 is fixation point position B, and the fixation point position calculated based on fixation region image 3 and fixation region image 5 is fixation point position C. Then the correspondence between eye images and fixation point positions is [(eye image 1 - fixation point position A), (eye image 2 - fixation point position B), (eye image 3 - fixation point position C), (eye image 4 - fixation point position B), (eye image 5 - fixation point position C)].

[0048] Step B30: Construct a calibration dataset based on each of the eye images and each of the gaze points, and perform eye-tracking calibration processing based on the calibration dataset. The calibration dataset includes multiple sets of calibration data, each set of calibration data including one of the eye images and the gaze point position corresponding to the eye image.

[0049] A calibration dataset is constructed based on each eye image and its corresponding gaze point location. This dataset contains multiple sets of calibration data, each set consisting of an eye image and its corresponding gaze point location.

[0050] It should be noted that, in order to optimize dataset quality and reduce computational redundancy, before constructing the calibration dataset, the "eye image-gaze point location" data pairs obtained based on eye pose can be deduplicated. The calibration dataset can then be constructed based on the deduplicated "eye image-gaze point location". The deduplication strategy can be based on complete consistency of eye pose, that is, retaining data pairs with unique eye pose features, thereby constructing an efficient calibration dataset that contains sufficient sample diversity while avoiding redundancy.

[0051] For example, suppose the obtained "eye image-fixation point position" data pairs are [(eye image 1-fixation point position A), (eye image 2-fixation point position B), (eye image 3-fixation point position C), (eye image 4-fixation point position B), (eye image 5-fixation point position C)], and eye image 2 and eye image 4 have the exact same eye pose. Then (eye image 2-fixation point position B) and (eye image 4-fixation point position B) are actually two pairs of duplicate data. Either data pair (eye image 2-fixation point position B) or (eye image 4-fixation point position B) can be retained. Then the calibration dataset constructed after deduplication can be [(eye image 1-fixation point position A), (eye image 2-fixation point position B), (eye image 3-fixation point position C), (eye image 5-fixation point position C)].

[0052] In one possible implementation, refer to Figure 3 As shown, the step of determining the fixation point position corresponding to the calibrated eye image based on the fixation region image in the fixation region image set includes: Step C10: Determine the gaze region image in the gaze region image set as the calibration gaze region image; Step C20: If there is a common intersection among all the calibrated gaze region images, then the center point of the common intersection is determined as the gaze point position corresponding to the calibrated eye image; When all calibrated gaze region images share a common intersection, it indicates that the wearer has gazed at the same object multiple times or at the same location on different objects. In this case, the geometric center localization method can be used to determine the gaze point location. Specifically, by calculating the spatial overlap of the bounding boxes corresponding to each calibrated gaze region image, their maximum common intersection is determined. Then, the coordinates of the geometric center point of this common intersection are used as the gaze point location corresponding to the calibrated eye image. The center point of the common intersection can specifically be the center of the largest inscribed circle of the common intersection.

[0053] Step C30: If there is no common intersection among all the calibrated gaze region images, and there is an intersection between at least two of the calibrated gaze region images, then the target intersection with the most intersections is determined based on the polygon space indexing algorithm, and the center point of the target intersection is determined as the gaze point position corresponding to the calibrated eye image. When there is no complete common intersection among all calibrated gaze region images, but some calibrated images do intersect, it indicates the presence of dirty data in multiple calibrated gaze region images. A polygon spatial indexing algorithm is then activated for optimization to remove dirty data with fewer intersections. Specifically, this involves: constructing the spatial index relationship of the bounding boxes of each calibrated gaze region image; calculating the intersection regions between each pair and establishing an intersection relationship map; then counting the frequency of each intersection region being included, selecting the target intersection region with the most intersections; finally, calculating the center point of this target intersection and determining it as the gaze point location corresponding to the calibrated eye image. Specifically, the center point of the target intersection can be the center of the largest inscribed circle of the target intersection.

[0054] Furthermore, if there is no intersection between any two calibrated gaze region images, the collected multiple pairs of gaze region images and eye images are saved or deleted, and the process returns to the step of collecting the wearer's visual field image and corresponding eye image in response to the wearer's input operation.

[0055] When there is no overlap between any two calibrated gaze region images, it indicates a significant gaze dispersion in the collected dataset, failing to achieve effective spatial consistency. Two processing strategies can be used in this case: First, temporarily save all currently acquired visual field and eye image data to a cache for secondary processing when supplementing subsequent data; second, directly delete the data acquired in the current batch to ensure the purity of the calibration dataset. After retaining or deleting the "gaze region image - eye image" data pairs, return to the initial data acquisition step, respond to the wearer's input again, and acquire new visual field and eye image pairs until a dataset meeting the spatial consistency requirements is obtained.

[0056] In this embodiment, when all calibrated gaze region images share a common intersection, it indicates that the user has maintained a highly consistent spatial attention distribution across multiple gazes. In this case, the center point of the common intersection is determined as the gaze point location, effectively eliminating random errors in a single acquisition and improving the stability and repeatability of gaze point localization. When there is partial intersection between calibrated gaze region images, the most representative target intersection is selected by statistically analyzing frequency using a polygonal spatial indexing algorithm. This optimization strategy based on spatial distribution density can intelligently identify and eliminate abnormal data points deviating from the main gaze region, thereby enhancing the resistance to interference factors such as unavoidable eye movement and recognition errors during the acquisition process. Overall, through hierarchical spatial relationship discrimination and optimization, while ensuring gaze point localization accuracy, the robustness and reliability of eye-tracking calibration in practical application scenarios are significantly improved.

[0057] In one possible implementation, the step of determining the target intersection with the highest number of intersections based on the polygon spatial indexing algorithm includes: Step D10: For each calibrated gaze region image, set the image confidence level based on the object contour size within the calibrated gaze region image, wherein the image confidence level and the object contour size have a negative correlation trend. Image analysis techniques can be used to identify the contours of objects within each calibrated gaze region image and calculate their dimensions. When the object contour size is large, the corresponding image confidence value is low; conversely, when the object contour size is small, the image confidence value is high. This design is based on the visual cognition principle of eye tracking: smaller objects typically require more precise gaze localization, while larger objects exhibit greater gaze point deviation. In practice, computer vision algorithms (such as instance segmentation models) are used to extract the contour boundaries of objects within the calibrated gaze region image, and the diagonal length of its envelope rectangle or the contour pixel area is calculated as a size quantification indicator. Then, by using a preset size threshold or a continuous function (such as an inverse proportional function, a negative exponential function, etc.), the object contour size can be mapped to the image confidence value in the [0, 1] interval, establishing a quantification conversion model from size features to image confidence value.

[0058] Step D20: Based on the polygon spatial indexing algorithm and the credibility of each image, determine the target intersection with the largest number of weighted intersections, wherein the image credibility is used to weight the number of intersections corresponding to the calibrated gaze region image.

[0059] Building upon the construction of a polygonal spatial index, a confidence-weighted intersection statistics algorithm is introduced. Specifically, a spatial index structure (such as an R-tree or quadtree) of the bounding boxes of each calibrated gaze region image is first established, and the intersection relationship between pairs of images is quickly determined through spatial queries. When counting the frequency of intersections, instead of a simple counting method, each intersection count is multiplied by the confidence weight of the two images involved in the intersection to obtain a weighted intersection value. Subsequently, the weighted values ​​of all possible intersection regions are accumulated, and the intersection region with the largest accumulated weighted value is selected as the target intersection. This weighting mechanism effectively reduces the influence of low-confidence images containing large-area target objects in the decision-making process, while enhancing the influence of images containing small-area high-precision target objects in determining the final gaze point location, thereby improving the overall accuracy and reliability of gaze point localization.

[0060] In one possible implementation, refer to Figure 4 As shown, the step of performing eye-tracking calibration processing based on the calibration dataset includes: Step E10: Divide the calibration dataset into a calibration dataset and a validation dataset. The calibration dataset contains multiple fixation region images corresponding to the eye images, while the validation dataset contains a single fixation region image corresponding to the eye images. The calibration dataset was divided into two subsets: a calibration dataset and a validation dataset. The division was based on the quantity of gaze region images: the calibration dataset contains eye images for each gaze region image set containing multiple gaze region images. This design ensures that the calibration process can utilize spatially consistent multiple gaze samples, thereby improving the stability and generalization ability of the eye calibration process. The validation dataset, on the other hand, adopts a single-sample criterion, where each eye image corresponds to a gaze region image set containing only a single gaze region image. This setting simulates a single gaze scenario in real-world use, effectively verifying the performance on independent samples after calibration and ensuring the reliability of the calibrated eye-tracking model in practical applications.

[0061] Step E20: Perform eye-tracking calibration using the calibration dataset, and after eye-tracking calibration is completed, perform calibration verification using the verification dataset to obtain the verification result; The eye-tracking model is trained and optimized using data from the calibration dataset. This step may involve machine learning algorithms, such as neural network training, to minimize the error between the predicted gaze location and the actual gaze location by adjusting the model parameters. After calibration, the calibrated model is validated using a validation dataset. The validation process involves comparing the model's predicted gaze location with the actual gaze location in the validation dataset and calculating error metrics (such as mean squared error or mean absolute error) to assess the model's accuracy.

[0062] Step E30: If the verification result indicates that the verification is successful, then the current eye-tracking calibration process ends.

[0063] If the verification results show that the error between the model's predicted gaze point position and the actual gaze point position is within an acceptable range, the calibration process is successful, and the model is now able to accurately track the user's eye movements. At this point, the calibration process can be ended, and the calibrated model can be used for subsequent eye-tracking tasks. If the verification results fail, the user can choose to return to step E20 and re-calibrate, or be prompted to perform further operations, such as re-acquiring image data or adjusting device settings. This embodiment does not impose specific limitations on these actions.

[0064] Based on the first, second, and / or third embodiments of this application, the content in the fourth embodiment of this application that is the same as or similar to the above-described embodiments one, two, and three can be referred to the above description and will not be repeated hereafter. Based on this, the step of acquiring the wearer's visual field image and eye image includes: Step F10: At the start of the input operation, acquire a frame of the wearer's visual field image and the corresponding eye image; When the wearer begins an input operation, such as the start of a voice command, the initial frame of a gesture, or the instant a button on the controller is pressed, the head-mounted device triggers its image acquisition module at the start of the operation, simultaneously capturing one frame of visual field image and one frame of eye image. For example, if the wearer inputs the voice command "What kind of flower is this?", the device acquires visual field and eye images at the moment the wearer says "this" to capture image data of the wearer at the initial fixation point and respond promptly to the input operation.

[0065] Alternatively, in step F20, during the time period between the start and end times of the input operation, multiple frames of the wearer's visual field images and corresponding eye images are acquired.

[0066] When the wearer begins input, the head-mounted device can continuously acquire multiple frames of visual field images and corresponding eye images at a certain frequency (e.g., 10 frames per second) throughout the entire input period. For example, if the wearer inputs the voice command "What kind of flower is this?", the device will continuously acquire multiple frames of visual field images and corresponding eye images at a certain frequency during the time the wearer says "What kind of flower is this?". This multi-frame acquisition strategy not only captures the dynamic features during the gaze process but also improves the reliability of gaze localization through time series analysis, providing richer training samples for calibration models.

[0067] In one possible implementation, after the step of acquiring multiple frames of the wearer's visual field images and corresponding eye images, the method further includes: Step G10: Group the multiple frames of eye images to obtain multiple sets of eye images, wherein the eye images in each set of eye images have a consistent eye pose. Grouping processing is performed on continuously acquired multi-frame eye images. Eye pose parameters, such as position and angle, are extracted from each frame. Based on these parameters, a clustering algorithm is used to group eye images with consistent eye poses into multiple eye image sets. "Consistent eye pose" is defined as the Euclidean distance between pose vectors being less than a preset threshold, or the difference in principal component analysis (PCA) features being within an acceptable range. Each generated eye image set maintains high consistency within itself, while relatively large differences in eye pose exist between different sets.

[0068] Step G20: Determine the target eye image set that contains the most eye images in each of the stated eye image sets; Statistical analysis was performed on the size of each eye image set. By counting the number of eye images contained in each set, the set containing the most image frames was selected as the target eye image set.

[0069] Furthermore, when multiple sets with the same maximum size exist, the set with the smaller variance in eye pose features can be selected first. If the variances are also the same, the set with the most recent timestamp is selected. This selection mechanism ensures that subsequent processing is based on the most stable and statistically significant sample data.

[0070] Step G30: Average the eye pose of all the eye images in the target eye image set to obtain the target eye image; Eye pose fusion processing is performed on all eye images in the target eye image set. Specifically, this includes: first, normalizing the coordinates of eye feature points in each image; then, calculating the average positions of key features such as the pupil center point set and corneal reflection point set; and finally, generating a comprehensive and representative target eye image using a feature fusion algorithm. This image retains the key details of the original image while effectively suppressing random noise and transient jitter interference in single-frame images through averaging effects.

[0071] Step G40: The target eye image is determined as the eye image corresponding to each frame of the visual field image.

[0072] The target eye image obtained after fusion processing is correlated with each frame of visual field image within the acquisition period, forming multiple pairs of "visual field image-eye image" data. This one-to-many mapping relationship ensures that even if there are micro-movements of the eyeball or momentary occlusions during actual acquisition, subsequent calibration processing can still be based on the highest quality and most stable eye image data, improving the reliability and consistency of calibration data and laying a high-quality data foundation for ensuring the accuracy of eye tracking calibration.

[0073] Furthermore, embodiments of this application also propose a head-mounted device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method described above.

[0074] refer to Figure 5 The diagram illustrates a structural schematic suitable for implementing the head-mounted device of the embodiments of this application. The head-mounted device in the embodiments of this application may also include, but is not limited to, AI glasses, XR glasses, AR headsets, VR headsets, etc. Figure 5The head-mounted device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0075] like Figure 5 As shown, the head-mounted device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the head-mounted device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the head-mounted device to communicate wirelessly or wiredly with other devices to exchange data. While head-mounted devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0076] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0077] The head-mounted device provided in this application, employing the eye-tracking calibration method described in the above embodiments, solves the technical problem that traditional eye-tracking calibration methods rely on user active participation in the standard calibration process, resulting in a long and complex calibration process. Compared with the prior art, the beneficial effects of the head-mounted device provided in this application are the same as those of the eye-tracking calibration method provided in the above embodiments, and other technical features of this head-mounted device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0078] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0079] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0080] In addition, to achieve the above objectives, embodiments of this application also provide a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the eye-tracking calibration method in the above embodiments.

[0081] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0082] The aforementioned computer-readable storage medium may be included in the head-mounted device; or it may exist independently and not assembled into the head-mounted device.

[0083] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a head-mounted device, cause the head-mounted device to: in response to an input operation by a wearer, acquire a visual field image and a corresponding eye image of the wearer; determine the wearer's gaze region based on the operation object targeted by the input operation, and crop the visual field image according to the gaze region to obtain a gaze region image; and after collecting multiple pairs of gaze region images and eye images, perform eye-tracking calibration processing based on the multiple pairs of gaze region images and eye images.

[0084] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0086] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the modules themselves.

[0087] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described eye-tracking calibration method. This solves the technical problem that traditional eye-tracking calibration methods rely on user-initiated standard calibration processes, which are time-consuming and complex. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the eye-tracking calibration method provided in the above embodiments, and will not be repeated here.

[0088] Furthermore, embodiments of this application also propose a computer program product, including a computer program that, when executed by a processor, implements the steps of the eye-tracking calibration method as described above.

[0089] The specific implementation of the computer program product in this application is basically the same as the embodiments of the above-described eye-tracking calibration method, and will not be repeated here.

[0090] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0091] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a head-mounted device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0093] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An eye-tracking calibration method, characterized in that, The eye-tracking calibration method is applied to a head-mounted device, and the eye-tracking calibration method includes the following steps: In response to the wearer's input operation, the system acquires the wearer's visual field image and corresponding eye image; Based on the target of the input operation, the wearer's gaze area is determined, and the visual field image is cropped according to the gaze area to obtain the gaze area image; After collecting multiple pairs of the gaze region images and the eye images, all the gaze region images are divided into at least one set of gaze region images, wherein the eye images corresponding to the gaze region images in each set of gaze region images have a consistent eye pose. For each set of gaze region images, the gaze region images in the gaze region image set are determined as calibration gaze region images; If there is a common intersection among all the calibrated gaze region images, the center point of the common intersection is determined as the gaze point position corresponding to the calibrated eye image, wherein the calibrated eye image is the eye image corresponding to each gaze region image in the gaze region images; If there is no common intersection among all the calibrated gaze region images, and there is an intersection between at least two of the calibrated gaze region images, then the target intersection with the most intersections is determined based on the polygon space indexing algorithm, and the center point of the target intersection is determined as the gaze point position corresponding to the calibrated eye image. A calibration dataset is constructed based on each of the eye images and each of the gaze points. Eye tracking calibration is performed based on the calibration dataset. The calibration dataset includes multiple sets of calibration data, each set of calibration data including one of the eye images and the gaze point position corresponding to the eye image.

2. The eye-tracking calibration method as described in claim 1, characterized in that, Before the step of determining the wearer's gaze region based on the operation target of the input operation, the method further includes: After the input operation is completed, the intention of the input operation is identified; If the operation intent indicates to operate on an object in the wearer's field of vision, then the step of determining the wearer's gaze area based on the operation object targeted by the input operation is performed; If the operation intent does not indicate operation on an object in the wearer's field of vision, the acquired field of vision image and the corresponding eye image are discarded.

3. The eye-tracking calibration method as described in claim 1, characterized in that, The step of determining the target intersection with the most intersections based on the polygon spatial indexing algorithm includes: For each calibrated gaze region image, an image confidence level is set based on the object contour size within the calibrated gaze region image, wherein the image confidence level and the object contour size have a negative correlation trend; Based on the polygon spatial indexing algorithm and the credibility of each image, the target intersection with the largest number of weighted intersections is determined, wherein the image credibility is used to weight the number of intersections corresponding to the calibrated gaze region image.

4. The eye-tracking calibration method as described in claim 1, characterized in that, The step of performing eye-tracking calibration based on the calibration dataset includes: The calibration dataset is divided into a calibration dataset and a validation dataset. The calibration dataset contains multiple fixation region images corresponding to the eye images, while the validation dataset contains a single fixation region image corresponding to the eye images. Eye-tracking calibration is performed using the calibration dataset, and after eye-tracking calibration is completed, calibration verification is performed using the verification dataset to obtain the verification result. If the verification result indicates that the verification is successful, the eye-tracking calibration process ends.

5. The eye-tracking calibration method as described in any one of claims 1 to 4, characterized in that, The step of acquiring the wearer's visual field image and eye image includes: At the start of the input operation, a frame of the wearer's visual field image and the corresponding eye image are acquired, or; During the time period between the start and end of the input operation, multiple frames of the wearer's visual field images and corresponding eye images are acquired.

6. The eye-tracking calibration method as described in claim 5, characterized in that, After the step of acquiring multiple frames of the wearer's visual field images and corresponding eye images, the method further includes: Multiple frames of eye images are grouped to obtain multiple sets of eye images, wherein the eye images in each set of eye images have a consistent eye pose; Determine the target eye image set that contains the most eye images in each of the aforementioned eye image sets; The target eye image is obtained by averaging the eye pose values ​​of all the eye images in the target eye image set. The target eye image is determined as the eye image corresponding to each frame of the visual field image.

7. A head-mounted device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the eye-tracking calibration method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the eye-tracking calibration method as described in any one of claims 1 to 6.

Images