Line of sight estimation method, apparatus, device, and medium

Abstract

This disclosure relates to a gaze estimation method, apparatus, device, and medium. The method includes: acquiring a target eye image corresponding to a target eye whose gaze direction needs to be estimated; the target eye image is an image of the target eye captured by a target camera; acquiring intrinsic and extrinsic parameters of the target camera, and obtaining camera information based on the intrinsic and extrinsic parameters; performing at least one target processing operation based on the target eye image and the camera information, and obtaining a gaze estimation result of the target eye image based on the output information of the last target processing operation; the target processing includes: extracting key target information based on the input information corresponding to the target processing, and fusing the key target information and specified information from the input information corresponding to the target processing to obtain the output information corresponding to the target processing. This disclosure helps to obtain more realistic and reliable gaze estimation results to better meet requirements.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a line-of-sight estimation method, apparatus, device, and medium. Background Technology

[0002] In many fields such as gaming, healthcare, and intelligent control, it is necessary to identify the direction of human eye gaze in order to implement appropriate strategies based on the estimated gaze direction. The inventors discovered that most existing gaze estimation technologies struggle to obtain a sufficiently accurate and reliable gaze direction, thus failing to adequately meet these requirements. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a line-of-sight estimation method, apparatus, device, and medium.

[0004] This disclosure provides a gaze estimation method, the method comprising: acquiring a target eye image corresponding to a target eye whose gaze direction is to be estimated; wherein the target eye image is an image of the target eye captured by a target camera; acquiring intrinsic and extrinsic parameters of the target camera, and obtaining camera information based on the intrinsic and extrinsic parameters; performing at least one target processing based on the target eye image and the camera information, and obtaining a gaze estimation result of the target eye image based on the output information of the last target processing; the target processing comprising: extracting target key information based on the input information corresponding to the target processing, and fusing the target key information and specified information in the input information corresponding to the target processing to obtain the output information corresponding to the target processing; wherein the input information of the first target processing includes the target eye image and the camera information, and the input information of subsequent target processing includes the output information of the previous target processing and the camera information.

[0005] Optionally, the step of performing at least one target processing based on the target eye image and the camera information, and obtaining a gaze estimation result of the target eye image based on the output information of the last target processing, includes: inputting the target eye image and the camera information into a preset gaze estimation model; wherein the gaze estimation model includes at least one target network, and when there are multiple target networks, the multiple target networks are connected sequentially; performing target processing through the target network; wherein the at least one target network has a one-to-one correspondence with the at least one target processing, and different target networks have the same network structure but different network parameters; and obtaining a gaze estimation result of the target eye image based on the output information of the target processing corresponding to the last target network.

[0006] Optionally, the target network includes a first network unit, a second network unit, and a third network unit; the target processing through the target network includes: performing concatenation processing through the first network unit based on different types of information contained in the input information of the target network to obtain target concatenation information; extracting target key information from the target concatenation information through the second network unit based on low-rank matrix factorization technology; wherein the target key information includes key eye features and key camera features; performing fusion processing through the third network unit based on the target key information and specified information in the input information corresponding to the target processing, and obtaining output information based on the fusion processing result; wherein the specified information includes information related to image features.

[0007] Optionally, the gaze estimation model is obtained through the following steps: acquiring eye sample images; wherein the eye sample images are images of eye samples captured by a camera sample, and the eye sample images carry eye sample gaze labels and intrinsic and extrinsic parameters of the camera sample; based on the eye sample images and the intrinsic and extrinsic parameters of the camera sample, obtaining the gaze estimation result of the eye sample images through a preset initial network model; wherein the structure of the initial network model is the same as the structure of the gaze estimation model; based on the difference between the gaze estimation result of the eye sample images and the eye sample gaze labels, adjusting the parameters of the first network unit, the second network unit, and the third network unit in the initial network model, and using the parameter-adjusted initial network model as the first network model; generating a gaze estimation model based on the first network model.

[0008] Optionally, generating a gaze estimation model based on the first network model includes: acquiring a calibration eye image; wherein the calibration eye image is an image of the target eye captured by the target camera, and the calibration eye image carries a gaze label of the target eye and the intrinsic and extrinsic parameters of the target camera; obtaining a gaze estimation result of the calibration eye image based on the calibration eye image and the intrinsic and extrinsic parameters of the target camera through the first network model; adjusting the parameters of the second network unit in the initial network model based on the difference between the gaze estimation result of the calibration eye image and the gaze label of the target eye, and using the parameter-adjusted first network model as the gaze estimation model.

[0009] Optionally, the method further includes: determining a target region to be processed from the target eye image; wherein the target region is determined based on the iris of the target eye; performing a specified operation based on the target region and the target eye image; wherein the specified operation includes: extracting the target region from the target eye image to obtain a target region image, and performing a size adjustment operation on the target region image; performing a stitching process based on the camera information and the operation information of the specified operation to obtain first stitching information; the step of performing at least one target processing based on the target eye image and the camera information includes: performing at least one target processing based on the target eye image and the first stitching information; wherein the input information for the first target processing includes the target eye image and the first stitching information; the input information for subsequent target processing includes the output information of the previous target processing and the first stitching information.

[0010] Optionally, the target eye image includes images of the target eye captured by different target cameras at the same time; the gaze estimation result of the target eye includes the three-dimensional gaze direction and the three-dimensional pupil position of the target eye.

[0011] This disclosure also provides a gaze estimation device, comprising: an eye image acquisition module, configured to acquire a target eye image corresponding to a target eye whose gaze direction is to be estimated; wherein the target eye image is an image of the target eye captured by a target camera; a camera information acquisition module, configured to acquire intrinsic and extrinsic parameters of the target camera, and obtain camera information based on the intrinsic and extrinsic parameters; and a target processing execution module, configured to perform at least one target processing based on the target eye image and the camera information, and obtain a gaze estimation result of the target eye image based on the output information of the last target processing; the target processing includes: extracting target key information based on the input information corresponding to the target processing, and fusing the target key information and specified information in the input information corresponding to the target processing to obtain the output information corresponding to the target processing; wherein the input information of the first target processing includes the target eye image and the camera information, and the input information of subsequent target processing includes the output information of the previous target processing and the camera information.

[0012] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the line-of-sight estimation method as provided in this disclosure.

[0013] This disclosure also provides a computer-readable storage medium storing a computer program for performing the line-of-sight estimation method as provided in this disclosure.

[0014] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the line-of-sight estimation method provided in this disclosure.

[0015] The technical solution provided in this disclosure fully considers the influence of the camera on gaze estimation. Compared with related technologies that do not introduce camera parameters, it can obtain the intrinsic and extrinsic parameters of the target camera used to capture the target eye image, obtain camera information based on the intrinsic and extrinsic parameters, and introduce the target eye image and camera information into gaze estimation. Moreover, it performs at least one target processing, and obtains the gaze estimation result of the target eye image based on the output information of the last target processing. The target processing includes: extracting target key information based on the input information corresponding to the target processing, and fusing the target key information and specified information in the input information corresponding to the target processing to obtain the output information corresponding to the target processing. The input information of the first target processing includes the target eye image and camera information. Subsequent target processing will further extract key information based on the output information of the previous target processing and fuse the key information with the specified information in the input information. The fused output information is then used as the input information for the next target processing. This progressive approach can extract more in-depth and comprehensive key information based on the target eye image and camera information, thereby helping to obtain more realistic and reliable gaze estimation results to better meet the requirements.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic flowchart illustrating a line-of-sight estimation method provided in an embodiment of this disclosure;

[0020] Figure 2 This is a schematic diagram of the structure of a line-of-sight estimation model provided in an embodiment of the present disclosure;

[0021] Figure 3 This is a schematic diagram of the structure of a target network provided in an embodiment of the present disclosure;

[0022] Figure 4 A flowchart illustrating a method for obtaining a line-of-sight estimation model provided in an embodiment of this disclosure;

[0023] Figure 5 This is a schematic diagram of the structure of a line-of-sight estimation device provided in an embodiment of the present disclosure;

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0026] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0027] Figure 1 This is a flowchart illustrating a gaze estimation method provided in an embodiment of the present disclosure. The method can be executed by a gaze estimation device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S106:

[0028] Step S102: Obtain the target eye image corresponding to the target eye whose gaze direction is to be estimated; wherein, the target eye image is the image of the target eye captured by the target camera. In order to obtain the three-dimensional gaze estimation result, in some implementation examples, the target eye image includes images of the target eye captured by different target cameras at the same time. Based on this, the gaze estimation result of the target eye includes the three-dimensional gaze direction and the three-dimensional pupil position of the target eye.

[0029] The target eye can be the eye of the target object to be estimated (such as a user). For example, the target user can wear electronic devices such as VR (Virtual Reality) glasses, smart glasses, eye trackers, or other electronic devices that require estimation of the user's eye gaze direction. These electronic devices can integrate a camera to capture images of the target user's eyes as target eye images. Alternatively, the user can also directly capture images of the target user's eyes using a camera at a preset position without wearing any device. There are no restrictions on the method of capturing target eye images. Compared to related technologies where at most one camera is used for image capture per eye, embodiments of this disclosure can use different cameras simultaneously to capture eye images for a single eye. The target eye can be monocular or binocular; that is, the left eye and / or the right eye can both be used as target eyes. In some implementation examples, the number of target cameras corresponding to the same eye can be one, two, or more, and can be flexibly set. Taking a scenario with two target cameras as an example, the target eye image includes the first eye image captured by the first target camera at the target time and the second eye image captured by the second target camera at the target time. For instance, the left eye corresponds to both cameras, and the right eye corresponds to both cameras. The left eye corresponds to the first and second left eye images captured by the two cameras simultaneously, and the right eye corresponds to the first and second right eye images captured by the two cameras simultaneously. Capturing images of the same eye simultaneously from both cameras helps in accurately estimating subsequent 3D gaze information.

[0030] It should be noted that a prompt message can be sent to the user before collecting eye images, and the user's eye images can only be collected after authorization is obtained. For example, the prompt message can be sent to the user in the form of a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control for the user to choose "agree" or "disagree" to allow the electronic device to collect their eye images. It is understood that the above notification and user authorization process is merely illustrative and does not constitute a limitation on the implementation of this disclosure.

[0031] Step S104: Obtain the intrinsic and extrinsic parameters of the target camera, and obtain camera information based on the intrinsic and extrinsic parameters.

[0032] This disclosure fully considers the influence of the camera on gaze estimation. Compared with related technologies that do not introduce camera parameters, it can obtain the intrinsic and extrinsic parameters of the target camera used to acquire images of the target eye. The intrinsic and extrinsic parameters of the target camera are the intrinsic parameters and extrinsic parameters of the target camera. The intrinsic and extrinsic parameters of the target camera can be obtained with reference to related technologies, and will not be described in detail here. In some implementations, intrinsic and extrinsic parameters can be directly used as camera information. In other implementations, additional processing, such as vectorization, can be applied to the intrinsic and extrinsic parameters. It is understood that camera intrinsic parameters are used to describe the inherent characteristics of the camera, specifically parameters related to the camera's optical properties, such as focal length, principal point, radial distortion coefficient, and tangential distortion coefficient. Camera intrinsic parameters can be represented by a 3x3 matrix. Camera extrinsic parameters are used to describe the camera's position and orientation relative to the world coordinate system, which can also be understood as the camera pose. Camera extrinsic parameters can be represented by a rotation matrix R and a translation vector T. The rotation matrix R is usually a 3x3 matrix, and the translation vector T is usually a 1x3 matrix. In practical applications, the matrices corresponding to the camera intrinsic and extrinsic parameters can be converted into vectors, and the vectors converted from each matrix can be concatenated to obtain camera information represented in vector form, which is more convenient for subsequent model processing.

[0033] Step S106: Based on the target eye image and camera information, perform at least one target processing step, and obtain the gaze estimation result of the target eye image based on the output information of the last target processing step. The target processing includes: extracting key target information based on the input information corresponding to the target processing step, and fusing the key target information and specified information in the input information corresponding to the target processing step to obtain the output information corresponding to the target processing step. The input information for the first target processing step includes the target eye image and camera information, and the input information for subsequent target processing steps includes the output information of the previous target processing step and the camera information. In other words, if the number of times the target processing is executed is N, and N is greater than or equal to 2, then the input information for the i-th target processing step is the output information of the (i-1)-th target processing step and the camera information, where i is an integer between 2 and N.

[0034] The aforementioned key target information affects the accuracy of gaze estimation. This information can include key features of the target eye and the target camera, such as the eye's visual axis and optical axis features, and the camera's inherent device features and pose features. There are no restrictions on what information the model learns during training can be used as key target information. If the target processing is executed multiple times, the multiple target processing steps are executed sequentially, with different processing parameters used in each step. These parameters include, but are not limited to, the stride. The stride can be the size of the convolution kernel or pooling window sliding across the input data in the feature extraction process, and can be used to adjust the feature scale, thereby adjusting the receptive field of the feature map. Each target processing step can perform operations such as key information extraction and information fusion based on its respective input information and appropriate parameters such as step size. Although different target processing steps are the same, the specific processing parameters used are different, which can effectively ensure the comprehensiveness and depth of the processing results. The output information of the previous target processing can be used as the input information for the subsequent target processing. Through a progressive approach, the depth and comprehensiveness of the output information of the final target processing are effectively guaranteed. In addition, the input information of each target processing step includes camera information, which can be continuously enhanced during each target processing step. Combining the camera information with the target eye image for gaze analysis comprehensively ensures the reliability of the gaze estimation result of the target eye image. In some specific examples, the output information of the final target processing step includes the three-dimensional gaze direction and three-dimensional pupil position of the target eye. The output information of the final target processing step can be directly used as the gaze estimation result of the target eye image. In other specific examples, the output information of the final target processing step can be further optimized by calibration processing, and the gaze estimation result of the target eye image can be obtained based on the optimized processing result of the output information of the final target processing step.

[0035] The technical solution provided in this disclosure can incorporate target eye images and camera information into gaze estimation. By incorporating camera information, the model can clearly understand the inherent characteristics and pose of the target camera that captured the target eye image, thereby helping to obtain a more accurate gaze estimation result corresponding to the target eye image. Furthermore, through the aforementioned progressive target processing method, more in-depth and comprehensive key information can be extracted based on the target eye image and camera information, thus helping to obtain a more realistic and reliable gaze estimation result to better meet the requirements.

[0036] In some specific implementation examples, the method provided by this disclosure further includes the following steps 1 to 3:

[0037] Step 1: Determine the target region to be processed from the target eye image; the target region is determined based on the iris of the target eye. For example, the size of the target region is twice the size of the iris of the target eye. In this way, the area used to represent the eye in the target eye image can be further reasonably reduced. While ensuring accurate analysis of the gaze situation based on the target region, this helps to further narrow the processing range of the model, allowing the model to focus on processing the target region and improve the model's processing efficiency.

[0038] Step 2: Perform specified operations based on the target region and the target eye image. These specified operations include: extracting the target region from the target eye image to obtain a target region image, and performing a size adjustment operation on the target region image. For example, the size of the extracted target region image can be adjusted to a specified size for subsequent processing. In practical applications, the operation information of the specified operations can be represented using methods such as affine transformation matrices, or other forms; no restrictions are placed here.

[0039] Step 3: Perform splicing processing based on camera information and operation information of specified operation to obtain first splicing information. In other words, the first splicing information carries both camera intrinsic and extrinsic parameters and operation information of specified operation (which can be simply referred to as image operation information). During splicing processing, the above types of information can be converted into vectors of specified dimensions, and then the vectors corresponding to each type of information are spliced ​​together. That is, the first splicing information can be represented by splicing vectors.

[0040] Based on this, at least one target processing step is performed based on the target eye image and camera information, including: performing at least one target processing step based on the target eye image and first stitching information; wherein, the input information for the first target processing includes the target eye image and the first stitching information; the input information for subsequent target processing steps includes the output information of the previous target processing step and the first stitching information. In other words, camera information is already included within the first stitching information. In addition, the first stitching information also includes image manipulation information, which facilitates reasonable analysis by combining camera information and information representing the target region of the eye during target processing. In some embodiments, the steps of performing at least one target processing step based on the target eye image and camera information, and obtaining the gaze estimation result of the target eye image based on the output information of the last target processing step, can be performed with reference to the following steps A to C:

[0041] Step A: Input the target eye image and camera information into a preset gaze estimation model; wherein, the gaze estimation model includes at least one target network, and if there are multiple target networks, the multiple target networks are connected sequentially. Each target network is used to perform corresponding target processing, and the number of target networks is the same as the number of times the target processing is executed.

[0042] Step B involves performing target processing through the target network; wherein at least one target network corresponds one-to-one with at least one target processing step, and different target networks have the same network structure but different network parameters, including but not limited to step size. The input information of the target network is the input information of its corresponding target processing step, and the output information of the target network is the output information of its corresponding target processing step. For example, refer to... Figure 2 The diagram shows the structure of a gaze estimation model, simply illustrating three target networks: target network 1, target network 2, and target network 3. The step size of target network 1 is 8, the step size of target network 2 is 16, and the step size of target network 3 is 32. It should be noted that... Figure 2 This is just an example; in practical applications, the number of target networks and their respective step sizes can be flexibly set according to requirements, and no restrictions are imposed here. Additionally, Figure 2 The diagram illustrates that the input information of a non-first target network includes the output information of the previous target network and the first stitching information. The first stitching information includes camera information and image operation information, as detailed above. In practical applications, it may not be necessary to determine the target region or perform operations such as image matting and size adjustment in the target eye image. In this case, the input information of the target network may not include image operation information, but only the output information of the previous target network and camera information. The specific settings can be flexibly configured and are not restricted here.

[0043] For example, the target network includes a first network unit, a second network unit, and a third network unit. Based on this, step B can be performed with reference to steps B1 to B3 as follows:

[0044] Step B1 involves stitching different types of information contained in the input information of the target network through a first network unit to obtain target stitched information. In practical applications, these different types of information may include target eye images or output information from the previous target network, as well as camera information or first stitched information. The first network unit may include a stitching network layer capable of stitching different types of information to obtain target stitched information. For example, the target stitched information can be represented in the form of feature vectors. Before stitching different types of information, operations such as feature vector transformation or scale unification can be performed to ensure effective stitching of different types of information. The target stitched information comprehensively covers information such as target eye images or output information from the previous target network, as well as camera information and image manipulation information, facilitating accurate and comprehensive extraction of the required key features.

[0045] Step B2: Based on low-rank matrix factorization technology, the key information of the target is extracted from the target splicing information through the second network unit; the key information of the target includes key eye features and key camera features.

[0046] In a specific implementation, the second network unit may include a LoRA (Low-Rank Adaptation) network layer. The core idea of ​​the LoRA network layer is to achieve model fine-tuning by introducing low-rank matrix factorization (LMF) technology. In this embodiment, the gaze estimation model can be trained by inserting a LoRA network layer. Specifically, based on LMF technology, the LoRA layer can decompose the weight matrix of the gaze estimation model into the product of two low-rank matrices, achieving efficient and targeted fine-tuning of model parameters. This allows the trained gaze estimation model to selectively extract key features related to the target eye and key features related to the target camera. In a specific implementation, the LoRA layer can first perform dimensionality reduction on the target stitching information to extract key features from the target stitching information, and then perform dimensionality increase on the key features obtained from the dimensionality reduction to match the feature dimensions of the target stitching information. By increasing the dimensionality of the key features, each feature dimension can carry key information.

[0047] Step B3 involves fusing the target key information and specified information from the input information corresponding to the target processing through a third network unit, and obtaining output information based on the fusion processing result. The specified information includes information related to image features, such as information in the input information other than the first concatenation information. The third network unit may include a fusion network layer, which can fuse the target key information with the specified information to obtain a fusion processing result. This fusion processing result can be directly used as output information, or further processing such as deep feature extraction can be performed on the fusion processing result to obtain the output information; no restrictions are placed on this step.

[0048] Step C involves obtaining the gaze estimation result of the target eye image based on the output information of the target processing corresponding to the last target network. In practical applications, the output information of the target processing corresponding to the last target network can be directly used as the gaze estimation result of the target eye. Alternatively, the output information of the target processing corresponding to the last target network can be a fusion feature, which is then further analyzed and processed to obtain the gaze estimation result of the target eye.

[0049] For illustrative purposes, please refer to Figure 3 The schematic diagram of a target network illustrates that the target network includes at least a first network unit, a second network unit, and a third network unit connected in sequence. For example, the first network unit includes a splicing network layer, the second network unit includes a LoRA network layer, and the third network unit includes a fusion network layer. The specific functions of each of these network units can be found in the aforementioned related content and will not be repeated here. It should be noted that... Figure 3 This is merely an illustrative example and should not be considered a limitation. In practical applications, other network layers may also be included. For example, the first network unit and / or the third network unit may also include other network layers such as a dimension transformation layer. For example, the dimension transformation layer may be placed before the splicing network layer, and / or the dimension transformation layer may be placed after the fusion network layer, so as to connect the target networks with different parameters before and after. The specific settings can be flexibly configured.

[0050] In some implementation examples, the gaze estimation model is obtained through steps a to d as follows:

[0051] Step a: Obtain eye sample images. These eye sample images are images of eye samples captured by the camera sample, and they carry eye sample gaze labels and intrinsic and extrinsic parameters of the camera sample. The camera sample can be the same as or different from the target camera mentioned above; no restriction is placed here. Similarly, the eye sample can be the same as or different from the target eye. In practical applications, a large number of eye sample images can be collected. Different eye sample images can correspond to different eye samples, and different eye sample images can also correspond to different camera samples, thereby enhancing the robustness of the model.

[0052] Step b: Based on the intrinsic and extrinsic parameters of the eye sample image and camera sample, the gaze estimation result of the eye sample image is obtained through a preset initial network model. The structure of the initial network model is the same as that of the gaze estimation model. In other words, the gaze estimation model is a model obtained by adjusting the parameters of the initial network model. The method of obtaining the gaze estimation result of the eye sample image based on the intrinsic and extrinsic parameters of the eye sample image and camera sample through the preset initial network model can refer to the aforementioned method of obtaining the gaze estimation result through the gaze estimation model. That is, the gaze estimation result of the eye sample image is obtained by performing target processing through each target network in the initial network model. The specific method can be referred to the above content and will not be repeated here.

[0053] Step c: Based on the difference between the gaze estimation results of the eye sample images and the gaze labels of the eye samples, adjust the parameters of the first network unit, the second network unit, and the third network unit in the initial network model, and use the parameter-adjusted initial network model as the first network model.

[0054] In practical applications, the aforementioned differences can be represented by the loss value calculated based on a specified loss function. Then, the network parameters of the initial network model are adjusted with the goal of reducing the differences. This includes, but is not limited to, adjusting the parameters of the first, second, and third network units in the initial network model. If the initial network model also contains other network units, their parameters will also be adjusted. The adjustment of the initial network model parameters ends when a preset first condition is met, and the parameter-adjusted initial network model is used as the first network model. The first condition can be, for example, that the loss value corresponding to the aforementioned differences converges to a first preset threshold.

[0055] Step d: Obtain a gaze estimation model based on the first network model. In some implementations, the first network model can be directly used as the gaze estimation model. In other implementations, to further ensure that the final gaze estimation model can more accurately estimate the gaze of the target eye, the first network model can be calibrated using a calibration image of the target eye (with a known gaze direction) acquired by the target camera. This allows the second network units (such as LoRA network layers) in the first network model to learn the features of the target eye and the target camera, further optimizing the parameters of the second network units. This enables more accurate analysis of the target eye image with the known gaze direction, resulting in a more precise gaze estimation result. In some specific implementation examples, step d can be performed as follows: Steps d1 to d3:

[0056] Step d1: Acquire a calibration eye image. This calibration eye image is an image of the target eye captured by the target camera, and it carries a gaze label for the target eye and the intrinsic and extrinsic parameters of the target camera. In practical applications, a specified gaze point can be displayed on the interface to remind the user to focus their target eye on that gaze point, thereby acquiring the calibration eye image and obtaining the corresponding gaze label. The number of calibration eye images is not limited here.

[0057] Step d2 involves obtaining the gaze estimation result of the calibrated eye image based on the intrinsic and extrinsic parameters of the target camera and the calibrated eye image using the first network model. The specific method for obtaining the gaze estimation result can be found in the aforementioned related content and will not be repeated here.

[0058] Step d3: Based on the difference between the gaze estimation result of the calibrated eye image and the gaze label of the target eye, adjust the parameters of the second network unit in the initial network model, and use the parameter-adjusted first network model as the gaze estimation model. The difference between the gaze estimation result of the calibrated eye image and the gaze label of the target eye can also be represented by a loss value. Adjust the network parameters of the second network unit (such as the LoRA network layer) in the first network model with the goal of reducing this difference until a preset second condition is met, then stop adjusting the parameters of the first network model (that is, stop adjusting the parameters of the second network unit), and use the parameter-adjusted first network model as the gaze estimation model. The second condition can be that the loss value corresponding to the above difference converges to a second preset threshold. It should be noted that in step d3, only the parameters of the second network unit are adjusted, while the parameters of the first and third network units remain unchanged. This step can be understood as a fine-tuning and calibration step of the model. That is, when estimating the gaze of the target eye, a calibration eye image of the target eye can be acquired first, and the parameters of the second network unit can be updated online. Since it is only a local fine-tuning of the model parameters, this method of adjusting the model parameters is more efficient and faster. It helps the second network unit to learn the key features of the target eye and the target camera more accurately and reliably with the help of the calibration eye image. Thus, when performing gaze analysis on the target eye image, the gaze estimation results can be obtained more accurately.

[0059] For easier understanding, please refer to the following: Figure 4 The flowchart shown illustrates a method for obtaining a gaze estimation model, mainly including the following steps S402 to S412:

[0060] Step S402: Obtain an eye sample image; wherein, the eye sample image is an image of an eye sample acquired by the camera sample, and the eye sample image carries an eye sample gaze label and the intrinsic and extrinsic parameters of the camera sample.

[0061] Step S404: Based on the intrinsic and extrinsic parameters of the eye sample image and the camera sample, the gaze estimation result of the eye sample image is obtained through a preset initial network model; wherein, the structure of the initial network model is the same as the structure of the gaze estimation model.

[0062] Step S406: Based on the difference between the gaze estimation result of the eye sample image and the gaze label of the eye sample, adjust the parameters of the first network unit, the second network unit and the third network unit in the initial network model, and use the parameter-adjusted initial network model as the first network model.

[0063] Step S408: Obtain a calibration eye image; wherein, the calibration eye image is an image of the target eye captured by the target camera, and the calibration eye image carries the target eye gaze label and the intrinsic and extrinsic parameters of the target camera.

[0064] Step S410: Based on the calibrated eye image and the intrinsic and extrinsic parameters of the target camera, the gaze estimation result of the calibrated eye image is obtained through the first network model.

[0065] Step S412: Based on the difference between the gaze estimation result of the calibrated eye image and the gaze label of the target eye, adjust the parameters of the second network unit in the initial network model, and use the parameter-adjusted first network model as the gaze estimation model.

[0066] The specific implementation methods of the above steps can be referred to the aforementioned related content, and will not be repeated here. Through the above method, the model can be trained first using sample eye images, and then fine-tuned using calibrated eye images. This allows the final gaze estimation model to more accurately and reliably extract the key features of the target eye and the target camera, thus enabling more precise analysis of the gaze estimation results. Furthermore, this embodiment of the present disclosure introduces camera information when using the gaze estimation model for gaze estimation, and through a progressive approach, it can extract more in-depth and comprehensive key information based on the target eye image and camera information, further improving the accuracy of the gaze estimation results.

[0067] Corresponding to the aforementioned line-of-sight estimation method, this disclosure further provides a line-of-sight estimation device. Figure 5 This is a schematic diagram of a line-of-sight estimation device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 5 As shown, the line-of-sight estimation device includes:

[0068] The eye image acquisition module 502 is used to acquire the target eye image corresponding to the target eye whose gaze direction is to be estimated; wherein, the target eye image is the image of the target eye captured by the target camera;

[0069] The camera information acquisition module 504 is used to acquire the intrinsic and extrinsic parameters of the target camera and obtain camera information based on the intrinsic and extrinsic parameters;

[0070] The target processing execution module 506 is used to perform at least one target processing based on the target eye image and camera information, and obtain the gaze estimation result of the target eye image based on the output information of the last target processing; the target processing includes: extracting target key information based on the input information corresponding to the target processing, and fusing the target key information and specified information in the input information corresponding to the target processing to obtain the output information corresponding to the target processing;

[0071] The input information for the first target processing includes the target eye image and camera information, while the input information for subsequent target processing includes the output information of the previous target processing and camera information.

[0072] The apparatus provided in this disclosure fully considers the influence of the camera on gaze estimation. Compared with related technologies that do not introduce camera parameters, it can acquire the intrinsic and extrinsic parameters of the target camera used to capture the target eye image, obtain camera information based on the intrinsic and extrinsic parameters, and introduce the target eye image and camera information into gaze estimation. Moreover, it performs at least one target processing and obtains the gaze estimation result of the target eye image based on the output information of the last target processing. This progressive approach can extract more in-depth and comprehensive key information based on the target eye image and camera information, thereby helping to obtain more realistic and reliable gaze estimation results to better meet the needs.

[0073] In some embodiments, the target processing execution module 506 is specifically used to: input the target eye image and the camera information into a preset gaze estimation model; wherein the gaze estimation model includes at least one target network, and when there are multiple target networks, the multiple target networks are connected sequentially; perform target processing through the target networks; wherein the at least one target network has a one-to-one correspondence with the at least one target processing, and different target networks have the same network structure but different network parameters; and obtain the gaze estimation result of the target eye image based on the output information of the target processing corresponding to the last target network.

[0074] In some embodiments, the target network includes a first network unit, a second network unit, and a third network unit; the target processing execution module 506 is specifically used for: performing concatenation processing through the first network unit based on different types of information contained in the input information of the target network to obtain target concatenation information; extracting target key information from the target concatenation information through the second network unit based on low-rank matrix factorization technology; wherein the target key information includes key eye features and key camera features; performing fusion processing through the third network unit based on the target key information and specified information in the input information corresponding to the target processing, and obtaining output information based on the fusion processing result; wherein the specified information includes information related to image features.

[0075] In some embodiments, the apparatus further includes a model acquisition module for obtaining the gaze estimation model through the following steps: acquiring an eye sample image; wherein the eye sample image is an image of an eye sample captured by a camera sample, and the eye sample image carries an eye sample gaze label and the intrinsic and extrinsic parameters of the camera sample; based on the eye sample image and the intrinsic and extrinsic parameters of the camera sample, obtaining a gaze estimation result of the eye sample image through a preset initial network model; wherein the structure of the initial network model is the same as the structure of the gaze estimation model; based on the difference between the gaze estimation result of the eye sample image and the eye sample gaze label, adjusting the parameters of the first network unit, the second network unit, and the third network unit in the initial network model, and using the parameter-adjusted initial network model as the first network model; generating a gaze estimation model based on the first network model.

[0076] In some implementations, the model acquisition module is specifically used for: acquiring a calibration eye image; wherein the calibration eye image is an image of the target eye captured by the target camera, and the calibration eye image carries a target eye gaze label and the intrinsic and extrinsic parameters of the target camera; based on the calibration eye image and the intrinsic and extrinsic parameters of the target camera, obtaining a gaze estimation result of the calibration eye image through a first network model; based on the difference between the gaze estimation result of the calibration eye image and the target eye gaze label, adjusting the parameters of the second network unit in the initial network model, and using the parameter-adjusted first network model as the gaze estimation model.

[0077] In some embodiments, the apparatus further includes: a stitching module for determining a target region to be processed from the target eye image; wherein the target region is determined based on the iris of the target eye; performing a specified operation based on the target region and the target eye image; wherein the specified operation includes: extracting the target region from the target eye image to obtain a target region image, and performing a size adjustment operation on the target region image; performing stitching processing based on the camera information and the operation information of the specified operation to obtain first stitching information; the target processing execution module 506 is specifically used for: performing at least one target processing based on the target eye image and the first stitching information; wherein the input information for the first target processing includes the target eye image and the first stitching information; the input information for subsequent target processing includes the output information of the previous target processing and the first stitching information.

[0078] In some embodiments, the target eye image includes images of the target eye captured by different target cameras at the same time; the gaze estimation result of the target eye includes the three-dimensional gaze direction and three-dimensional pupil position of the target eye.

[0079] The line-of-sight estimation device provided in this disclosure can execute the line-of-sight estimation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0081] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0082] The following is for reference. Figure 6 This diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0083] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0084] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0085] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0086] In addition to the methods and devices described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0087] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the line-of-sight estimation method provided in embodiments of this disclosure.

[0088] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0089] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the line-of-sight estimation method in this disclosure.

[0090] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0091] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0092] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0093] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0094] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0095] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A line of sight estimation method, characterized by, include: Acquire the target eye image corresponding to the target eye whose gaze direction is to be estimated; wherein, the target eye image is the image of the target eye captured by the target camera; Obtain the intrinsic and extrinsic parameters of the target camera, and obtain camera information based on the intrinsic and extrinsic parameters; Based on the target eye image and the camera information, at least one target processing is performed, and a gaze estimation result of the target eye image is obtained based on the output information of the last target processing; the target processing includes: extracting key target information based on the input information corresponding to the target processing, and fusing the key target information and specified information in the input information corresponding to the target processing to obtain the output information corresponding to the target processing; The input information for the first target processing includes the target eye image and the camera information, while the input information for the non-first target processing includes the output information of the previous target processing and the camera information.

2. The method of claim 1, wherein, The step of performing at least one target processing operation based on the target eye image and the camera information, and obtaining a gaze estimation result of the target eye image based on the output information of the last target processing operation, includes: The target eye image and the camera information are input into a preset gaze estimation model; wherein, the gaze estimation model includes at least one target network, and when there are multiple target networks, the multiple target networks are connected sequentially; Target processing is performed through the target network; wherein, the at least one target network corresponds one-to-one with the at least one target processing, and the different target networks have the same network structure but different network parameters; The gaze estimation result of the target eye image is obtained based on the output information of the target processing corresponding to the last target network.

3. The method of claim 2, wherein, The target network includes a first network unit, a second network unit, and a third network unit; The execution of target processing through the target network includes: Based on the different types of information contained in the input information of the target network, the first network unit performs splicing processing to obtain the target spliced ​​information; Based on low-rank matrix factorization, key target information is extracted from the target splicing information through the second network unit; wherein, the key target information includes key eye features and key camera features; Based on the target key information and the specified information in the input information corresponding to the target processing, the third network unit performs fusion processing, and output information is obtained based on the fusion processing result; wherein, the specified information includes information related to image features.

4. The method of claim 3, wherein, The gaze estimation model is obtained through the following steps: Acquire an eye sample image; wherein the eye sample image is an image of an eye sample captured by a camera sample, and the eye sample image carries an eye sample gaze label and the intrinsic and extrinsic parameters of the camera sample; Based on the intrinsic and extrinsic parameters of the eye sample image and the camera sample, the gaze estimation result of the eye sample image is obtained through a preset initial network model; wherein, the structure of the initial network model is the same as the structure of the gaze estimation model; Based on the difference between the gaze estimation result of the eye sample image and the gaze label of the eye sample, the parameters of the first network unit, the second network unit and the third network unit in the initial network model are adjusted, and the initial network model with adjusted parameters is used as the first network model. A gaze estimation model is obtained based on the first network model.

5. The method of claim 4, wherein, The gaze estimation model obtained based on the first network model includes: Acquire a calibration eye image; wherein the calibration eye image is an image of the target eye captured by the target camera, and the calibration eye image carries a target eye gaze tag and the intrinsic and extrinsic parameters of the target camera; Based on the calibrated eye image and the intrinsic and extrinsic parameters of the target camera, the gaze estimation result of the calibrated eye image is obtained through the first network model; Based on the difference between the gaze estimation result of the calibrated eye image and the gaze label of the target eye, the parameters of the second network unit in the initial network model are adjusted, and the first network model with adjusted parameters is used as the gaze estimation model.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The target region to be processed is determined from the target eye image; wherein the target region is determined based on the iris of the target eye; Perform a specified operation based on the target region and the target eye image; wherein the specified operation includes: extracting the target region from the target eye image to obtain a target region image, and performing a size adjustment operation on the target region image; The first stitching information is obtained by stitching together the camera information and the operation information of the specified operation. The step of performing at least one target processing based on the target eye image and the camera information includes: performing at least one target processing based on the target eye image and the first stitching information; wherein, the input information for the first target processing includes the target eye image and the first stitching information; the input information for subsequent target processing includes the output information of the previous target processing and the first stitching information.

7. The method according to any one of claims 1 to 5, characterized in that, The target eye image includes images of the target eye captured by different target cameras at the same time; the gaze estimation result of the target eye includes the three-dimensional gaze direction and the three-dimensional pupil position of the target eye.

8. A line-of-sight estimation device, characterized in that, include: An eye image acquisition module is used to acquire a target eye image corresponding to the target eye whose gaze direction is to be estimated; wherein, the target eye image is an image of the target eye captured by a target camera; A camera information acquisition module is used to acquire the intrinsic and extrinsic parameters of the target camera and obtain camera information based on the intrinsic and extrinsic parameters; The target processing execution module is used to perform at least one target processing based on the target eye image and the camera information, and obtain a gaze estimation result of the target eye image based on the output information of the last target processing; the target processing includes: extracting key target information based on the input information corresponding to the target processing, and fusing the key target information and specified information in the input information corresponding to the target processing to obtain the output information corresponding to the target processing; The input information for the first target processing includes the target eye image and the camera information, while the input information for the non-first target processing includes the output information of the previous target processing and the camera information.

9. An electronic device, characterized in that, The electronic device includes: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the line-of-sight estimation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the line-of-sight estimation method according to any one of claims 1-7.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the line-of-sight estimation method according to any one of claims 1-7.

Images