Data generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data generation method and device, electronic equipment and a storage medium. The method comprises the steps that a simulation eye segmentation graph and a first sight line direction corresponding to the simulation eye segmentation graph are acquired; generating a simulation eye image based on the simulation eye segmentation image through an image generation model; wherein the image generation model is constructed based on a first real eye segmentation image and a first real eye image; determining a first data pair according to the simulated eye image and the first line-of-sight direction; wherein the first data pair is used for constructing a sight tracking model. On the basis of relatively low hardware dependency, the data pair of the eye image and the sight line direction with relatively high quality can be efficiently obtained.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, and particularly, to a data generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the related art, a gaze tracking model can be constructed by using pairs of eye images and gaze directions.

[0003] The above pair acquisition process usually includes a guide point displayed on a screen, and an eye image of a subject is acquired in response to determining that the subject looks at the guide point. This acquisition process is inefficient because the subject needs time to look at the guide point and determine the direction of the gaze. It is difficult for the subject to concentrate for a long time, and there is a risk of acquiring an incorrect eye image, which affects the accuracy of the model construction. Since the guide point needs to be displayed correctly, the product hardware needs to be complete, and the data pair acquisition cannot be performed in the early stage, which affects the progress of the model construction.

[0004] In summary, how to efficiently acquire pairs of eye images and gaze directions with high quality without relying on hardware has become a technical problem to be solved. SUMMARY

[0005] Embodiments of the present disclosure provide a data generation method and device, electronic equipment and storage medium, which can efficiently acquire pairs of eye images and gaze directions with high quality with low hardware dependency.

[0006] In a first aspect, the embodiments of the present disclosure provide a data generation method, comprising:

[0007] obtaining a simulated eye segmentation map and a first gaze direction corresponding to the simulated eye segmentation map;

[0008] generating a simulated eye image based on the simulated eye segmentation map by using an image generation model, wherein the image generation model is constructed based on a first real eye segmentation map and a first real eye image;

[0009] determining a first data pair according to the simulated eye image and the first gaze direction, wherein the first data pair is used to construct a gaze tracking model.

[0010] In a second aspect, the embodiments of the present disclosure also provide a data generation device, comprising:

[0011] an obtaining module configured to obtain a simulated eye segmentation map and a first gaze direction corresponding to the simulated eye segmentation map;

[0012] an image generation module configured to generate a simulated eye image based on the simulated eye segmentation map by using an image generation model, wherein the image generation model is constructed based on a first real eye segmentation map and a first real eye image;

[0013] a data pair determination module configured to determine a first data pair according to the simulated eye image and the first gaze direction, wherein the first data pair is used to construct a gaze tracking model.

[0014] In a third aspect, an electronic device is provided, and the electronic device includes:

[0015] one or more processors;

[0016] a storage device configured to store one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data generation method according to any of the embodiments of the present disclosure.

[0018] In a fourth aspect, a storage medium containing computer executable instructions is provided, and the computer executable instructions are used to execute the data generation method according to any of the embodiments of the present disclosure when executed by a computer processor.

[0019] In the technical solution of the embodiments of the present disclosure, a simulated eye segmentation map and a first gaze direction corresponding to the simulated eye segmentation map can be obtained; a simulated eye image is generated based on the simulated eye segmentation map by using an image generation model, wherein the image generation model is constructed based on a first real eye segmentation map and a first real eye image; and a first data pair is determined according to the simulated eye image and the first gaze direction, wherein the first data pair is used to construct a gaze tracking model.

[0020] By using the image generation model, a simulated eye image can be generated based on a simulated eye segmentation map, and a data pair of a simulated eye image and a gaze direction can be determined. Thus, the data pair can be collected without using a guided point, and the efficiency, naturalness and correctness of data collection can be improved. In addition, the technical solution of the embodiments of the present disclosure can collect data in advance in an early stage of a hardware development cycle, and thus the progress of the overall project can be accelerated. The data pair of an eye image and a gaze direction with high quality can be efficiently obtained with low dependence on hardware. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent as various embodiments of the present disclosure are described in conjunction with the following drawings, in which like reference numbers represent like elements throughout the drawings. It should be noted that the drawings are schematic and elements in the drawings are not necessarily to scale.

[0022] Figure 1 A flowchart of a data generation method provided by an embodiment of the present disclosure is shown in FIG. 1.

[0023] Figure 2 An eye segmentation diagram in a data generation method provided by an embodiment of the present disclosure is shown in FIG. 2.

[0024] Figure 3 A schematic block diagram of a construction process of an image generation model in a data generation method provided by an embodiment of the present disclosure is shown in FIG. 3.

[0025] Figure 4 A flowchart of acquiring a simulated eye segmentation map in a data generation method provided by an embodiment of the present disclosure is shown in FIG. 4.

[0026] Figure 5 A schematic diagram of a simulation system in a data generation method provided by an embodiment of the present disclosure is shown in FIG. 5.

[0027] Figure 6 A flowchart of a data generation method provided by an embodiment of the present disclosure is shown in FIG. 6.

[0028] Figure 7 A structural schematic diagram of a data generation apparatus provided by an embodiment of the present disclosure is shown in FIG. 7.

[0029] Figure 8 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown in FIG. 8. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail by referring to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided to make the present disclosure more thorough and complete. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0031] It is understood that each step described in the method embodiments of the present disclosure can be executed in different orders, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "include," and derivations thereof, is used herein to mean "including, but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related terms shall be construed accordingly.

[0033] It should be noted that the terms "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the terms "one", "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that "one or more" should be understood unless otherwise explicitly indicated in the context.

[0035] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0036] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws, regulations and provisions.

[0037] Figure 1 A flowchart of a data generation method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case of generating a data pair of an eye image and a gaze direction. The method can be executed by a data generation device, which can be implemented in the form of software and / or hardware, and can be configured in an electronic device, such as a computer.

[0038] As shown in Figure 1 The data generation method provided by the embodiment can include:

[0039] S110, obtaining a simulated eye segmentation map and a first gaze direction corresponding to the simulated eye segmentation map.

[0040] In the embodiment of the present disclosure, the eye image includes an infrared image of the eye under an infrared light source, and the eye segmentation map can be understood as a segmented image of the eye image. The eye segmentation map can include a sclera region, an iris region, a pupil region, and a light spot region. The light spot region can be understood as a region where a light spot (also known as Purkinje spot) is formed by reflection of the light source on the cornea region of the eye.

[0041] Exemplarily, Figure 2An eye segmentation diagram in a data generation method provided by an embodiment of the present disclosure. Referring to Figure 2 The white scattered small dot area is a light spot area, and the three large-area continuous areas from light to dark are a pupil area, an iris area, and a sclera area, and the black background is other skin areas outside the eye frame.

[0042] In this embodiment, the simulation eye segmentation diagram includes an eye segmentation diagram obtained based on a simulation system, and the simulation eye image includes an eye image generated based on the simulation eye segmentation diagram. The eye image obtained through real acquisition can be referred to as a real eye image. The real eye image can be segmented through an existing image segmentation algorithm to obtain a real eye segmentation diagram.

[0043] In an embodiment of the present disclosure, the prefixes "first", "second", etc. of the real eye image can distinguish the eye images collected at different stages. The prefixes "first", "second", etc. of the real eye segmentation diagram can correspond to the segmentation images of different real eye images. Among them, the real eye images corresponding to different prefixes are essentially the same, and the real eye segmentation diagrams corresponding to different prefixes are essentially the same.

[0044] In this embodiment, the simulation system can include a parameter-controllable eyeball model, so that the gaze direction corresponding to each simulation eye segmentation diagram can be obtained during the determination of the simulation eye segmentation diagram based on the simulation system, and can be referred to as the first gaze direction. It can be understood that the gaze directions corresponding to the different prefixes "first", "second", etc. in this paper are essentially the same, for example, they can all include the angle of the vertical direction and the angle of the horizontal direction.

[0045] S120, generating a simulation eye image based on the simulation eye segmentation diagram through an image generation model.

[0046] In an embodiment of the present disclosure, the image generation model is constructed based on the first real eye segmentation diagram and the first real eye image.

[0047] Among them, the acquisition device of the first real eye image is relatively simple, and can not include a display system, but only a wearable device including a near-eye camera, so that the first real eye image can be acquired in the early stage of hardware development of the wearable device to speed up the progress of the overall project. During the acquisition of the first real eye image, the collector does not need to cooperate and can look in different directions at will, so that efficient acquisition of natural eye images can be realized.

[0048] Among them, the first real eye image can be segmented according to a pre-constructed segmentation model of the eye image to obtain a first real eye segmentation diagram corresponding to each first real eye image.

[0049] In this embodiment, the image generation model can include a diffusion model, and the noise image can be denoised in multiple steps to generate the eye image. In the multiple-step denoising process, the control condition can also be determined according to the eye segmentation map, and the generation of the eye image is controlled according to the control condition, so that each region in the eye image has a corresponding relationship with each region in the eye segmentation map.

[0050] Exemplarily, Figure 3 A schematic block diagram of the construction process of the image generation model in the data generation method provided by the embodiments of the present disclosure is shown. Referring to Figure 3 In some optional implementation manners, the construction process of the image generation model can include: performing first-stage construction of the image generation model according to the second real eye image; and performing second-stage construction of the image generation model according to the first real eye segmentation map and the first real eye image.

[0051] The second real eye image can include the first real eye image, and in addition, can also include a real eye image obtained in other manners. It should be noted that the second real eye image should be obtained in compliance with the requirements of corresponding laws, regulations and relevant provisions. It can be considered that the data amount of the second real eye image is greater than that of the first real eye image, and the second real eye image has high richness and strong diversity.

[0052] Figure 3 (a) Exemplarily shows the first-stage construction process of the image generation model, and the first-stage construction process can include: randomly sampling a first noise image; wherein the first noise image, for example, includes a Gaussian noise image. The first noise image is denoised by the image generation model for T (T≥1) steps to obtain a first predicted eye image; wherein the image generation model used in each denoising process can be the same. A first loss value is constructed according to the first predicted eye image and the second real eye image; wherein the first loss value can be constructed by using an existing image loss value constructor. The image generation model is constructed in the first stage according to the first loss value.

[0053] Figure 3 (b) Exemplarily shows the second-stage construction process of the image generation model, and the second-stage construction process can include: randomly sampling a second noise image; wherein the second noise image can also include a Gaussian noise image. Feature extraction is performed on the first real eye segmentation map, and a sample control condition is constructed according to the extracted features; wherein the feature extraction can be performed on the first real eye segmentation map by using an existing image feature extraction manner; wherein the extracted features can be directly used as the sample control condition, or the extracted features can be processed by encoding compression, size transformation, etc. to obtain the sample control condition. For example, Figure 3(b), the sample control condition can be determined according to the first real eye segmentation map by controlling the condition model. The image generation model is initialized according to the first stage construction result, and the second noise image is denoised for T steps according to the sample control condition by using the initialized image generation model, to obtain a second predicted eye image; wherein the image generation model used in each step of denoising process can be the same, and the sample control condition can be the same. A second loss value is constructed according to the second predicted eye image and the corresponding first real eye image; wherein the second predicted eye image has a corresponding relationship with the first real eye segmentation map, the first real eye segmentation map has a corresponding relationship with the first real eye image, and the second predicted eye image also has a corresponding relationship with the first real eye image; wherein the second loss value can be constructed by using the existing image loss value constructor. The image generation model is constructed in the second stage according to the second loss value.

[0054] In these optional implementations, the generalization ability and generation effect of the image generation model can be improved by using the second real eye image with high richness and strong diversity to construct the image generation model in the first stage, so that the image generation model can generate natural and realistic eye images. The image generation model can generate eye images according to the eye segmentation map by using the first real eye segmentation map and the corresponding first real eye image to construct the image generation model.

[0055] In this embodiment, the third noise image can be randomly sampled; wherein the third noise image can also include a Gaussian noise image. Feature extraction is performed on the simulation eye segmentation map, and a control condition is constructed according to the extracted features; wherein the feature extraction can be performed on the simulation eye segmentation map by using the existing image feature extraction method; wherein the control condition can be constructed according to the construction method of the sample control condition. The third noise image is denoised for T steps according to the control condition by using the constructed image generation model, to obtain a simulation eye image.

[0056] S130, determining a first data pair according to the simulation eye image and the first gaze direction; wherein the first data pair is used to construct a gaze tracking model.

[0057] In the embodiments of the present disclosure, the simulation eye image has a corresponding relationship with the simulation eye segmentation map, the simulation eye segmentation map has a corresponding relationship with the first gaze direction, and the simulation eye image and the first gaze direction also have a corresponding relationship. The first data pair containing the simulation eye image and the first gaze direction can be constructed according to the corresponding relationship.

[0058] The first data pair can be used to construct the gaze tracking model, and the construction process can include: inputting a simulated eye image into the gaze tracking model; outputting a gaze direction prediction value through the gaze tracking model; constructing a third loss value based on the first gaze direction and the gaze direction prediction value; and constructing the gaze tracking model based on the third loss value. Since the first gaze direction is obtained from simulation, it has absolute accuracy; and since the simulated eye image is generated by an image generation model, it does not require acquisition based on guide points, which improves data collection efficiency, data naturalness, and accuracy. By constructing the gaze tracking model based on the first data pair, the gaze tracking model can achieve better gaze direction prediction performance.

[0059] In the technical solution of this disclosure embodiment, a simulated eye segmentation map and a first gaze direction corresponding to the simulated eye segmentation map can be obtained; a simulated eye image is generated based on the simulated eye segmentation map using an image generation model; wherein, the image generation model is constructed based on the first real eye segmentation map and the first real eye image; a first data pair is determined according to the simulated eye image and the first gaze direction; wherein, the first data pair is used to construct a gaze tracking model.

[0060] By using an image generation model, a simulated eye image can be generated based on a segmented image of the simulated eye, thus determining the data pair of the simulated eye image and the gaze direction. Therefore, data pair acquisition does not require a guide point-based approach, improving data collection efficiency, data naturalness, and accuracy. Furthermore, the technical solution of this disclosure allows for early data acquisition in the early stages of the hardware development cycle, accelerating the overall project progress. It enables the efficient acquisition of high-quality eye image and gaze direction data pairs with low hardware dependence.

[0061] This embodiment can be combined with various optional solutions in the data generation methods provided in the above embodiments. The data generation method provided in this embodiment details the process of obtaining a simulated eye segmentation map. By constructing a simulation system containing an eyeball model, the iris region, pupil region, and light spot region of the eyeball model under a light source can be simulated to obtain an initial simulated segmentation map of the sclera-free region. By searching for a segmentation map similar to the initial simulated segmentation map in the first real eye segmentation map, the real target sclera region can be extracted. Furthermore, based on the initial simulated segmentation map and the target sclera region, a simulated eye segmentation map that closely resembles the real situation can be synthesized.

[0062] Figure 4 This is a schematic flowchart illustrating the process of obtaining a simulated eye segmentation map in a data generation method provided in an embodiment of this disclosure. Figure 4 As shown, the data generation method provided in this embodiment, in which the simulated eye segmentation map is obtained, may include:

[0063] S410. Obtain the initial simulation segmentation map of the eyeball model; wherein, the initial simulation segmentation map includes the first pupil region.

[0064] In this embodiment, since the shape of the sclera region is related to variables such as the shape of the eye socket and the open / closed state of the eyelids, and these variables have low controllability, they can be omitted initially. Instead, a simulation system is used to simulate the segmentation map of the eye without the sclera region to obtain an initial simulated segmentation map. It can be assumed that the initial simulated segmentation map may include the iris region and the light spot region, in addition to the first pupil region.

[0065] In some optional implementations, obtaining the initial simulation segmentation map of the eye model may include: acquiring a simulation system containing the eye model, a light source, and a camera; using the simulation system, determining the image of the eye model in the camera under the light source based on the received simulation parameters, and obtaining the initial simulation segmentation map; wherein the simulation parameters include the biological parameters of the eye model, and the device parameters of the light source and the camera; wherein the biological parameters include the first line of sight direction.

[0066] For example, Figure 5 This is a schematic diagram of a simulation system in a data generation method provided in an embodiment of the present disclosure. Figure 5 A two-dimensional side view of the eye model and simulated optical path in a three-dimensional simulation system is shown. See also Figure 5 The eyeball model can be approximated by two spheres: the sclera and the cornea, with the interface between the two spheres serving as the iris. The center of the pupil and the center of the iris can coincide, and the pupil can be simulated as an ellipse that approximates a circle.

[0067] Figure 5 In this system, two infrared LEDs can be deployed as light sources, and an infrared camera can be deployed below the eyeball model to obtain an initial simulation segmentation map. The size, reflectivity, and refractive index of the eyeball model can be pre-set based on existing medical data. The first line of sight can also be input as a biological parameter into the simulation model to control the position and shape of the pupil within the eyeball model. Furthermore, the device parameters of the light sources and cameras can be configured according to the hardware design parameters of the wearable device under development. These device parameters include, for example, the number, position, emission range, and emission frequency of the light sources, as well as the number, position, acquisition range, and acquisition frequency of the cameras.

[0068] The simulation system can simulate the imaging of an eye model under a light source in a camera based on simulation parameters such as biological parameters and equipment parameters. The simulated imaging can include at least one of the following: simulating the imaging of the pupil in the eye model after refraction on the corneal surface in an infrared camera; simulating the imaging of the light source after reflection on the corneal surface in an infrared camera; simulating the imaging of the iris edge in the camera. In extreme camera views, the iris edge may be the tangent from the camera to the corneal bulb.

[0069] Since the eye model lacks texture details, its image in the camera can only show the division of the iris region, pupil region, and spot region, without showing the texture of each region. Therefore, the initial simulation segmentation map of the eye model can be determined based on the image from the camera in the simulation system. For example, different filling methods can be used to fill different regions in the image to obtain the initial simulation segmentation map of the eye model.

[0070] Among these optional implementation methods, the simulation system can obtain an initial simulation segmentation map with controllable key features such as the position and shape of the pupil and light spot, high diversity, and high accuracy in matching the line of sight direction.

[0071] S420. Based on the similarity of the first pupil region and the second pupil region in the first real eye segmentation image in the preset dimension parameters, determine the target scleral region from the scleral region of the first real eye segmentation image.

[0072] In this embodiment, the first realistic eye segmentation image may include, in addition to the second pupil region, a sclera region, an iris region, and a light spot region. By calculating the similarity between the first pupil region and the second pupil region in a preset dimensional parameter, the sclera region that best matches the initial simulation segmentation image can be found from the first realistic eye segmentation image for use in the synthesis of the simulated eye segmentation image.

[0073] Since the center position of the pupil region is related to the user's wearing position, and the major axis of the pupil region is closely related to the direction of gaze, the preset dimensional parameters may include at least one of the following: the center position of the ellipse fitted based on the pupil region, and the angle between the major axis of the ellipse and a preset baseline. The center position can be represented by pixel coordinates in the segmentation image. The preset baseline may include, for example, a horizontal reference axis used to define the direction of gaze.

[0074] Specifically, the first pupil region can be fitted to a first ellipse, and the first center position of the first ellipse and the first angle between the major axis of the first ellipse and a preset baseline can be obtained. The second pupil region can be fitted to a second ellipse, and the second center position of the second ellipse and the second angle between the major axis of the second ellipse and the preset baseline can be obtained. Specifically, K (K≥1) nearest neighbor pupil regions can be determined from the second pupil region based on the K-nearest neighbor method, according to the first and second center positions, and / or based on the first and second angles. Furthermore, the first similarity between the first and second center positions can be determined based on existing coordinate similarity algorithms; the second similarity between the first and second angles can be determined based on existing numerical similarity algorithms. And similar pupil regions can be determined from the second pupil region based on the first similarity and / or the second similarity.

[0075] Specifically, the target pupil region can be determined based on the K nearest neighbor pupil regions or similar pupil regions. For example, the pupil region with the highest similarity among the K nearest neighbor pupil regions or similar pupil regions can be determined as the target pupil region; or, the top N pupil regions with a similarity greater than a preset threshold and ranked by similarity among the K nearest neighbor pupil regions or similar pupil regions can be determined as the target pupil region. Furthermore, the scleral region of the first real eye segmentation map to which the target pupil region belongs can be determined as the target scleral region.

[0076] S430. Based on the initial simulation segmentation map and the target scleral region, determine the simulation eye segmentation map.

[0077] In this embodiment, the initial simulated segmentation map can be combined with the actual target scleral region to obtain a complete simulated eye segmentation map with a known line of sight direction.

[0078] In some optional implementations, determining the simulated eye segmentation map based on the initial simulated segmentation map and the target scleral region may include: determining the synthesis position based on the center position of the second pupil region in the first real eye segmentation map to which the target scleral region belongs; obtaining at least one of the following enhancement parameters: displacement parameter, scaling parameter, and rotation parameter; and synthesizing the target scleral region onto the initial simulated segmentation map based on the synthesis position and enhancement parameters to obtain the simulated eye segmentation map.

[0079] Specifically, the second center position corresponding to the target scleral region can be determined as the synthesis position; at least one enhancement parameter among the displacement parameter, scaling parameter, and rotation parameter of the center position can be randomly generated; based on the synthesis position, the target scleral region can be processed according to the enhancement parameter, and the processed target scleral region can be synthesized into the initial simulation segmentation map to obtain a complete simulation eye segmentation map that closely resembles the real situation.

[0080] Among these optional implementations, enhancing the target scleral region based on enhancement parameters can enrich the diversity of simulated eye segmentation images. Since the pupil region and spot region are not altered during the synthesis process, the gaze direction of the simulated eye segmentation image is the same as the first gaze direction of the corresponding initial simulated segmentation image, thus obtaining a first data pair consisting of the simulated eye image and the corresponding first gaze direction.

[0081] The technical solution of this disclosure provides a detailed description of the process for obtaining a simulated eye segmentation map. By constructing a simulation system including an eyeball model, the iris region, pupil region, and light spot region of the eyeball model under a light source can be simulated to obtain an initial simulated segmentation map of the sclera-free region. By searching for a segmentation map similar to the initial simulated segmentation map from the first real eye segmentation map, the real target sclera region can be extracted. Furthermore, based on the initial simulated segmentation map and the target sclera region, a simulated eye segmentation map that closely resembles reality can be synthesized. In addition, the data generation method provided in this disclosure embodiment belongs to the same disclosed concept as the data generation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0082] This embodiment can be combined with various optional solutions in the data generation methods provided in the above embodiments. The data generation method provided in this embodiment supplements the data pair generation process. In the later stages of the hardware development cycle, a third real eye image and its corresponding second gaze direction can be acquired through hardware based on guide point acquisition. Furthermore, by segmenting, enhancing, and regenerating the third real eye image, rich eye images can be generated without changing the second gaze direction, which can improve the diversity of data pairs and is beneficial to improving the construction effect of the gaze tracking model.

[0083] For example, Figure 6 This is a schematic flowchart illustrating a data generation method provided in an embodiment of this disclosure. Figure 6 As shown, the data generation method provided in this embodiment may further include:

[0084] S610. Obtain a third real eye image based on the guide point acquisition, and the second gaze direction corresponding to the third real eye image.

[0085] In this embodiment of the disclosure, a first data pair can be determined in the early stages of the hardware development cycle for constructing a gaze tracking model. In the later stages of the hardware development cycle, a guide point can be displayed through a display module in the hardware device, and in response to determining that the subject is looking at the guide point, an eye image can be acquired to obtain a third real eye image and the corresponding second gaze direction.

[0086] Since the gaze tracking model is constructed using the first data pair, it already has a superior gaze direction prediction capability. Therefore, it can collect a smaller number of third-party real eye images to achieve fine-tuning of the gaze tracking model applied to the device.

[0087] S620. Segment the third real eye image to obtain the second real eye segmentation image.

[0088] In this embodiment, the third real eye image can be segmented using existing image segmentation algorithms to obtain the second real eye segmentation image.

[0089] S630. Perform data augmentation on the second real eye segmentation map to obtain the third real eye segmentation map.

[0090] In this embodiment, for example, data enhancement can be performed on the second real eye segment image by adding different noise levels to obtain eye segment images with varying image quality. Alternatively, data enhancement can be performed on the second real eye segment image by adjusting the edges of the sclera region to simulate blinking, resulting in eye segment images with different blinking degrees. Furthermore, other methods can be used to enhance the data of the second real eye segment image, and the iris region, pupil region, and light spot region can be fixed during the enhancement process to avoid altering the second gaze direction.

[0091] S640. Using an image generation model, an enhanced eye image is generated based on a third real eye segmentation map.

[0092] The image generation model is constructed based on a first real eye segmentation map and a first real eye image. The process of generating enhanced eye images using the image generation model is similar to generating simulated eye images using the same model, and will not be elaborated upon here. Because the image generation model inherently possesses versatility, it can generate a wide variety of enhanced eye images. Therefore, while ensuring the fine-tuning effect of the line tracing model, the amount of data collected for the third real eye image can be reduced, thus reducing the time spent on subsequent data acquisition.

[0093] S650. Based on the third real eye image, the enhanced eye image, and the second gaze direction, determine a second data pair; wherein the second data pair is used to construct a gaze tracking model.

[0094] Since the third real eye image and the second gaze direction have a corresponding relationship during the acquisition process, the enhanced eye image does not change the gaze direction of the third real eye image, and the enhanced eye image and the second gaze direction also have a corresponding relationship. A second data pair can be constructed based on the third real eye image and its corresponding second gaze direction, as well as the enhanced eye image and its corresponding second gaze direction. The gaze tracking model can be fine-tuned based on this second data pair, and the fine-tuning process can refer to the gaze tracking model construction process, which will not be elaborated here.

[0095] The technical solution of this disclosure supplements the data pair generation process. In the later stages of the hardware development cycle, a third real eye image and its corresponding second gaze direction can be acquired via hardware based on guide point acquisition. Furthermore, by segmenting, enhancing, and regenerating the third real eye image, richer eye images can be generated without changing the second gaze direction, increasing the diversity of data pairs and improving the construction effect of the gaze tracking model. In addition, the data generation method provided in this disclosure belongs to the same disclosed concept as the data generation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0096] Figure 7 This is a schematic diagram of a data generation apparatus provided in an embodiment of the present disclosure. The data generation apparatus provided in this embodiment is suitable for generating data pairs of eye images and gaze direction.

[0097] like Figure 7 As shown, the data generation apparatus provided in this embodiment may include:

[0098] The acquisition module 710 is used to acquire the simulated eye segmentation map and the first viewing direction corresponding to the simulated eye segmentation map;

[0099] The image generation module 720 is used to generate a simulated eye image based on a simulated eye segmentation map using an image generation model; wherein, the image generation model is constructed based on a first real eye segmentation map and a first real eye image;

[0100] The data pair determination module 730 is used to determine a first data pair based on the simulated eye image and the first gaze direction; wherein, the first data pair is used to construct a gaze tracking model.

[0101] In some alternative implementations, the module can be used for:

[0102] Obtain the initial simulation segmentation map of the eyeball model; wherein, the initial simulation segmentation map includes the first pupil region;

[0103] Based on the similarity of the first pupil region and the second pupil region in the first real eye segmentation map in a preset dimension parameter, the target scleral region is determined from the scleral region of the first real eye segmentation map.

[0104] Based on the initial simulation segmentation map and the target scleral region, the simulated eye segmentation map is determined.

[0105] In some alternative implementations, the module can be used for:

[0106] Obtain a simulation system that includes an eye model, light source, and camera;

[0107] The simulation system determines the image of the eyeball model in the camera under the light source based on the received simulation parameters, and obtains the initial simulation segmentation map.

[0108] The simulation parameters include the biological parameters of the eyeball model, as well as the device parameters of the light source and camera; among them, the biological parameters include the first line of sight direction.

[0109] In some optional implementations, the preset dimension parameters include at least one of the following: the center position of the ellipse fitted based on the pupil region, and the angle between the major axis of the ellipse and the preset baseline.

[0110] In some alternative implementations, the module can be used for:

[0111] The synthesis location is determined based on the center position of the second pupil region in the first real eye segmentation image to which the target scleral region belongs;

[0112] Obtain at least one of the following enhancement parameters: translation parameter, scaling parameter, and rotation parameter;

[0113] Based on the synthesis location and enhancement parameters, the target scleral region is synthesized into the initial simulation segmentation map to obtain a simulated eye segmentation map.

[0114] In some alternative implementations, the data generation apparatus may further include:

[0115] The model building module is used to build image generation models based on the following process:

[0116] The first stage of image generation model construction is carried out based on the second real eye image;

[0117] Based on the first real eye segmentation map and the first real eye image, the image generation model is constructed in the second stage; wherein the amount of data in the second real eye image is greater than the amount of data in the first real eye image.

[0118] In some optional implementations, the acquisition module can also be used to acquire a third real eye image based on the guide point, and the second gaze direction corresponding to the third real eye image;

[0119] The data generation device may further include: a segmentation module for segmenting the third real eye image to obtain a second real eye segmentation map;

[0120] The data generation device may further include: an enhancement module for performing data enhancement on the second real eye segmentation map to obtain a third real eye segmentation map;

[0121] The image generation module can also be used to generate enhanced eye images based on a third real eye segmentation map using an image generation model;

[0122] The data pair determination module can also be used to determine a second data pair based on a third real eye image, an enhanced eye image, and a second gaze direction; wherein the second data pair is used to construct a gaze tracking model.

[0123] The data generation apparatus provided in this disclosure can execute the data generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0124] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0125] The following is for reference. Figure 8 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 8 The diagram below shows the structure of the terminal device or server 800. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0126] like Figure 8 As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0127] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0128] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, it performs the functions defined above in the data generation method of embodiments of this disclosure.

[0129] The electronic device provided in this embodiment and the data generation method provided in the above embodiments belong to the same disclosed concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0130] This disclosure provides a storage medium for computer-executable instructions, which, when executed by a computer processor, can be used to perform the data generation method provided in the above embodiments.

[0131] It should be noted that the storage medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory (FLASH), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores executable instructions that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable executable instructions. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit executable instructions for use by or in connection with an instruction execution system, apparatus, or device. Executable instructions contained on the storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0132] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0133] The aforementioned storage medium may be included in the aforementioned electronic device; or it may exist independently and not be assembled into the electronic device.

[0134] The aforementioned storage medium carries one or more executable instructions, which, when executed by the electronic device, cause the electronic device to:

[0135] Obtain a simulated eye segmentation image and a first gaze direction corresponding to the simulated eye segmentation image; generate a simulated eye image based on the simulated eye segmentation image using an image generation model; wherein the image generation model is constructed based on the first real eye segmentation image and the first real eye image; determine a first data pair based on the simulated eye image and the first gaze direction; wherein the first data pair is used to construct a gaze tracking model.

[0136] Executable instructions for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The executable instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the data generation method provided in any embodiment of this disclosure.

[0138] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units and modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0141] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] According to one or more embodiments of this disclosure, a data generation method is provided, the method comprising:

[0144] Obtain a segmented image of the simulated eye and the first line of sight corresponding to the segmented image of the simulated eye;

[0145] A simulated eye image is generated based on the simulated eye segmentation map using an image generation model; wherein the image generation model is constructed based on a first real eye segmentation map and a first real eye image.

[0146] Based on the simulated eye image and the first gaze direction, a first data pair is determined; wherein, the first data pair is used to construct a gaze tracking model.

[0147] According to one or more embodiments of this disclosure, a data generation method is provided, further comprising:

[0148] In some alternative implementations, obtaining the simulated eye segmentation image includes:

[0149] Obtain an initial simulation segmentation map of the eyeball model; wherein, the initial simulation segmentation map includes the first pupil region;

[0150] Based on the similarity of the first pupil region and the second pupil region in the first real eye segmentation image in a preset dimension parameter, the target scleral region is determined from the scleral region of the first real eye segmentation image.

[0151] Based on the initial simulation segmentation map and the target scleral region, the simulated eye segmentation map is determined.

[0152] According to one or more embodiments of this disclosure, a data generation method is provided, further comprising:

[0153] In some optional implementations, obtaining the initial simulation segmentation map of the eyeball model includes:

[0154] Obtain a simulation system that includes the eyeball model, light source, and camera;

[0155] The simulation system determines the image of the eyeball model in the camera under the light source based on the received simulation parameters, thereby obtaining the initial simulation segmentation map.

[0156] The simulation parameters include the biological parameters of the eyeball model, as well as the device parameters of the light source and the camera; wherein the biological parameters include the first line of sight.

[0157] According to one or more embodiments of this disclosure, a data generation method is provided, further comprising:

[0158] In some optional implementations, the preset dimension parameters include at least one of the following: the center position of the ellipse fitted based on the pupil region, and the angle between the major axis of the ellipse and the preset baseline.

[0159] According to one or more embodiments of this disclosure, a data generation method is provided, further comprising:

[0160] In some optional implementations, determining the simulated eye segmentation map based on the initial simulated segmentation map and the target scleral region includes:

[0161] The synthesis position is determined based on the center position of the second pupil region in the first real eye segmentation image to which the target scleral region belongs;

[0162] Obtain at least one of the following enhancement parameters: translation parameter, scaling parameter, and rotation parameter;

[0163] Based on the synthesis location and enhancement parameters, the target scleral region is synthesized onto the initial simulation segmentation map to obtain the simulated eye segmentation map.

[0164] According to one or more embodiments of this disclosure, a data generation method is provided, further comprising:

[0165] In some optional implementations, the process of constructing the image generation model includes:

[0166] The first stage of image generation model construction is carried out based on the second real eye image;

[0167] Based on the first real eye segmentation map and the first real eye image, the image generation model is constructed in the second stage; wherein the data volume of the second real eye image is greater than the data volume of the first real eye image.

[0168] According to one or more embodiments of this disclosure, a data generation method is provided, further comprising:

[0169] In some optional implementations, a third real eye image based on the guide point is acquired, and the second gaze direction corresponding to the third real eye image is obtained;

[0170] The third real eye image is segmented to obtain a second real eye segmentation image;

[0171] Data augmentation is performed on the second real eye segmentation map to obtain the third real eye segmentation map;

[0172] An enhanced eye image is generated based on the third real eye segmentation map using the image generation model.

[0173] A second data pair is determined based on the third real eye image, the enhanced eye image, and the second gaze direction; wherein the second data pair is used to construct the gaze tracking model.

[0174] According to one or more embodiments of the present disclosure, a data generation apparatus is provided, the apparatus comprising:

[0175] The acquisition module is used to acquire a simulated eye segmentation image and a first line of sight corresponding to the simulated eye segmentation image;

[0176] An image generation module is used to generate a simulated eye image based on the simulated eye segmentation map using an image generation model; wherein the image generation model is constructed based on a first real eye segmentation map and a first real eye image;

[0177] The data pair determination module is used to determine a first data pair based on the simulated eye image and the first gaze direction; wherein the first data pair is used to construct a gaze tracking model.

[0178] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0179] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0180] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A data generation method, characterized in that, include: Obtain a segmented image of the simulated eye and the first line of sight corresponding to the segmented image of the simulated eye; A simulated eye image is generated based on the simulated eye segmentation map using an image generation model; wherein the image generation model is constructed based on a first real eye segmentation map and a first real eye image. Based on the simulated eye image and the first gaze direction, a first data pair is determined; wherein, the first data pair is used to construct a gaze tracking model.

2. The method according to claim 1, characterized in that, The process of obtaining the simulated eye segmentation image includes: Obtain an initial simulation segmentation map of the eyeball model; wherein, the initial simulation segmentation map includes the first pupil region; Based on the similarity of the first pupil region and the second pupil region in the first real eye segmentation image in a preset dimension parameter, the target scleral region is determined from the scleral region of the first real eye segmentation image. Based on the initial simulation segmentation map and the target scleral region, the simulated eye segmentation map is determined.

3. The method according to claim 2, characterized in that, The process of obtaining the initial simulation segmentation map of the eyeball model includes: Obtain a simulation system that includes the eyeball model, light source, and camera; The simulation system determines the image of the eyeball model in the camera under the light source based on the received simulation parameters, thereby obtaining the initial simulation segmentation map. The simulation parameters include the biological parameters of the eyeball model, as well as the device parameters of the light source and the camera; wherein the biological parameters include the first line of sight.

4. The method according to claim 2, characterized in that, The preset dimension parameters include at least one of the following: the center position of the ellipse fitted based on the pupil region, and the angle between the major axis of the ellipse and the preset baseline.

5. The method according to claim 2, characterized in that, The step of determining the simulated eye segmentation map based on the initial simulated segmentation map and the target scleral region includes: The synthesis position is determined based on the center position of the second pupil region in the first real eye segmentation image to which the target scleral region belongs; Obtain at least one of the following enhancement parameters: translation parameter, scaling parameter, and rotation parameter; Based on the synthesis location and enhancement parameters, the target scleral region is synthesized onto the initial simulation segmentation map to obtain the simulated eye segmentation map.

6. The method according to claim 1, characterized in that, The process of constructing the image generation model includes: The first stage of image generation model construction is carried out based on the second real eye image; Based on the first real eye segmentation map and the first real eye image, the image generation model is constructed in the second stage; wherein the data volume of the second real eye image is greater than the data volume of the first real eye image.

7. The method according to claim 1, characterized in that, Also includes: Acquire a third real eye image based on the guide point, and the second gaze direction corresponding to the third real eye image; The third real eye image is segmented to obtain a second real eye segmentation image; Data augmentation is performed on the second real eye segmentation map to obtain the third real eye segmentation map; An enhanced eye image is generated based on the third real eye segmentation map using the image generation model. A second data pair is determined based on the third real eye image, the enhanced eye image, and the second gaze direction; wherein the second data pair is used to construct the gaze tracking model.

8. A data generation apparatus, characterized in that, include: The acquisition module is used to acquire a simulated eye segmentation image and a first line of sight corresponding to the simulated eye segmentation image; An image generation module is used to generate a simulated eye image based on the simulated eye segmentation map using an image generation model; wherein, the image generation model is constructed based on a first real eye segmentation map and a first real eye image; The data pair determination module is used to determine a first data pair based on the simulated eye image and the first gaze direction; wherein the first data pair is used to construct a gaze tracking model.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data generation method as described in any one of claims 1-7.

10. A storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the data generation method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data generation method as described in any one of claims 1-7.

Images