Line of sight estimation method and device with adaptive computing resources

Abstract

The present application discloses a method and device for line of sight estimation with adaptive computing resources. The method includes: obtaining a hardware resource index of a device, the hardware resource index is used to characterize the hardware resource usage of the device; adjusting the number of calibration reference images in a calibration reference image set according to the hardware resource index; and estimating the line of sight direction of the collected human eye image according to the adjusted calibration reference image set. The present application solves the technical problem that when the line of sight direction of the human eye is estimated by using a differential network model at this stage, the optimization of the line of sight direction reasoning process is not considered, which affects the user experience.

Description

Technical Field

[0001] The present application relates to the field of gaze tracking technology, and in particular to a gaze estimation method and device with adaptive computing resources. Background Art

[0002] Due to the many differences in the structure of the human eyeball, the angles between the visual axis and the optical axis of different people's eyes are different. In order to obtain high-precision line of sight estimation, calibration processing is generally required to correct individual differences. Using differential network model technology is a method to achieve calibration-based line of sight estimation. When using a differential network model to estimate the line of sight direction of the human eye, a reference calibration image is required. Generally speaking, the more calibration reference images are used, the better the estimation accuracy of the line of sight direction, but the marginal improvement is getting smaller and smaller, and it takes more time. Once the line of sight direction estimation process occupies more hardware resources, it will further affect the user experience.

[0003] At present, when using differential network models to estimate the gaze direction of the human eye, the optimization of the gaze direction reasoning process is not taken into account, which affects the user experience. No effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for estimating the line of sight with adaptive computing resources, so as to at least solve the technical problem that when a differential network model is currently used to estimate the line of sight direction of the human eye, the optimization of the line of sight direction reasoning process is not taken into account, thus affecting the user experience.

[0005] According to one aspect of an embodiment of the present application, a line of sight estimation method with computing resource adaptability is provided, including: obtaining a hardware resource indicator of a device, the hardware resource indicator being used to characterize the hardware resource usage of the device; adjusting the number of calibration reference images in a calibration reference image set according to the hardware resource indicator; and estimating the line of sight direction of a captured human eye image according to the adjusted calibration reference image set.

[0006] Optionally, before adjusting the number of calibration reference images in the calibration reference image set according to the hardware resource indicator, the above method also includes: obtaining an estimation accuracy indicator of the line of sight, and determining the minimum number of calibration reference images in the calibration reference image set according to the estimation accuracy indicator.

[0007] Optionally, the hardware resource indicator includes at least one of the following: a processor resource occupancy rate of the device, and a remaining battery power of the device.

[0008] Optionally, the number of calibration reference images in the calibration reference image set is adjusted based on hardware resource indicators, including at least one of the following: if the processor resource occupancy of the device is higher than a first threshold, reducing the current number of calibration reference images until the current number of calibration reference images is equal to a minimum number; if the battery power of the device is lower than a second threshold, reducing the current number of calibration reference images until the current number of calibration reference images is equal to the minimum number.

[0009] Optionally, the above method further includes: adjusting the estimation accuracy index of the line of sight according to the hardware resource index.

[0010] Optionally, adjusting the line of sight estimation accuracy indicator according to the hardware resource indicator includes: if the processor resource occupancy rate of the device is higher than a third threshold, reducing the line of sight estimation accuracy indicator; and / or

[0011] If the battery level of the device is lower than a fourth threshold, the line of sight estimation accuracy indicator is reduced.

[0012] Optionally, the line of sight estimation accuracy indicator includes: a line of sight estimation accuracy quality indicator expected by a user.

[0013] Optionally, the gaze direction of the collected human eye images is estimated based on the adjusted calibration reference image set, including: performing regression calculation on the adjusted calibration reference image set and the collected human eye images based on a preset differential network model to obtain an offset in the gaze direction; superimposing the offset on the true value of the calibration reference image to obtain a gaze direction estimate of the collected human eye images; accumulating the gaze direction estimates; and averaging the accumulated gaze direction estimates to obtain a final gaze direction estimate.

[0014] According to another aspect of an embodiment of the present application, another line of sight estimation method with computing resource adaptability is provided, including: obtaining a hardware resource indicator of a device and a line of sight estimation accuracy indicator, the hardware resource indicator being used to characterize hardware resource usage of the device; determining a minimum number of calibration reference images in a calibration reference image set based on the line of sight estimation accuracy indicator; adjusting the number of calibration reference images in the calibration reference image set to a target number based on the hardware resource indicator, wherein the target number is greater than or equal to the minimum number; and estimating the line of sight direction of a captured human eye image based on the adjusted calibration reference image set.

[0015] According to another aspect of an embodiment of the present application, a line of sight estimation device with computing resource adaptability is also provided, including: an acquisition module, used to acquire hardware resource indicators of a device, the hardware resource indicators are used to characterize the hardware resource usage of the device; an adjustment module, used to adjust the number of calibration reference images in a calibration reference image set according to the hardware resource indicators; and a processing module, used to estimate the line of sight direction of a collected human eye image based on the adjusted calibration reference image set.

[0016] According to another aspect of the embodiments of the present application, a storage medium is further provided, the storage medium including a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above sight line estimation method with computing resource adaptation.

[0017] According to another aspect of the embodiments of the present application, a processor is further provided, and the processor is used to run a program, wherein the program executes the above sight line estimation method with computing resource adaptation when running.

[0018] In an embodiment of the present application, a hardware resource indicator of a device is obtained, and the hardware resource indicator is used to characterize the hardware resource usage of the device; the number of calibration reference images in a calibration reference image set is adjusted according to the hardware resource indicator; and the line of sight direction of the collected human eye images is estimated according to the adjusted calibration reference image set. By adjusting the calibration image set involved in the line of sight direction estimation according to the hardware resource usage of the device, the purpose of reasonably controlling the line of sight direction estimation process to occupy the hardware resources of the device while ensuring that the line of sight direction estimation achieves the expected accuracy is achieved, thereby achieving the technical effect of optimizing the line of sight direction reasoning process and improving the user experience, thereby solving the technical problem that at this stage, when a differential network model is used to estimate the line of sight direction of the human eye, the optimization of the line of sight direction reasoning process is not taken into account, thereby affecting the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a flow chart of a method for line of sight estimation with adaptive computing resources according to an embodiment of the present application;

[0021] Figure 2 is a schematic diagram showing how the line of sight direction estimation error varies with the number of calibration reference images;

[0022] Figure 3 is a flow chart of a method for estimating a sight direction with adaptive computing resources according to an embodiment of the present application;

[0023] Figure 4 is a flow chart of another method for line of sight estimation with adaptive computing resources according to an embodiment of the present application;

[0024] Figure 5 It is a structural diagram of a sight line estimation device with adaptive computing resources according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] In the related art, when a differential network model is used to estimate the gaze direction of the human eye, the optimization of the gaze direction reasoning process is not taken into consideration, and the set of calibration reference images involved in the gaze direction reasoning process is not dynamically adjusted according to the computing resources available to the device.

[0028] To solve the above problems, the present application adjusts the number of calibration reference images in the calibration reference image set participating in the gaze direction reasoning process by detecting the hardware resource load of the device and the user's expected gaze direction estimation accuracy. On the premise of ensuring that the gaze direction estimation achieves the expected accuracy, the gaze direction estimation process reasonably controls the hardware resources occupied by the device.

[0029] Based on the above principles, according to an embodiment of the present application, a method embodiment of a line of sight estimation method with adaptive computing resources is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 1 is a flow chart of a method for line of sight estimation with adaptive computing resources according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0031] Step S102: Acquire a hardware resource indicator of the device, where the hardware resource indicator is used to characterize the hardware resource usage of the device.

[0032] In some embodiments of the present application, an algorithm for estimating the line of sight direction runs on a processor of a device, wherein the device may be a terminal device, such as a mobile phone terminal, or a server device.

[0033] Step S104 , adjusting the number of calibration reference images in the calibration reference image set according to the hardware resource indicator.

[0034] According to an optional embodiment of the present application, in the process of estimating the gaze direction of the real-time collected human eye image, a large number of calibration reference images are required to participate in the calculation. Generally speaking, the more calibration reference images there are, the higher the accuracy of the accumulated gaze direction estimation results will be, but the process will also consume more computing resources. The computing resources consumed by the estimation and reasoning process of the gaze direction are proportional to the size of the calibration image set. When the device is a mobile terminal device, the reasoning process of the gaze direction occupies the computing resources of the device and also consumes energy.

[0035] Figure 2 It is a schematic diagram showing how the line of sight estimation error changes with the number of calibration reference images. Figure 2 As shown in the figure, as the number of calibration reference images increases, the line of sight direction estimation error gradually decreases. However, in actual applications, as the scale of calibration reference images continues to increase, the marginal improvement in line of sight direction accuracy gradually decreases. In other words, although the line of sight direction estimation accuracy will be improved by increasing the number of calibration reference images, the marginal improvement is getting smaller and smaller. At the same time, as the number of calibration reference images increases, the time spent on the line of sight direction estimation process will also increase. Once the line of sight direction estimation process occupies more hardware resources, it will further affect the user experience of the device.

[0036] Therefore, step S104 is executed to adjust the number of calibration reference images by detecting the hardware resource indicators of the device. Under the premise of ensuring the accuracy of line of sight direction estimation, the number of calibration reference images is adjusted to improve the user experience.

[0037] Step S106, estimating the sight direction of the collected human eye images according to the adjusted calibration reference image set.

[0038] According to an optional embodiment of the present application, step S106 is executed to regress the compensation amount of the sight direction between the real-time collected consumer eye image and the calibration image through a differential network model, and the compensation amount is superimposed on the true value of the calibration image to obtain the real-time collected consumer eye sight direction.

[0039] Through the above steps, by dynamically adjusting the set of calibration images involved in the line of sight direction estimation according to the hardware resource usage of the device, the purpose of reasonably occupying the hardware resources of the device while ensuring that the line of sight direction estimation reaches the expected accuracy is achieved, thereby achieving the technical effect of optimizing the line of sight direction reasoning process and improving the user experience.

[0040] In some embodiments of the present application, before executing step S104, the following steps need to be performed: obtaining an estimation accuracy index of the line of sight, and determining the minimum number of calibration reference images in the calibration reference image set based on the estimation accuracy index.

[0041] The line of sight estimation accuracy index is the line of sight direction estimation accuracy quality index expected by the user. For example, if the line of sight direction estimation accuracy quality index expected by the user is A, to achieve this index, at least M calibration reference images are required to participate in the calculation, and there is a corresponding relationship between the M calibration reference images and the estimation accuracy index A. Optionally, a mapping relationship between different estimation accuracy indicators and the minimum number of calibration reference images required to achieve the corresponding estimation accuracy indicator is stored on the device. Therefore, before executing step S104, the line of sight direction estimation accuracy index expected by the user is first obtained, and then the minimum number of calibration reference images corresponding to the estimation accuracy indicator is determined, and this minimum number is used as a reference for adjusting the number of calibration reference images.

[0042] According to an optional embodiment of the present application, the hardware resource indicator includes at least one of the following: a processor resource occupancy rate of the device and a remaining battery power of the device.

[0043] According to an optional embodiment of the present application, the above-mentioned processor includes but is not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing device (DSP), an embedded neural network processor (NPU) and a field programmable gate array (FPGA).

[0044] Graphics Processing Unit (GPU), also known as visual processor or display chip, is a microprocessor that is specially used for image processing in personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.). The function of the image processing unit (GPU) is to convert and drive the display information required by the computer system, provide scanning signals to the display, and control the correct display of the display. It is an important component that connects the display to the motherboard of the personal computer.

[0045] Since the process of line of sight estimation requires a large number of calibration reference images and collected human eye images to participate in the calculation, it involves the processing of images and requires GPU computing resources, so it is necessary to detect the current GPU resource usage of the device; at the same time, on mobile devices, the line of sight estimation process consumes energy while using computing resources, so it is also necessary to detect the remaining battery power of the mobile device. In addition, the CPU resource usage of the device can also be detected.

[0046] In some embodiments of the present application, step S104 is implemented by the following method: if the processor resource occupancy rate of the device is higher than the first threshold, the current number of calibration reference images is reduced until the current number of calibration reference images is equal to the minimum number; if the processor resource occupancy rate is detected to be higher than a preset value, such as the GPU resource occupancy rate is detected to be 80%, it means that the line of sight estimation process has occupied more GPU resources at this time. If the user uses the GPU to process some other tasks at this time, it will inevitably lead to slow calculation speed and poor experience. To solve this problem, the number of calibration reference images involved in the calculation during the line of sight estimation process is reduced to free up more GPU computing resources for processing other tasks.

[0047] Optionally, the number of calibration reference images can be reduced as needed, but the premise is that the number of calibration reference images cannot be less than the minimum number of calibration reference images corresponding to the estimation accuracy index. When the number of calibration reference images after adjustment is equal to the minimum number of calibration reference images corresponding to the estimation accuracy index, the optimal calibration reference image set is obtained.

[0048] If the battery power of the device is lower than the second threshold, the current number of calibration reference images is reduced until the current number of calibration reference images is equal to the minimum number. The line of sight accuracy estimation process consumes energy while occupying computing resources. The more calibration reference images involved in the calculation, the more energy is consumed. When it is detected that the remaining battery power is lower than a preset value, it is necessary to appropriately reduce the number of calibration reference images involved in the calculation to save battery power. Similarly, the number of calibration reference images can be reduced as needed, but the premise is to ensure that the number of calibration reference images cannot be lower than the minimum number of calibration reference images corresponding to the estimation accuracy index.

[0049] In some embodiments of the present application, the above method may also adjust the estimation accuracy index of the line of sight according to the hardware resource index.

[0050] According to an optional embodiment of the present application, adjusting the estimation accuracy index of the line of sight according to the hardware resource index includes: if the processor resource occupancy rate of the device is higher than a third threshold, reducing the estimation accuracy index of the line of sight; and / or

[0051] If the battery level of the device is lower than a fourth threshold, the line of sight estimation accuracy indicator is reduced.

[0052] Hardware resource indicators include processor resource occupancy rate and the remaining battery power of the device. The processor resource occupancy rate and the remaining battery power will also affect the estimation accuracy of the line of sight. When the processor resource occupancy rate is too high or the remaining battery power is too low, the estimation accuracy of the line of sight will be reduced. That is to say, when the processor resource occupancy rate reaches a certain preset value or the battery power is reduced to a certain preset value, the line of sight direction estimation process cannot achieve the line of sight estimation accuracy indicator expected by the user in advance. At this time, it is necessary to appropriately adjust the line of sight estimation accuracy indicator according to the detected hardware resource indicators to improve the user's experience when estimating the line of sight direction. For example, when the GPU resource occupancy rate of the graphics processor reaches 70%, the line of sight estimation accuracy indicator is reduced; for another example, when the remaining battery power is less than 20%, the line of sight estimation accuracy indicator also needs to be reduced. The line of sight estimation accuracy indicators include: the line of sight estimation accuracy quality indicators expected by the user.

[0053] Through the above steps, the calibration image set is adjusted according to the available processor resources or battery power of the device to achieve a balance between the amount of computation required for the gaze direction inference estimation process and the gaze direction estimation accuracy.

[0054] In some embodiments of the present application, the gaze direction of the collected human eye images is estimated based on the adjusted calibration reference image set, including: performing regression calculation on the adjusted calibration reference image set and the collected human eye images based on a preset differential network model to obtain an offset in the gaze direction; superimposing the offset on the true value of the calibration reference image to obtain a gaze direction estimate of the collected human eye images; accumulating the gaze direction estimates; and averaging the accumulated gaze direction estimates to obtain a final gaze direction estimate.

[0055] The basic idea of ​​the differential network model is to classify data streams according to predetermined rules. Since the angle between the visual axis and the optical axis of each person's eyes is different, the general deep learning model cannot distinguish individual differences well when estimating the direction of the human eye's line of sight. The use of the differential network model to estimate the direction of the human eye's line of sight can make up for this shortcoming. The optical axis, also known as the eye axis, refers to a hypothetical line from the center of the cornea to the optic nerve and the fovea of ​​the retina. The visual axis is the line from the gaze point through the node to the fovea of ​​the retina.

[0056] The process of estimating the sight direction using the differential network model mainly includes two stages, namely the differential model training stage and the sight direction inference estimation stage. The sight direction inference estimation formula is as follows:

[0057] Among them, d p (I, F) represents the estimation result of the difference network between the estimated eye image and the calibration reference image, g gt (F) represents the ground truth result of the calibration reference image, D c Identify the set of calibration reference images, g sm (I) Identifies the accumulated estimate of the line of sight for the eye image to be estimated.

[0058] Figure 3 is a flow chart of a method for estimating the sight direction with adaptive computing resources according to an embodiment of the present application, such as Figure 3 As shown, the method comprises the following steps:

[0059] Step S302, retrieving hardware resource indicators;

[0060] Hardware resource indicators include but are not limited to parameters such as processor resource usage and battery remaining power.

[0061] According to an optional embodiment of the present application, the above-mentioned processor includes but is not limited to a central processing unit CPU, a graphics processing unit GPU, a digital signal processing device DSP, an embedded neural network processor NPU and a field programmable gate array FPGA.

[0062] Step S304, passing parameters such as processor resource occupancy rate and battery remaining power, and calling the calibration reference image management module;

[0063] The operation of the calibration reference image management module depends on the resource load of the system and the user's expected line of sight estimation accuracy quality. The module dynamically returns the calibration reference images involved in the calculation during operation. A possible function definition used in the calibration reference image management module is f(usage_battery, usage_gpu, quality), which selects the calibration reference image for the line of sight inference estimation process from the entire set of calibration reference images based on the hardware GPU resource occupancy (usage_gpu) and the battery available resources (usage_battery), and the user's expected line of sight estimation accuracy quality.

[0064] Step S306, extracting a calibration reference image and removing the extracted calibration reference image from the calibration reference image set; extracting a calibration reference image for performing gaze direction estimation calculation from the calibration reference image set, and then removing the extracted calibration reference image from the calibration reference image set.

[0065] Step S308, providing the real-time collected consumer eye image, the extracted calibration reference image and the corresponding posture information to the differential network model for regression calculation to obtain the offset of the sight direction;

[0066] Step S310, superimposing the offset obtained in step S308 on the true value of the calibration reference image, obtaining a gaze direction estimate for the real-time captured consumer eye image, and updating the accumulated gaze direction estimate; in machine learning, the true value represents the real effective value of the image or the labeling of the image.

[0067] Step S312, determining whether there are other calibration reference images in the calibration reference image set, and if the determination result is yes, repeating steps S306 to S310; if the determination result is no, executing step S314;

[0068] Step S314, taking the average value of the accumulated gaze direction estimation to obtain the final result of the gaze direction estimation.

[0069] Steps S302 to S314 provide a method for estimating the sight line of a human eye image collected in real time using a differential network model. It should be noted that: Figure 3 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0070] Figure 4 is a flowchart of another method for line of sight estimation with adaptive computing resources according to an embodiment of the present application. Figure 4 As shown, the method comprises the following steps:

[0071] Step S402, obtaining a hardware resource indicator of the device and a line of sight estimation accuracy indicator, where the hardware resource indicator is used to characterize the hardware resource usage of the device.

[0072] Step S404: determining the minimum number of calibration reference images in the calibration reference image set according to the line of sight estimation accuracy index.

[0073] Step S406 , adjusting the number of calibration reference images in the calibration reference image set to a target number according to the hardware resource indicator, wherein the target number is greater than or equal to a minimum number.

[0074] Step S408: Estimating the sight direction of the collected human eye images according to the adjusted calibration reference image set.

[0075] It should be noted that Figure 4 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0076] Figure 5 is a structural diagram of a sight line estimation device with adaptive computing resources according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0077] The acquisition module 50 is used to acquire the hardware resource index of the device, where the hardware resource index is used to characterize the hardware resource usage of the device.

[0078] In some embodiments of the present application, an algorithm for estimating the line of sight direction runs on a processor of a device, where the device may be a terminal device, such as a mobile phone terminal, or a server device.

[0079] The adjustment module 52 is configured to adjust the number of calibration reference images in the calibration reference image set according to the hardware resource indicator.

[0080] In the process of estimating the gaze direction of real-time human eye images, a large number of calibration reference images are required to participate in the calculation. Generally speaking, the more calibration reference images there are, the higher the accuracy of the accumulated gaze direction estimation results will be, but the process will also consume more computing resources. The computing resources consumed by the estimation and reasoning process of the gaze direction are proportional to the size of the calibration image set. When the device is a mobile terminal device, the reasoning process of the gaze direction occupies the computing resources of the device and also consumes energy.

[0081] As the number of calibration reference images increases, the error in gaze direction estimation gradually decreases. However, in actual applications, as the scale of calibration reference images continues to increase, the marginal improvement in gaze direction accuracy gradually decreases. In other words, although the accuracy of gaze direction estimation will be improved by increasing the number of calibration reference images, the marginal improvement is getting smaller and smaller. At the same time, as the number of calibration reference images increases, the time spent on gaze direction estimation will also increase. Once the gaze direction estimation process occupies more hardware resources, it will further affect the user experience of the device.

[0082] Therefore, the number of calibration reference images is adjusted by detecting the hardware resource indicators of the device. Under the premise of ensuring the accuracy of line of sight direction estimation, the number of calibration reference images is adjusted to improve the user experience.

[0083] The processing module 54 is used to estimate the sight direction of the collected human eye image according to the adjusted calibration reference image set.

[0084] In the embodiment of the present application, the compensation amount of the sight direction between the real-time collected consumer eye picture and the calibration picture is regressed through the differential network model, and the compensation amount is superimposed on the true value of the calibration image to obtain the real-time collected consumer eye sight direction. In the machine learning model, the true value refers to the label value or true value of the calibration image used for training.

[0085] According to another aspect of the embodiments of the present application, a storage medium is further provided, the storage medium including a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above line of sight estimation method.

[0086] The storage medium is used to store a program that performs the following functions: obtaining the hardware resource index of the device, which is used to characterize the hardware resource usage of the device; adjusting the number of calibration reference images in the calibration reference image set according to the hardware resource index; and estimating the line of sight direction of the collected human eye image according to the adjusted calibration reference image set. Or

[0087] Obtain the hardware resource index of the device and the estimation accuracy index of the line of sight, where the hardware resource index is used to characterize the hardware resource usage of the device; determine the minimum number of calibration reference images in the calibration reference image set based on the estimation accuracy index of the line of sight; adjust the number of calibration reference images in the calibration reference image set to a target number based on the hardware resource index, wherein the target number is greater than or equal to the minimum number; and estimate the line of sight direction of the collected human eye image based on the adjusted calibration reference image set.

[0088] According to another aspect of the embodiments of the present application, a processor is further provided, and the processor is used to run a program, wherein the above line of sight estimation method is executed when the program is run.

[0089] The processor is used to run the following program: obtain the hardware resource index of the device, which is used to characterize the hardware resource usage of the device; adjust the number of calibration reference images in the calibration reference image set according to the hardware resource index; and estimate the line of sight direction of the collected human eye image according to the adjusted calibration reference image set. Or

[0090] Obtain the hardware resource index of the device and the estimation accuracy index of the line of sight, where the hardware resource index is used to characterize the hardware resource usage of the device; determine the minimum number of calibration reference images in the calibration reference image set according to the estimation accuracy index of the line of sight; adjust the number of calibration reference images in the calibration reference image set to a target number according to the hardware resource index, wherein the target number is greater than or equal to the minimum number; and estimate the line of sight direction of the collected human eye image according to the adjusted calibration reference image set.

[0091] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0092] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0095] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0097] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for line of sight estimation with adaptive computing resources, characterized in that: include: Obtaining a hardware resource indicator of a device, wherein the hardware resource indicator is used to characterize the hardware resource usage of the device; adjusting the number of calibration reference images in the calibration reference image set according to the hardware resource indicator; Estimate the gaze direction of the collected human eye images based on the adjusted calibration reference image set: According to a preset differential network model, a regression calculation is performed on the adjusted calibration reference image set and the collected human eye image to obtain an offset of the line of sight direction; the offset is superimposed on the true value of the calibration reference image to obtain an estimate of the line of sight direction of the collected human eye image; the line of sight direction estimates are accumulated; and the average of the accumulated line of sight direction estimates is taken to obtain a final line of sight direction estimate; Before adjusting the number of calibration reference images in the calibration reference image set according to the hardware resource indicator, obtain a line of sight estimation accuracy indicator, determine the minimum number of calibration reference images in the calibration reference image set according to the line of sight estimation accuracy indicator, and use the minimum number as a benchmark when adjusting the number of calibration reference images, wherein the line of sight estimation accuracy indicator is a line of sight direction estimation accuracy quality indicator expected by a user, and the minimum number is the minimum number of calibration reference images required to achieve the corresponding line of sight estimation accuracy.

2. The method according to claim 1, characterized in that The hardware resource indicator includes at least one of the following: The processor resource occupancy rate of the device and the remaining battery power of the device.

3. The method according to claim 1 or 2, characterized in that: Adjusting the number of calibration reference images in the calibration reference image set according to the hardware resource indicator includes at least one of the following: If the processor resource occupancy rate of the device is higher than a first threshold, reducing the current number of the calibration reference images until the current number of the calibration reference images is equal to the minimum number; If the battery level of the device is lower than a second threshold, the current number of the calibration reference images is reduced until the current number of the calibration reference images is equal to the minimum number.

4. The method according to claim 2, characterized in that: The method further comprises: The estimation accuracy index of the line of sight is adjusted according to the hardware resource index.

5. The method according to claim 4, characterized in that Adjusting the estimation accuracy index of the line of sight according to the hardware resource index includes: If the processor resource occupancy rate of the device is higher than a third threshold, lowering the estimation accuracy index of the line of sight; and / or If the battery power of the device is lower than a fourth threshold, the estimation accuracy index of the line of sight is reduced.

6. A method for line of sight estimation with adaptive computing resources, characterized in that: include: Acquire a hardware resource indicator of a device and a line of sight estimation accuracy indicator, wherein the hardware resource indicator is used to characterize the hardware resource usage of the device; Determining a minimum number of calibration reference images in a calibration reference image set according to the line of sight estimation accuracy index, wherein the line of sight estimation accuracy index is a line of sight direction estimation accuracy quality index expected by a user, and the minimum number is a minimum number of calibration reference images required to achieve the corresponding line of sight estimation accuracy; adjusting the number of calibration reference images in the calibration reference image set to a target number according to the hardware resource indicator, wherein the target number is greater than or equal to the minimum number; The gaze direction of the collected human eye image is estimated based on the adjusted calibration reference image set: Performing regression calculation on the adjusted calibration reference image set and the collected human eye image according to a preset differential network model to obtain an offset in the line of sight direction; Superimposing the offset onto the true value of the calibration reference image to obtain an estimation of the sight direction of the collected human eye image; accumulating the gaze direction estimates; The accumulated line of sight direction estimates are averaged to obtain a final line of sight direction estimate.

7. A sight line estimation device with adaptive computing resources, characterized in that: include: An acquisition module, used to acquire a hardware resource indicator of a device, wherein the hardware resource indicator is used to characterize the hardware resource usage of the device; An adjustment module, configured to adjust the number of calibration reference images in the calibration reference image set according to the hardware resource indicator; A processing module, used for estimating the sight direction of the collected human eye image based on the adjusted calibration reference image set; According to a preset differential network model, regression calculation is performed on the adjusted calibration reference image set and the collected human eye image to obtain an offset in the line of sight direction; the offset is superimposed on the true value of the calibration reference image to obtain a line of sight direction estimate of the collected human eye image; the line of sight direction estimates are accumulated; the accumulated line of sight direction estimates are averaged to obtain a final line of sight direction estimate; before adjusting the number of calibration reference images in the calibration reference image set according to the hardware resource indicator, an estimation accuracy indicator of the line of sight is obtained, and a minimum number of calibration reference images in the calibration reference image set is determined according to the estimation accuracy indicator of the line of sight, and the minimum number is used as a benchmark when adjusting the number of calibration reference images, wherein the estimation accuracy indicator of the line of sight is a line of sight direction estimation accuracy quality indicator expected by a user, and the minimum number is the minimum number of calibration reference images required to achieve the corresponding line of sight estimation accuracy.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute the line of sight estimation method with computing resource adaptation described in any one of claims 1 to 5 or the line of sight estimation method with computing resource adaptation described in claim 6.

9. A processor, characterized in that: The processor is used to run a program, wherein when the program is run, the line of sight estimation method with computing resource adaptation described in any one of claims 1 to 5 or the line of sight estimation method with computing resource adaptation described in claim 6 is executed.

Images