Area determination method, device, equipment, storage medium and program product

Abstract

The present application relates to a region determination method, device, computer equipment, storage medium and computer program product, which can be used in the field of financial technology or other related fields, including: obtaining eye movement data; inputting the eye movement data into a plurality of pre-constructed eye movement data estimation models, and obtaining a plurality of estimation results of each region for the plurality of eye movement data estimation models through the plurality of eye movement data estimation models; using the estimation results corresponding to the eye movement data estimation models corresponding to each region as the target estimation results; based on the target estimation results, screening out candidate fixation regions from each region; based on the plurality of estimation results corresponding to the candidate fixation regions, constructing a fixation region recognition framework; based on the fixation region recognition framework, determining the target fixation region from the candidate fixation regions. The present method can accurately determine the target fixation region.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a region determination method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of artificial intelligence technology, the automatic confirmation technology of smart terminal interface areas has emerged. This technology collects user voice and confirms the interface area that the user is concerned about based on voice keyword extraction.

[0003] In the above technical solution, if the keywords extracted based on the user's voice are related to multiple areas of the above-mentioned smart terminal interface, the smart terminal will provide multiple areas for the user to choose. Or if a user cannot pronounce or pronounces inaccurately due to special reasons, the smart terminal will not be able to accurately determine the area that the user is paying attention to on the interface. Summary of the invention

[0004] Based on this, it is necessary to provide a region determination method, apparatus, computer equipment, computer-readable storage medium and computer program product that can accurately determine the user's focus area in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for determining an area. The method comprises:

[0006] Acquire eye movement data; the eye movement data is used to characterize changes in the gaze point of a human eye on a gaze interface, the interface comprising a plurality of areas;

[0007] The eye movement data is input into a plurality of pre-constructed eye movement data estimation models, and a plurality of estimation results for each region for the plurality of eye movement data estimation models are obtained through the plurality of eye movement data estimation models; the plurality of eye movement data estimation models correspond to the plurality of regions one by one, and the plurality of estimation results are used to characterize the probability that each region is a fixation region of the human eye;

[0008] Taking the estimation results corresponding to the eye movement data estimation models corresponding to the respective regions as target estimation results; and screening out candidate fixation regions from the respective regions based on the target estimation results;

[0009] Based on a plurality of estimation results corresponding to the candidate gaze areas, a gaze area recognition framework is constructed; and based on the gaze area recognition framework, a target gaze area is determined from the candidate gaze areas.

[0010] In one of the embodiments, based on the target estimation result, selecting candidate gaze areas from the various areas includes: obtaining a final estimation result corresponding to the current area based on the target estimation result; the final estimation result is a first preset value or a second preset value; if the final estimation result of the current area is the first preset value, determining the current area as the candidate gaze area.

[0011] In one of the embodiments, constructing a gaze area recognition framework based on multiple estimated results corresponding to the candidate gaze areas includes: calibrating multiple estimated results corresponding to each candidate gaze area through a calibration method constructed based on a non-parametric method to obtain multiple calibration results corresponding to each candidate gaze area; fusing the multiple calibration results corresponding to each candidate gaze area to obtain a fusion result corresponding to each candidate gaze area; and constructing a gaze area recognition framework based on the fusion results corresponding to each candidate gaze area.

[0012] In one of the embodiments, the target gaze area is determined from the candidate gaze areas based on the gaze area recognition framework, including: obtaining recognition results corresponding to the respective candidate gaze areas based on the gaze area recognition framework; if the value of the recognition result corresponding to the current candidate gaze area is the maximum value among the values ​​of all the recognition results, and the value of the recognition result corresponding to the current candidate gaze area is greater than a preset threshold, then the current candidate gaze area is determined to be the target gaze area.

[0013] In one embodiment, the obtaining of eye movement data includes: obtaining a human eye image captured with a human eye, and obtaining the pupil center coordinates on the human eye image, and obtaining the corneal reflection spot center coordinates on the human eye image; obtaining the line of sight direction of the human eye based on the pupil center coordinates and the corneal reflection spot center coordinates; obtaining the gaze point coordinates of each gaze point of the human eye on the interface based on the line of sight direction; obtaining the dwell time of each gaze point, and the distance between each gaze point and the next gaze point corresponding to each gaze point; using the gaze point coordinates, the dwell time and the distance as the eye movement data.

[0014] In one of the embodiments, obtaining the pupil center coordinates on the human eye image includes: obtaining the pupil area in the human eye image; obtaining the boundary point coordinates of a preset number of boundary points on the boundary of the pupil area; and obtaining the pupil center coordinates based on the boundary point coordinates.

[0015] In one of the embodiments, obtaining the center coordinates of the corneal reflection spot on the human eye image includes: filtering the human eye image to obtain a plurality of corneal reflection spot images; obtaining the center coordinates of a plurality of corneal reflection spot images corresponding to the plurality of corneal reflection spot images; and fusing the center coordinates of the plurality of corneal reflection spot images to obtain the center coordinates of the corneal reflection spot.

[0016] In one of the embodiments, after determining the target gaze area from the candidate gaze areas, the method further includes: in response to a trigger request for the target gaze area, switching the interface to a target interface corresponding to the target gaze area.

[0017] In one of the embodiments, the method further includes: setting the display brightness of the target gaze area on the interface to be greater than the display brightness of a non-target gaze area; the non-target gaze area is an area among the multiple areas other than the target gaze area.

[0018] In a second aspect, the present application further provides a region determination device. The device comprises:

[0019] An eye movement data acquisition module, used to acquire eye movement data; the eye movement data is used to characterize the changes in the gaze point of the human eye on the gaze interface, the interface including multiple areas;

[0020] An estimation result acquisition module is used to input the eye movement data into a plurality of pre-constructed eye movement data estimation models, and obtain a plurality of estimation results of each region for the plurality of eye movement data estimation models through the plurality of eye movement data estimation models; the plurality of eye movement data estimation models correspond to the plurality of regions one by one, and the plurality of estimation results are used to characterize the probability that each region is a fixation region of the human eye;

[0021] A candidate fixation region determination module is used to use the estimation results corresponding to the eye movement data estimation model corresponding to each region as the target estimation result; based on the target estimation result, select the candidate fixation region from each region;

[0022] The target gaze area determination module is used to construct a gaze area recognition framework based on a plurality of estimation results corresponding to the candidate gaze areas; and to determine a target gaze area from the candidate gaze areas based on the gaze area recognition framework.

[0023] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0024] Acquire eye movement data; the eye movement data is used to characterize changes in the gaze point of a human eye on a gaze interface, the interface comprising a plurality of areas;

[0025] The eye movement data is input into a plurality of pre-constructed eye movement data estimation models, and a plurality of estimation results for each region for the plurality of eye movement data estimation models are obtained through the plurality of eye movement data estimation models; the plurality of eye movement data estimation models correspond to the plurality of regions one by one, and the plurality of estimation results are used to characterize the probability that each region is a fixation region of the human eye;

[0026] Taking the estimation results corresponding to the eye movement data estimation models corresponding to the respective regions as target estimation results; and screening out candidate fixation regions from the respective regions based on the target estimation results;

[0027] Based on a plurality of estimation results corresponding to the candidate gaze areas, a gaze area recognition framework is constructed; and based on the gaze area recognition framework, a target gaze area is determined from the candidate gaze areas.

[0028] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0029] Acquire eye movement data; the eye movement data is used to characterize changes in the gaze point of a human eye on a gaze interface, the interface comprising a plurality of areas;

[0030] The eye movement data is input into a plurality of pre-constructed eye movement data estimation models, and a plurality of estimation results for each region for the plurality of eye movement data estimation models are obtained through the plurality of eye movement data estimation models; the plurality of eye movement data estimation models correspond to the plurality of regions one by one, and the plurality of estimation results are used to characterize the probability that each region is a fixation region of the human eye;

[0031] Taking the estimation results corresponding to the eye movement data estimation models corresponding to the respective regions as target estimation results; and screening out candidate fixation regions from the respective regions based on the target estimation results;

[0032] Based on a plurality of estimation results corresponding to the candidate gaze areas, a gaze area recognition framework is constructed; and based on the gaze area recognition framework, a target gaze area is determined from the candidate gaze areas.

[0033] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0034] Acquire eye movement data; the eye movement data is used to characterize changes in the gaze point of a human eye on a gaze interface, the interface comprising a plurality of areas;

[0035] The eye movement data is input into a plurality of pre-constructed eye movement data estimation models, and a plurality of estimation results for each region for the plurality of eye movement data estimation models are obtained through the plurality of eye movement data estimation models; the plurality of eye movement data estimation models correspond to the plurality of regions one by one, and the plurality of estimation results are used to characterize the probability that each region is a fixation region of the human eye;

[0036] Taking the estimation results corresponding to the eye movement data estimation models corresponding to the respective regions as target estimation results; and screening out candidate fixation regions from the respective regions based on the target estimation results;

[0037] Based on a plurality of estimation results corresponding to the candidate gaze areas, a gaze area recognition framework is constructed; and based on the gaze area recognition framework, a target gaze area is determined from the candidate gaze areas.

[0038] The above-mentioned area determination method, device, computer equipment, storage medium and computer program product obtain eye movement data; the eye movement data is used to characterize the changes in the gaze point of the human eye on the gaze interface, and the interface includes multiple areas; the eye movement data is input into multiple pre-constructed eye movement data estimation models, and multiple estimation results of each area for multiple eye movement data estimation models are obtained through multiple eye movement data estimation models; multiple eye movement data estimation models correspond to multiple areas one by one, and multiple estimation results are used to characterize the probability that each area is a gaze area of ​​the human eye; the estimation results corresponding to the eye movement data estimation model corresponding to each area are used as target estimation results; based on the target estimation results, candidate gaze areas are screened out from each area; based on multiple estimation results corresponding to the candidate gaze areas, a gaze area recognition framework is constructed; based on the gaze area recognition framework, the target gaze area is determined from the candidate gaze areas. The present application extracts eye movement data, then inputs the eye movement data into a pre-constructed eye movement data estimation model, screens out candidate gaze areas, and finally uses the gaze area recognition framework to accurately determine the target gaze area. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a flow chart of a region determination method in one embodiment;

[0040] Figure 2 A schematic diagram of a process for screening candidate fixation areas in one embodiment;

[0041] Figure 3 A schematic diagram of a process for constructing a gaze region recognition framework in one embodiment;

[0042] Figure 4 A schematic diagram of a process for determining a target gaze area in one embodiment;

[0043] Figure 5 A flowchart of user operation of a smart terminal in one embodiment;

[0044] Figure 6-1 A system interface effect diagram of a smart terminal in an embodiment;

[0045] Figure 6-2 This is a system interface effect diagram of a smart terminal in another embodiment;

[0046] Figure 7 is a structural block diagram of a region determination device in one embodiment;

[0047] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] It should be noted that the terms "first\second" involved in the embodiments of the present invention are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0050] In one embodiment, Figure 1 As shown, a method for determining an area is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0051] Step S101, obtaining eye movement data; the eye movement data is used to characterize the changes in the gaze point of the human eye on the gaze interface, and the interface includes multiple areas.

[0052] Among them, the eye movement data is the data generated by the change of the human eye's line of sight, for example, the eye movement data can be the specific direction of the human eye's line of sight, the time the human eye's line of sight stays in a certain direction, the angle of change of the human eye's line of sight from one direction to another, etc. The interface is a pre-set plane for the human eye to gaze at, and the interface includes multiple pre-set areas. As for the gaze point, it is the intersection of the human eye's line of sight and the above plane, that is, the point where the human eye gazes on the interface.

[0053] Specifically, an infrared camera is used to capture a human eye image to obtain the human eye image, which is then binarized to obtain a result of the binary image processing. The result is filtered using a Gaussian function to remove noise from the human eye image, and the noise-removed image input value is used to pre-construct a human eye gaze direction recognition model to obtain human eye gaze direction information, and based on the gaze direction information, eye movement data is obtained.

[0054] Step S102, input the eye movement data into a plurality of pre-constructed eye movement data estimation models, and obtain a plurality of estimation results of each region for the plurality of eye movement data estimation models through the plurality of eye movement data estimation models; the plurality of eye movement data estimation models correspond one-to-one to the plurality of regions, and the plurality of estimation results are used to characterize the probability that each region is a fixation area of ​​the human eye.

[0055] Among them, the pre-built multiple eye movement data prediction models are pre-built multiple binary classification models, which correspond one-to-one to the areas in the above-mentioned interface. The binary classification model is constructed based on a gradient boosting iterative decision tree. The binary classification model is used to identify whether the corresponding area is the area where the human eye is looking at, and the estimation result is the output value of the binary classification model, which is used to characterize the probability that each of the above-mentioned areas is the area where the human eye is looking at.

[0056] Specifically, the eye movement data is input into the binary classification model corresponding to one of the regions, and the estimated results of the above-mentioned regions can be obtained. The estimated results are the output values ​​of the binary classification model, wherein the estimated result corresponding to the above-mentioned region has the smallest error, and the estimated results corresponding to the remaining regions have relatively large errors. Then repeat this operation, input the eye movement data into the binary classification models corresponding to the remaining regions, and obtain multiple estimated results for each region for multiple eye movement data estimation models. For example, the above interface has four regions A, B, C, and D, then there are four binary classification models A, B, C, and D. Now input the eye movement data into model A, and obtain estimated results A, estimated results B, estimated results C, and estimated results D. Among them, the estimated result A corresponding to region A has a small error, while the estimated results B, estimated results C, and estimated results D have relatively large errors. Then input the eye movement data into models B, C, and D, respectively, and finally obtain a total of 16 estimated results, with 4 estimated results corresponding to 4 models corresponding to each region.

[0057] Step S103: taking the estimation results corresponding to the eye movement data estimation models corresponding to the respective regions as target estimation results; and screening out candidate fixation regions from the respective regions based on the target estimation results.

[0058] Among them, the target estimation result is the estimation result with the smallest error corresponding to each area, that is, the estimation result corresponding to the estimation model corresponding to each area. For example, the target estimation result of the above-mentioned area A is the estimation result A, and the candidate gaze area is the area in the above-mentioned multiple areas where the possibility of human eye gaze is more than 50%.

[0059] Specifically, from a plurality of estimation results corresponding to the current region, an estimation result corresponding to the recognition model corresponding to the current region is selected as a target estimation result. If the target estimation result is greater than a preset threshold, the current region is determined to be a candidate fixation region.

[0060] Step S104, constructing a gaze area recognition framework based on a plurality of estimation results corresponding to the candidate gaze areas; and determining a target gaze area from the candidate gaze areas based on the gaze area recognition framework.

[0061] Among them, the gaze area recognition framework is a recognition framework constructed based on DS evidence theory reasoning, which is used to determine the target gaze area, and the target gaze area is the area in the above interface that the human eye is most likely to gaze at.

[0062] Specifically, multiple estimation results corresponding to the current candidate gaze area are fused to obtain the fusion result of the current candidate gaze area, and the fusion results of the remaining areas are obtained similarly. A gaze area recognition framework is constructed based on the fusion result, and then the gaze area recognition framework outputs the corresponding number of the target gaze area.

[0063] In the above-mentioned area determination method, eye movement data is obtained; the eye movement data is used to characterize the changes in the gaze point of the human eye on the gaze interface, and the interface includes multiple areas; the eye movement data is input into multiple pre-constructed eye movement data estimation models, and multiple estimation results of each area for multiple eye movement data estimation models are obtained through multiple eye movement data estimation models; multiple eye movement data estimation models correspond to multiple areas one by one, and multiple estimation results are used to characterize the probability that each area is a gaze area of ​​the human eye; the estimation results corresponding to the eye movement data estimation models corresponding to each area are used as target estimation results; based on the target estimation results, candidate gaze areas are screened out from each area; based on multiple estimation results corresponding to the candidate gaze areas, a gaze area recognition framework is constructed; based on the gaze area recognition framework, the target gaze area is determined from the candidate gaze areas. The present application extracts eye movement data, and then inputs the eye movement data into a pre-constructed eye movement data estimation model to screen out candidate gaze areas, and finally through the gaze area recognition framework, the target gaze area can be accurately determined.

[0064] In one embodiment, Figure 2 As shown, based on the target estimation result, the candidate fixation area is screened out from each area, including the following steps:

[0065] Step S201, based on the target estimation result, obtain a final estimation result corresponding to the current area; the final estimation result is a first preset value or a second preset value.

[0066] Among them, the final estimated result is two preset results. The preset result to which the target estimated result is closer is the final estimated result, and the first preset value or the second preset value is the above two preset results, for example, the first preset value is 1, and the second preset value is 0.

[0067] Specifically, when the target estimation result is close to 1, the final estimation result corresponding to the target estimation result is 1; when the target estimation result is close to 0, the final estimation result corresponding to the target estimation result is 0;

[0068] Step S202: if the final estimation result of the current area is a first preset value, the current area is determined as a candidate fixation area.

[0069] Specifically, if the final estimation result of the current region is 1, the current region is determined as a candidate fixation region.

[0070] In this embodiment, by converting the target estimation result into the final estimation result, the candidate fixation areas can be accurately screened out based on the final estimation result.

[0071] In one embodiment, Figure 3As shown, based on multiple estimation results corresponding to the candidate fixation areas, a fixation area recognition framework is constructed, including the following steps:

[0072] Step S301 , calibrating a plurality of estimated results corresponding to each candidate fixation area by a calibration method constructed based on a non-parametric method, to obtain a plurality of calibration results corresponding to each candidate fixation area.

[0073] Among them, the calibration method constructed based on the non-parametric method is a calibration result of the output value of a binary classification model, which can be a Histogram Binning non-parametric method, and the calibration result is the estimated result after calibration and optimization through the scheme, and the calibration result is the probability value of the candidate fixation area being the target fixation area.

[0074] Specifically, a non-parametric method Histogram Binning is used to calibrate and optimize multiple estimation results corresponding to each candidate fixation area, so as to obtain multiple calibration results corresponding to each candidate fixation area.

[0075] Step S302: fuse multiple calibration results corresponding to each candidate fixation area to obtain a fusion result corresponding to each candidate fixation area.

[0076] The fusion result is the orthogonal sum of multiple calibration results corresponding to each candidate fixation area based on DS evidence theory reasoning.

[0077] Specifically, the fusion rules of DS evidence theory reasoning are as follows:

[0078]

[0079] in, m j (A i ) are multiple calibration results corresponding to the candidate fixation area A, m(A) is the fusion result of the candidate fixation area A, and the multiple calibration results corresponding to each candidate fixation area are fused by the fusion rule to obtain the fusion result corresponding to each candidate fixation area. For example, assuming that the above area A is the candidate fixation area, the four calibration results A, B, C, and D corresponding to area A are fused to obtain the fusion result corresponding to area A, that is, the final probability that area A is the target fixation area.

[0080] Step S303: constructing a gaze region recognition framework based on the fusion results corresponding to each candidate gaze region.

[0081] Specifically, a set of fusion results corresponding to each candidate fixation area is formed, and the set is the fixation area recognition framework. In other words, the fusion results corresponding to each candidate fixation area are the values ​​corresponding to each event in the fixation area recognition framework. For example, candidate fixation area A is an event in the recognition framework, and the fusion result A corresponding to candidate fixation area A is the value corresponding to the event.

[0082] In this embodiment, by fusing multiple calibration results corresponding to each candidate gaze area, a gaze area recognition framework can be accurately constructed.

[0083] In one embodiment, Figure 4 As shown, the following steps are included: based on a gaze area recognition framework, a target gaze area is determined from candidate gaze areas.

[0084] Step S401: obtaining recognition results corresponding to each candidate gaze area based on a gaze area recognition framework.

[0085] The recognition result is the probability that the candidate fixation area is the target fixation area.

[0086] Specifically, based on the above gaze region recognition framework, the hidden set of the above gaze region recognition framework is obtained, that is, the set of all subsets of the above gaze region recognition framework. For example, the existing candidate gaze regions A, B, and C have corresponding fusion results of a, b, and c, respectively, then the gaze region recognition framework is a set {a, b, c}, and the hidden set of the recognition framework is {a, b, c, {a, b}, {a, c}, {c, d}, {a, b, c}}, then the recognition result corresponding to the candidate gaze region is the sum of the probabilities of all related events of the candidate gaze region in the corresponding hidden set, for example, event A is the candidate gaze region A is the target gaze region, Bel (A) represents the sum of the probabilities of all related events of event A, that is, the total trust of event A, that is, the recognition result corresponding to the candidate gaze region A.

[0087] Step S402: If the value of the recognition result corresponding to the current candidate gaze area is the maximum value of all the recognition results and the value of the recognition result corresponding to the current candidate gaze area is greater than a preset threshold, the current candidate gaze area is determined to be the target gaze area.

[0088] Specifically, the maximum value in the above recognition results is selected. If the maximum value is greater than a preset threshold (for example, 0.5), the candidate gaze area corresponding to the maximum value is the target gaze area. If the maximum value is not greater than the preset threshold, the recognition result is that there is no target gaze area.

[0089] In this embodiment, the target gaze area can be accurately identified through the gaze area identification framework.

[0090] In one embodiment, obtaining eye movement data includes the following steps:

[0091] A human eye image captured with a human eye is obtained, and the coordinates of the center of the pupil on the human eye image and the coordinates of the center of the corneal reflection spot on the human eye image are obtained.

[0092] Among them, the human eye image is a pre-processed image of the human eye, and the pupil center coordinates are the pixel coordinates of the human eye pupil center in the human eye image. As for the corneal reflection spot center coordinates, they are the pixel coordinates of the corneal reflection spot center in the human eye image, where the corneal reflection spot center is the reflection spot of the light source on the cornea of ​​the human eye. Generally, there are multiple reflection spots corresponding to multiple light sources, and the shapes of these spots are generally different.

[0093] Specifically, the human eye is photographed by an infrared camera to obtain a human eye image, which is then binarized and then removed from the human eye image using a Gaussian function. The features of the human eye image are extracted through human eye feature extraction to obtain the pupil center coordinates and the corneal reflection spot center coordinates.

[0094] Based on the pupil center coordinates and the corneal reflected light spot center coordinates, the line of sight direction of the human eye is obtained.

[0095] Among them, the sight direction of the human eye is the gaze direction of the human eye.

[0096] Specifically, based on the coordinates of the center of the pupil and the center of the corneal reflection spot, the initial direction vector of the line of sight of the human eye is obtained. The initial direction vector is obtained in the pixel coordinate system of the human eye image. The initial direction vector is converted into a direction vector in the world coordinate system through the corresponding coordinate conversion function. Based on the direction vector, the line of sight of the human eye is obtained.

[0097] Based on the line of sight direction, the gaze point coordinates of each gaze point of the human eye on the interface are obtained.

[0098] The gaze point is the point where the human eye gazes on the above interface, and the gaze point coordinates are the coordinates of the gaze point on the interface.

[0099] Specifically, the coordinates of the intersection of the straight line where the sight line direction is located and the above interface are used as the coordinates of the gaze point on the interface.

[0100] Obtain the dwell time of each fixation point and the distance between each fixation point and the next fixation point corresponding to each fixation point; use the fixation point coordinates, dwell time and distance as eye movement data.

[0101] The dwell time refers to the time that the human eye stays at a certain fixation point, and the distance refers to the distance between two fixation points when the human eye's sight jumps from one fixation point to the next.

[0102] Specifically, there are many ways to represent eye movement data. Here, the gaze point coordinates, dwell time and distance are selected as eye movement data.

[0103] In this embodiment, the gaze direction of the human eye is obtained through the pupil center coordinates and the corneal reflection spot center coordinates, and the eye movement data can be accurately obtained through the gaze direction.

[0104] In one embodiment, obtaining the pupil center coordinates on the human eye image includes the following steps:

[0105] Get the pupil area in the human eye image.

[0106] The pupil area is the area where the pupil is located in the human eye image, which is an elliptical boundary.

[0107] Specifically, the pupil area is extracted by using a human eye image feature extraction tool.

[0108] Get the boundary point coordinates of a preset number of boundary points on the boundary of the pupil area.

[0109] The preset number of boundary points are 6 pre-set points on the boundary of the pupil ellipse area, and the boundary point coordinates are the pixel coordinates of the boundary points on the human eye image.

[0110] Specifically, the pixel coordinates of the above boundary points are extracted by using a human eye image feature extraction tool.

[0111] Based on the boundary point coordinates, the pupil center coordinates are obtained.

[0112] Specifically, the boundary curve fitting formula of the pupil ellipse area is as follows:

[0113] Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0;

[0114] Substituting the coordinates of the above six points into the above formula, the values ​​of A, B, C, D, and E can be solved, thereby solving the above formula, and then the pupil center coordinates can be obtained through the ellipse center coordinate formula.

[0115] In this embodiment, the pupil center coordinates can be accurately obtained through the boundary curve fitting formula of the pupil ellipse area.

[0116] In one embodiment, obtaining the center coordinates of the corneal reflection spot on the human eye image includes the following steps: filtering the human eye image to obtain multiple corneal reflection spot images; obtaining the center coordinates of multiple corneal reflection spot images corresponding to the multiple corneal reflection spot images; and fusing the center coordinates of the multiple corneal reflection spot images to obtain the center coordinates of the corneal reflection spot.

[0117] Specifically, the human eye image is filtered through the human eye image feature extraction tool to leave only a plurality of corneal reflection spot images, and then the central coordinates of the plurality of corneal reflection spot images are obtained through the human eye image feature extraction tool, and the average of these coordinates is taken to obtain the central coordinates of the corneal reflection spot.

[0118] In this embodiment, by acquiring the center coordinates of multiple corneal reflection spot images, the center coordinates of the corneal reflection spot can be accurately obtained.

[0119] In one embodiment, after the target gaze area is determined from the candidate gaze areas, the method further includes the following steps: in response to a trigger request for the target gaze area, switching the interface to a target interface corresponding to the target gaze area.

[0120] The trigger request is a request to enter the target interface, and the target interface is the interface corresponding to the target gaze area. Specifically, the user clicks the target gaze area in the interface to switch the interface to the target interface corresponding to the target gaze area.

[0121] In this embodiment, by triggering a request for a target gaze area, it is possible to accurately enter a target interface corresponding to the target gaze area.

[0122] In one embodiment, the above method further includes the following steps: setting the display brightness of the target gaze area on the interface to be greater than the display brightness of the non-target gaze area; the non-target gaze area is an area other than the target gaze area among the multiple areas.

[0123] The display brightness is the brightness of each area on the interface, and the non-target gaze area is the area other than the target gaze area among the multiple areas. Specifically, the target gaze area is highlighted so that the display brightness of the target gaze area on the interface is greater than the display brightness of the non-target gaze area.

[0124] In this embodiment, by highlighting the target gaze area, the user can accurately find and click the target gaze area.

[0125] In one embodiment, a method for determining the business module area of ​​a bank intelligent terminal system interface is also provided, which collects the eye movement data of users when performing business operations, performs corresponding preprocessing and feature extraction, and then performs cognitive calculations such as interaction intention reasoning based on the binary classification algorithm progressive gradient regression model, and at the same time, the classification results are optimally selected and displayed as visual instructions on the interactive interface to achieve more accurate prompts for customers when handling business. The above method includes the following steps:

[0126] 1. Eye movement data collection: The method based on pupil center-cornea reflection (PCCR) is used to obtain eye movement data. A camera device with infrared light is built into the self-service machine to shoot the human eye image to obtain the human eye image. Then, the human eye image is binarized to obtain the result of the binarized image processing. The result is filtered by Gaussian function to achieve the purpose of removing the noise of the human eye image. At this time, the eye image is obtained, and the next step is to extract the features of the human eye image, obtain the center of the human eye pupil and the center of the infrared light reflection spot (also known as the Purkin spot), and calculate the PCCR vector composed of the two, calibrate the target position, establish the mapping relationship between the human eye and the interface, and finally calculate by fitting.

[0127] 1.1. Pupil center extraction. The pupil geometry is extracted based on the pupil area and the pupil geometry. The pupil geometry is first roughly located twice, and then the roughly located pupil geometry is processed to accurately locate it.

[0128] The pupil is not a regular circle, but an ellipse, so an ellipse is used for fitting, and the fitting curve is as follows:

[0129] Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0;

[0130] Select six points in the pupil at random, substitute them into the above formula, use the least square method to calculate the coefficients of the ellipse A, B, C, D, E, F, fit the pupil curve, and then use the following formula to get the center of the ellipse:

[0131]

[0132] 1.2. Extraction of the center of the Purchin spot. Extracting the coordinates of the center of the Purchin spot is a key part. The sight direction is determined by the position of the pupil center in the human eye image and the coordinate position of the center of the Purchin spot. It includes two main stages, namely, the preprocessing of the pupil image and the positioning of the center of the Purchin spot. The geometric figure of the Purchin spot is obtained, and then the center of the geometric figure is obtained. The center data of the geometric figure is obtained by the following formula, where x f ,yf That is the center coordinate:

[0133]

[0134]

[0135] 1.3. PCCR vector fitting: PCCR vector is obtained from the pupil center data and Purkinje spot center data calculated above, and then the PCCR vector fitting data is obtained according to the image calibration points of the virtual scene. In the field of eye tracking research, the PCCR vector is usually fitted using the calibration point coordinates, and the calibration is used to obtain the corresponding relationship between the system gaze point coordinates and the actual gaze point coordinates.

[0136] 2. Extract eye movement features. Select three eye movement features, namely, fixation point coordinates, fixation point dwell time, and eye saccades, to analyze eye movement data.

[0137] 3. Calculation of classification results of the eye movement data binary classification model. In the present invention, a gradient boosting iterative decision tree is used to classify the eye movement gaze patterns of users when operating the self-service machine. The classification algorithm model is established as follows:

[0138] 3.1. Initialize the learner:

[0139]

[0140] 3.2. Establish M classification and regression trees:

[0141] Calculate the response value corresponding to the mth tree:

[0142]

[0143] For the leaf node area, substitute the formula to calculate the best fit value:

[0144]

[0145] Based on the best fit value, the strong learner can be updated:

[0146]

[0147] The final expression of the strong learner is obtained:

[0148]

[0149] Its classification model can be expressed as:

[0150]

[0151] The above eye movement features are input into the above binary classification model, and the corresponding expected output is 1 or 2. When the expected output is 1, it means that the user of the self-service machine module does not need to select a result. When the expected output is 2, it means that the user of the self-service machine module needs to select a result.

[0152] 4. Optimal calculation of multiple modules is performed to obtain the module operation set required by the above-mentioned intelligent terminal operation users. The BPA is assigned to the classifier through the recognition framework θ of DS evidence theory reasoning to obtain the final judgment result.

[0153] Input the above eye movement features into the above two-classification model to obtain the decision result Z of the above two-classification model p Z p1 and Z p2 , use Z p As the input of the decision layer, for any test sample Z p , estimate its category probability:

[0154]

[0155] Where f is the output function of the classifier, and the parameters λ and B can be obtained by calculating the log-likelihood function that minimizes the output function f of the training sample and the above two-classification model, and solving the optimization problem of the following two equations to obtain the category probability p i :

[0156]

[0157]

[0158] Where t represents the tth target category, and s represents the sth binary classification model mentioned above;

[0159] According to the upper error bound theorem, the classifier is assigned BPA through the recognition framework θ of DS evidence theory reasoning to obtain the final judgment result:

[0160] m s (θ)=P ε ;

[0161] m s (C t )=P t s (1-P ε );

[0162] 5. Confirm the corresponding area in the visual interactive interface, apply the eye movement data to the banking self-service machine system, design the system function interface for user self-service operation, and use the eye movement data of the user operating the self-service machine to provide the module function prompts that the user may need. Figure 6-1As shown in the figure, the system interface displays multiple business function modules. By obtaining the user's eye movement data and calculating the time when the user does not operate the interface in real time, if the user does not operate the self-service machine for more than 5 seconds, the user's eye movement data collected within these 5 seconds is classified and judged (such as Figure 5 Select the module you need to highlight and enlarge it (as shown in Figure 6-2 shown).

[0163] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiment of the present application also provides a region determination device for implementing the region determination method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more region determination device embodiments provided below can refer to the limitations of the region determination method above, and will not be repeated here.

[0165] In one embodiment, Figure 7 As shown, a region determination device is provided, comprising: an eye movement data acquisition module 701, an estimation result acquisition module 702, a candidate gaze region determination module 703 and a target gaze region determination module 704, wherein:

[0166] The eye movement data acquisition module 701 is used to acquire eye movement data; the eye movement data is used to characterize the changes in the gaze point of the human eye on the gaze interface, and the interface includes multiple areas;

[0167] The estimation result acquisition module 702 is used to input the eye movement data into a plurality of pre-built eye movement data estimation models, and obtain a plurality of estimation results of each region for the plurality of eye movement data estimation models through the plurality of eye movement data estimation models; the plurality of eye movement data estimation models correspond to the plurality of regions one by one, and the plurality of estimation results are used to characterize the probability that each region is a fixation region of the human eye;

[0168] The candidate fixation region determination module 703 is used to use the estimation results corresponding to the eye movement data estimation model corresponding to each region as the target estimation results; based on the target estimation results, the candidate fixation regions are screened out from each region;

[0169] The target gaze area determination module 704 is used to construct a gaze area recognition framework based on multiple estimation results corresponding to the candidate gaze areas; and determine the target gaze area from the candidate gaze areas based on the gaze area recognition framework.

[0170] In one of the embodiments, the candidate gaze area judgment module 703 is further used to obtain a final estimation result corresponding to the current area based on the target estimation result; the final estimation result is a first preset value or a second preset value; if the final estimation result of the current area is the first preset value, the current area is determined to be a candidate gaze area.

[0171] In one of the embodiments, the target gaze area judgment module 704 is further used to calibrate multiple estimated results corresponding to each candidate gaze area through a calibration method constructed based on a non-parametric method to obtain multiple calibration results corresponding to each candidate gaze area; fuse the multiple calibration results corresponding to each candidate gaze area to obtain a fusion result corresponding to each candidate gaze area; and construct a gaze area recognition framework based on the fusion results corresponding to each candidate gaze area.

[0172] In one of the embodiments, the target gaze area judgment module 704 is further used to obtain recognition results corresponding to each candidate gaze area based on the gaze area recognition framework; if the value of the recognition result corresponding to the current candidate gaze area is the maximum value of all the recognition results, and the value of the recognition result corresponding to the current candidate gaze area is greater than a preset threshold, then the current candidate gaze area is determined to be the target gaze area.

[0173] In one embodiment, the eye movement data acquisition module 701 is further used to acquire a human eye image in which a human eye is captured, and acquire the pupil center coordinates on the human eye image, and acquire the corneal reflection spot center coordinates on the human eye image; obtain the line of sight direction of the human eye based on the pupil center coordinates and the corneal reflection spot center coordinates; obtain the gaze point coordinates of each gaze point of the human eye on the interface based on the line of sight direction; acquire the dwell time of each gaze point, and the distance between each gaze point and the next gaze point corresponding to each gaze point; and use the gaze point coordinates, dwell time and distance as eye movement data.

[0174] In one embodiment, the eye movement data acquisition module 701 is further used to acquire the pupil area in the human eye image; acquire the boundary point coordinates of a preset number of boundary points on the boundary of the pupil area; and obtain the pupil center coordinates based on the boundary point coordinates.

[0175] In one embodiment, the eye movement data acquisition module 701 is further used to filter the human eye image to obtain multiple corneal reflection spot images; obtain multiple corneal reflection spot image center coordinates corresponding to the multiple corneal reflection spot images; and fuse the multiple corneal reflection spot image center coordinates to obtain the corneal reflection spot center coordinates.

[0176] In one embodiment, the target gaze area determination module 704 is further configured to switch the interface to a target interface corresponding to the target gaze area in response to a trigger request for the target gaze area.

[0177] In one embodiment, the target gaze area determination module 704 is further used to set the display brightness of the target gaze area on the interface to be greater than the display brightness of the non-target gaze area; the non-target gaze area is an area other than the target gaze area among the multiple areas.

[0178] Each module in the above-mentioned area determination device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0179] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for determining an area is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the computer device housing, or an external keyboard, touch pad or mouse, etc.

[0180] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0181] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0182] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0183] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0185] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0187] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for determining a region, It is characterized in that The method comprises: Acquire eye movement data; the eye movement data is used to characterize changes in the gaze point of a human eye on a gaze interface, the interface comprising a plurality of areas; The eye movement data is input into a plurality of pre-constructed eye movement data estimation models, and a plurality of estimation results for each region for the plurality of eye movement data estimation models are obtained through the plurality of eye movement data estimation models; the plurality of eye movement data estimation models correspond to the plurality of regions one by one, and the plurality of estimation results are used to characterize the probability that each region is a fixation region of the human eye; Taking the estimation results corresponding to the eye movement data estimation models corresponding to the respective regions as target estimation results; and screening out candidate fixation regions from the respective regions based on the target estimation results; Based on the multiple estimation results corresponding to the candidate fixation areas, a fixation area recognition framework is constructed, including: Calibrate multiple estimated results corresponding to each candidate fixation area by a calibration method constructed based on a non-parametric method to obtain multiple calibration results corresponding to each candidate fixation area; Fusing the multiple calibration results corresponding to the candidate fixation areas to obtain fusion results corresponding to the candidate fixation areas; Based on the fusion results corresponding to the candidate fixation regions, constructing the fixation region recognition framework; Based on the gaze area recognition framework, obtaining recognition results corresponding to each of the candidate gaze areas; If the value of the recognition result corresponding to the current candidate gaze area is the maximum value of all the recognition results, and the value of the recognition result corresponding to the current candidate gaze area is greater than a preset threshold, the current candidate gaze area is determined to be the target gaze area.

2. The method according to claim 1, It is characterized in that The step of selecting a candidate fixation area from each area based on the target estimation result includes: Based on the target estimation result, a final estimation result corresponding to the current area is obtained; the final estimation result is a first preset value or a second preset value; If the final estimation result of the current area is the first preset value, the current area is determined as the candidate fixation area.

3. The method according to claim 1, It is characterized in that The acquiring of eye movement data comprises: Acquire a human eye image captured with a human eye, obtain the coordinates of the center of the pupil on the human eye image, and obtain the coordinates of the center of the corneal reflection spot on the human eye image; Based on the pupil center coordinates and the cornea reflection spot center coordinates, obtaining the sight direction of the human eye; Based on the line of sight direction, obtaining the gaze point coordinates of each gaze point of the human eye on the interface; Acquire the dwell time of each gaze point and the distance between each gaze point and the next gaze point corresponding to each gaze point; and use the gaze point coordinates, the dwell time and the distance as the eye movement data.

4. The method according to claim 3, It is characterized in that The obtaining of the pupil center coordinates on the human eye image comprises: Acquire a pupil area in the human eye image; Obtaining boundary point coordinates of a preset number of boundary points on the boundary of the pupil region; Based on the boundary point coordinates, the pupil center coordinates are obtained.

5. The method according to claim 4, It is characterized in that The step of obtaining the center coordinates of the corneal reflection spot on the human eye image includes: Filtering the human eye image to obtain a plurality of corneal reflection spot images; Acquire the center coordinates of multiple corneal reflection spot images corresponding to the multiple corneal reflection spot images; The center coordinates of the multiple corneal reflection spot images are fused to obtain the center coordinates of the corneal reflection spot.

6. The method according to claim 1, It is characterized in that After determining the target gaze area from the candidate gaze areas, the method further includes: In response to a trigger request for the target gaze area, the interface is switched to a target interface corresponding to the target gaze area.

7. The method according to claim 6, It is characterized in that The method further comprises: The display brightness of the target gaze area on the interface is set to be greater than the display brightness of a non-target gaze area; the non-target gaze area is an area other than the target gaze area among the multiple areas.

8. A region determination device, It is characterized in that The device comprises: An eye movement data acquisition module, used to acquire eye movement data; the eye movement data is used to characterize the changes in the gaze point of the human eye on the gaze interface, the interface including multiple areas; An estimation result acquisition module is used to input the eye movement data into a plurality of pre-constructed eye movement data estimation models, and obtain a plurality of estimation results of each region for the plurality of eye movement data estimation models through the plurality of eye movement data estimation models; the plurality of eye movement data estimation models correspond to the plurality of regions one by one, and the plurality of estimation results are used to characterize the probability that each region is a fixation region of the human eye; A candidate fixation region determination module is used to use the estimation results corresponding to the eye movement data estimation model corresponding to each region as the target estimation result; based on the target estimation result, select the candidate fixation region from each region; The target gaze area judgment module is used to construct a gaze area recognition framework based on multiple estimated results corresponding to the candidate gaze areas, including: calibrating multiple estimated results corresponding to each candidate gaze area through a calibration method constructed based on a non-parametric method to obtain multiple calibration results corresponding to each candidate gaze area; fusing the multiple calibration results corresponding to each candidate gaze area to obtain the fusion results corresponding to each candidate gaze area; constructing the gaze area recognition framework based on the fusion results corresponding to each candidate gaze area; obtaining the recognition results corresponding to each candidate gaze area based on the gaze area recognition framework; if the value of the recognition result corresponding to the current candidate gaze area is the maximum value of all the recognition results, and the value of the recognition result corresponding to the current candidate gaze area is greater than a preset threshold, then the current candidate gaze area is determined to be the target gaze area.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images