Gaze region identification method and apparatus

Abstract

The application relates to a gaze area identification method and device. The method comprises the following steps: performing feature extraction on at least one acquired first image to obtain feature data of each first image, wherein the feature data comprises face feature point data, head pose data and eye feature data, and the first image comprises a face of a user; inputting the feature data into a trained line-of-sight estimation model to obtain gaze point coordinates corresponding to each first image and a gaze probability graph, wherein the gaze probability graph comprises probability values of gaze lines being in each gaze area of a screen gazed at by the user; processing the gaze point coordinates and the gaze probability graph of each first image to form a gaze point queue and a probability graph queue; and determining a user gaze area from a plurality of gaze areas according to the gaze point queue and the probability graph queue. The method and device provided in the application have small error, low jitter and low cost in user gaze area estimation, and can be deployed in a full scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vision, in particular to a gaze area identification method and device. BACKGROUND

[0002] Eye movement technology has a very high market value. Eye movement control as a new type of interaction has great exploration and research value in the fields of reading, gaming, psychology, marketing, etc. The value of eye movement technology in the field of social disabled persons protection has been widely recognized, and there are a small number of research results. Eye movement technology includes gaze point control based on terminal devices with screens such as mobile phones and tablets, and gaze line estimation control based on three-dimensional space. However, the implementation of gaze estimation in related technologies has the following problems: in some technical solutions, the error of gaze estimation can meet the use demand, but special hardware devices (such as near-infrared cameras) are required to realize gaze estimation, which is expensive and cannot be deployed in all scenarios. In some technical solutions, gaze estimation is realized by recognizing images in RGB color mode (RGB for short), but the error of gaze estimation is large and the estimation result is difficult to eliminate. Moreover, the RGB image is greatly affected by the environmental factors of the photographed image, resulting in large noise in the photographed RGB image, which also increases the error of gaze estimation, and cannot meet the use demand of terminal devices. Reducing the error, cost and jitter of gaze estimation and realizing the full-scene deployment of gaze estimation are technical problems to be solved. SUMMARY

[0003] Therefore, a gaze area identification method and device are provided.

[0004] In a first aspect, an embodiment of the present application provides a gaze area identification method, which comprises:

[0005] performing feature extraction on the obtained at least one first image to obtain feature data of each first image, the feature data comprising face feature point data, head pose data and eye feature data, and the first image comprising a face of a user;

[0006] inputting the feature data into a trained gaze estimation model to obtain gaze point coordinates and gaze probability maps corresponding to each first image, the gaze point coordinates representing coordinates of an intersection point of binocular gaze lines, the gaze probability maps comprising probability values of gaze lines being in gaze areas of a screen gazed at by the user, the gaze lines representing lines connecting the eyes and the gaze points, and the screen comprising a plurality of gaze areas pre-divided;

[0007] processing the gaze point coordinates and the gaze probability maps of each first image to form a gaze point queue and a probability map queue;

[0008] According to the gaze point queue and the probability graph queue, a user gaze region is determined from the plurality of gaze regions.

[0009] The method provided by the first aspect can realize gaze region estimation based on an RGB image. The method can be applied to a terminal device, so that a gaze region of a user when watching a screen can be estimated, the error of the estimated user gaze region is small, the jitter is low, the cost of implementing the method is low, and full-scene deployment can be realized.

[0010] According to the first aspect, in a first possible implementation manner of the method, processing the gaze point coordinates and the gaze probability graph of each first image to form a gaze point queue and a probability graph queue includes:

[0011] According to the gaze probability graph, a probability value of the gaze point coordinates corresponding to the gaze probability graph is determined.

[0012] According to the probability value of the gaze point coordinates of each first image, an effective gaze point coordinate is selected from each gaze point coordinate, and an effective gaze probability graph corresponding to the effective gaze point coordinate is determined.

[0013] According to the effective gaze probability graph and the effective gaze point coordinate, the gaze point queue and the probability graph queue are formed.

[0014] According to the first aspect, in a second possible implementation manner of the method, according to the gaze point queue and the probability graph queue, a user gaze region is determined from the plurality of gaze regions, including:

[0015] The gaze point queue and the probability graph queue are respectively filtered to obtain a filtered gaze point queue and a probability heat map.

[0016] According to the filtered gaze point queue and the probability heat map, a user gaze region is determined from the plurality of gaze regions.

[0017] In the second possible implementation manner of the method, before the gaze point queue and the probability graph queue are formed, the selection is performed first, and then the effective gaze point coordinate and the effective gaze probability graph are used to form the gaze point queue and the probability graph queue, so that the reliability and stability of data in the gaze point queue and the probability graph queue are improved, and the error and jitter of the user gaze region determination are reduced.

[0018] According to the second possible implementation manner of the first aspect, in a third possible implementation manner of the method, the gaze point queue and the probability graph queue are respectively filtered to obtain a filtered gaze point queue and a probability heat map, including:

[0019] The gaze point queue is filtered to obtain a filtered gaze point queue.

[0020] The gaze probability maps in the probability map queue are multiplied and normalized to form a probability heat map.

[0021] In this way, the reliability and stability of the data can be improved through filtering.

[0022] According to a fourth possible implementation of the method according to the first aspect, before determining the user gaze region from the plurality of gaze regions according to the filtered gaze point queue and the probability heat map, the method further comprises:

[0023] determining a probability value corresponding to each gaze point coordinate in the filtered gaze point queue according to the probability heat map;

[0024] screening according to the probability value of each gaze point coordinate, removing the gaze point coordinates with abnormal probability values in the filtered gaze point queue, and

[0025] updating the probability heat map according to the gaze probability map corresponding to the gaze point coordinates with abnormal probability values.

[0026] According to a fifth possible implementation of the method according to the first aspect, the method further comprises:

[0027] training the gaze estimation model using the feature data of the training images to obtain the trained gaze estimation model;

[0028] wherein the model parameters of the gaze estimation model are updated using the loss of the gaze estimation model determined according to the preset loss function during training;

[0029] wherein the preset loss function is determined according to a regression loss function and a cross-entropy loss function.

[0030] According to a second aspect, an embodiment of the present application provides a gaze region identification device, the device comprising:

[0031] a feature extraction module configured to perform feature extraction on the obtained at least one first image to obtain feature data of each first image, the feature data comprising face feature point data, head pose data and eye feature data, and the first image comprising a face of a user;

[0032] The gaze estimation module is configured to input the feature data into a trained gaze estimation model to obtain gaze point coordinates and gaze probability maps corresponding to each first image, wherein the gaze point coordinates represent coordinates of intersection points of binocular gaze lines, the gaze probability maps include probability values of gaze lines at each gaze region of a screen where the user gazes, the gaze lines represent lines connecting the eye and the gaze points, and the screen includes a plurality of pre-divided gaze regions.

[0033] The queue creation module is configured to process the gaze point coordinates and the gaze probability maps of each first image to form a gaze point queue and a probability map queue.

[0034] The region determination module is configured to determine a user gaze region from the plurality of gaze regions according to the gaze point queue and the probability map queue.

[0035] According to a first possible implementation of the device according to the second aspect, the queue creation module comprises:

[0036] The probability value determination submodule is configured to determine a probability value of the gaze point coordinates corresponding to the gaze probability map according to the gaze probability map.

[0037] The screening submodule is configured to screen valid gaze point coordinates from the gaze point coordinates according to the probability values of the gaze point coordinates of each first image, and determine valid gaze probability maps corresponding to the valid gaze point coordinates.

[0038] The queue formation submodule is configured to form a gaze point queue and a probability map queue according to the valid gaze probability maps and the valid gaze point coordinates.

[0039] According to a second possible implementation of the device according to the second aspect, the region determination module comprises:

[0040] The filtering submodule is configured to perform filtering processing on the gaze point queue and the probability map queue respectively to obtain a filtered gaze point queue and a probability heat map.

[0041] The region determination submodule is configured to determine a user gaze region from the plurality of gaze regions according to the filtered gaze point queue and the probability heat map.

[0042] According to the third possible implementation of the device according to the second possible implementation of the second aspect, the filtering submodule comprises:

[0043] The first filtering submodule is configured to perform filtering processing on the gaze point queue to obtain a filtered gaze point queue.

[0044] The second filtering sub-module is configured to normalize the gaze probability maps in the queue to form a probability heat map.

[0045] In a fourth possible implementation form of the apparatus according to the second possible implementation form of the second aspect, the apparatus further comprises:

[0046] The determining module is configured to determine, according to the probability heat map, a probability value corresponding to each gaze point coordinate in the filtered gaze point queue;

[0047] The screening and updating module is configured to screen, according to the probability value of each gaze point coordinate, remove a gaze point coordinate with an abnormal probability value in the filtered gaze point queue, and

[0048] update the probability heat map according to a gaze probability map corresponding to the gaze point coordinate with the abnormal probability value.

[0049] In a fifth possible implementation form of the apparatus according to the second aspect, the apparatus further comprises:

[0050] The model training module is configured to train the gaze estimation model by using feature data of training images to obtain the trained gaze estimation model.

[0051] In the training, a loss of the gaze estimation model determined according to a preset loss function is used to update model parameters of the gaze estimation model.

[0052] The preset loss function is determined according to a regression loss function and a cross-entropy loss function.

[0053] In a third aspect, embodiments of the present application provide a gaze region identification apparatus, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method of the first aspect or any one of the possible implementation forms of the first aspect.

[0054] In a fourth aspect, embodiments of the present application provide a non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the method of the first aspect or any one of the possible implementation forms of the first aspect.

[0055] In a fifth aspect, embodiments of the present application provide a computer program product, comprising computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the method of the first aspect or any one of the possible implementation forms of the first aspect.

[0056] These and other aspects of the application will become more fully understood from the following (detailed description). BRIEF DESCRIPTION OF DRAWINGS

[0057] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate exemplary embodiments, features, and aspects of the application and, together with the description, serve to explain the principles of the present application.

[0058] Figure 1 A structural schematic diagram of an electronic device 100 is shown.

[0059] Figure 2 A flowchart of a gaze region identification method according to an embodiment of the present application is shown.

[0060] Figure 3 A schematic diagram of facial feature points according to an embodiment of the present application is shown.

[0061] Figure 4 A schematic diagram of head pose extraction according to an embodiment of the present application is shown.

[0062] Figure 5 A schematic diagram of a user gazing at a screen according to an embodiment of the present application is shown.

[0063] Figure 6 A schematic diagram of a gaze probability map according to an embodiment of the present application is shown.

[0064] Figure 7 A schematic diagram of the principle of a line-of-sight estimation model according to an embodiment of the present application is shown.

[0065] Figure 8 A schematic diagram of filtering processing of a probability map queue according to an embodiment of the present application is shown.

[0066] Figure 9 A flowchart of a gaze region identification method according to an embodiment of the present application is shown.

[0067] Figure 10 A block diagram of a gaze region identification apparatus according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0068] Various exemplary embodiments, features, and aspects of the present application will be described herein below with reference to the accompanying drawings. Like reference numerals in the drawings denote functionally similar or the same elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0069] The term "exemplary" is used herein in the sense of being an example, instance, or illustration. Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0070] In addition, for a better illustration of the present application, numerous specific details are given in the following detailed description. Those skilled in the art will understand that the present application can be practiced without certain specific details. In some instances, well-known methods, means, elements and circuits have not been described in detail in order to avoid obscuring the present application.

[0071] To solve the above technical problems, the present application provides a gaze area identification method and device, the gaze area identification method of the present application can realize gaze area estimation based on RGB image. The method can be applied to a terminal device, so as to estimate the gaze area of a user when the user watches the screen, and the error of the estimated user gaze area is small, the jitter is low, the cost of realizing the method is low, and full-scene deployment can be realized.

[0072] Figure 1 A structural schematic diagram of an electronic device 100 is shown.

[0073] The terminal device of the present application can be an electronic device with a screen. The electronic device 100 can include at least one of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device, or a smart city device. The present application does not specially limit the specific type of the electronic device 100.

[0074] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) terminal 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyro sensor 180B, a barometric sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0075] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than illustrated, or combine certain components, or split certain components, or different arrangement of components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0076] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0077] The processor can generate operation control signals according to instruction opcodes and timing signals, and complete the control of fetching and executing instructions. The gaze area identification method provided in the present application can be implemented by the processor. The processor can run the gaze area identification method to determine the user's gaze area.

[0078] The processor 110 can also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 can be a cache memory. The memory can hold instructions or data that are frequently used or used by the processor 110. If the processor 110 needs to use the instructions or data, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0079] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 110 can connect the touch sensor, the audio module, the wireless communication module, the display, the camera, and other modules through at least one of the above interfaces.

[0080] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In other embodiments of the present application, the electronic device 100 can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0081] The electronic device 100 can realize the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0082] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the electronic device 100 can include one or more display screens 194. The display screen 194 is also referred to as a screen as described below. At least one control that can be operated by the user through the gaze operation can be displayed on the display screen 194. When the gaze area of the user determined by the gaze area identification method of the present application is identical to or partially overlaps with the area of a certain control, it can be determined that the control is triggered, and the processor 110 or other components of the electronic device 100 can perform the operation of the triggered control.

[0083] The electronic device 100 can implement the camera function through the camera module 193, an ISP, a video codec, a GPU, the display screen 194, and an application processor (AP) and a neural network processing unit (NPU). The electronic device 100 can use its own camera function to capture an image when detecting that the user gazes at the display screen 194 or receiving a photographing instruction, to obtain a first image. The first image is sent to the processor 110, so that the processor 110 can determine the gaze area of the user by using the gaze area identification method provided in the present application.

[0084] The camera module 193 can be configured to collect color image data and depth data of a photographed object. The ISP can be configured to process the color image data collected by the camera module 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm for noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the photographed scene. In some embodiments, the ISP can be disposed in the camera module 193.

[0085] In some embodiments, the camera module 193 can include a color camera module and a 3D sensing module.

[0086] In some embodiments, the photosensitive element of the camera of the color camera module can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, or the like format image signal.

[0087] In some embodiments, the 3D sensing module can be a time of flight (TOF) 3D sensing module or a structured light 3D sensing module. The structured light 3D sensing is an active depth sensing technology, and the basic components of the structured light 3D sensing module can include an infrared (IR) emitter, an IR camera module, and the like. The working principle of the structured light 3D sensing module is to first emit a specific pattern of light spots to the object to be photographed, then receive the light spot pattern coding on the surface of the object, and then compare the differences between the original projected light spot and the received light spot, and calculate the three-dimensional coordinates of the object using the principle of triangulation. The three-dimensional coordinates include the distance between the electronic device 100 and the object to be photographed. The TOF 3D sensing can be an active depth sensing technology, and the basic components of the TOF 3D sensing module can include an infrared (IR) emitter, an IR camera module, and the like. The working principle of the TOF 3D sensing module is to calculate the distance (i.e., the depth) between the TOF 3D sensing module and the object to be photographed by the time of infrared reflection, to obtain a 3D depth map.

[0088] The structured light 3D sensing module can also be applied to face recognition, motion game consoles, industrial machine vision detection, and the like. The TOF 3D sensing module can also be applied to game consoles, augmented reality (AR) / virtual reality (VR), and the like.

[0089] In some embodiments, the electronic device 100 can include one or more camera modules 193. Specifically, the electronic device 100 can include one front-facing camera module 193 and one rear-facing camera module 193. The front-facing camera module 193 can be typically used to capture color image data and depth data of the photographer facing the display screen 194, and the rear-facing camera module can be used to capture color image data and depth data of the object (e.g., a person, a landscape, etc.) facing the photographer.

[0090] In some embodiments, the electronic device 100 can include one or more camera modules 193. Specifically, the electronic device 100 can include one front-facing camera module 193 and one rear-facing camera module 193. The front-facing camera module 193 can be typically used to capture color image data and depth data of the photographer facing the display screen 194, and the rear-facing camera module can be used to capture color image data and depth data of the object (e.g., a person, a landscape, etc.) facing the photographer.

[0091] Figure 2 A flowchart of a gaze area identification method according to an embodiment of the present application is shown. The present application provides a gaze area identification method as shown in Figure 2 The gaze area identification method can be applied to a terminal device, and the method includes steps S11 to S14. The process of determining the user's gaze area according to the present application can be applied to the following scenario: the user controls the terminal device by gazing, and the terminal device can display controls operable by the user on the screen. When the identified user's gaze area and the position of a certain control on the screen are the same, the terminal device can perform the operation corresponding to the control. It can also be applied to the scenario of monitoring the user's eye movement process. The first image of the user gazing at the screen can be continuously captured within a period of time, and the user's eye movement process can be determined according to the change of the user's gaze area within the period of time.

[0092] In step S11, feature extraction is performed on the obtained at least one first image to obtain feature data of each first image, and the feature data includes face feature point data, head pose data, and eye feature data. The first image includes the user's face.

[0093] In the present embodiment, the face feature point data can be data representing a plurality of feature points of the user's face features, and the face feature points can include feature points representing the user's eyebrows, eyes, nose, mouth, and face contour. Figure 3 A schematic diagram of face feature points according to an embodiment of the present application is shown. The face feature point data obtained in the present embodiment can be as shown in Figure 3Some or all of the 68 feature points shown in (68 points numbered 1-68 are the positions of the human face feature points) Figure 3 The data of some or all of the 68 feature points numbered 1-68 are the positions of the human face feature points. The image coordinates of each feature point in the extracted human face feature point data can be represented in the form of a matrix, which can be a 1x n matrix, n is 2 times the number of human face feature points j, then the matrix is (x1, y1, x2, y2…xj, yj), jx2=n. For example, the human face feature point data can be the data of the points representing the features of the eyes, nose, and mouth in the human face, i.e., the data of the 41 feature points numbered 28-68 in the figure, the image coordinates of the 41 feature points can be represented as a 1x82 matrix, i.e., (x1, y1, x2, y2…x41, y41), then x1, y1 are the coordinates of the feature point numbered 28 in the image; x2, y2 are the coordinates of the feature point numbered 29 in the image…x41, y41 are the coordinates of the feature point numbered 68 in the image. A human face detection model can be created according to the number and position requirements of the human face feature points to be extracted, and the human face detection model is trained using sample images to obtain a human face detection model capable of extracting human face feature points. Then the first image is input into the trained human face detection model to obtain human face feature point data, i.e., a matrix composed of image coordinates of multiple feature points.

[0094] In this embodiment, the head pose data can be data representing the pose of the user's face in the first image. Figure 4 A schematic diagram of head pose extraction according to an embodiment of the present application is shown. As shown in Figure 4 The multiple coordinate points can be selected to form a head pose frame M for extracting the head pose, and the angles of the three dimensions pitch, yaw, and roll of the head are determined using the head pose frame M, and then the image coordinates of the multiple feature points constituting the head pose frame M in the first image and the values of the three dimensions pitch, yaw, and roll are taken as the head pose data. The multiple coordinate points constituting the head pose frame can be some or all of the human face feature points. For example, as shown in Figure 4 The 8 coordinate points constituting the head pose frame M can be selected, the head pose frame M is a solid structure like a prism, the four vertices of one bottom surface of the prism (as shown by the dashed rectangle in Figure 4 The four vertices of the other bottom surface of the prism (as shown by the solid rectangle in Figure 4 The four vertices of the other bottom surface of the prism (as shown by the solid rectangle in Figure 4The top point coordinate of the head pose frame M dashed rectangle can be determined according to the recognized facial feature points 37 and 40, another top point coordinate of the head pose frame M dashed rectangle can be determined according to the recognized facial feature points 42 and 45, and the length of the long side of the dashed rectangle can be determined according to the recognized facial feature points 42 and 58. Then, the coordinates of the four top points of the dashed rectangle can be determined. Further, the coordinates of the four top points of the solid rectangle can be determined according to the preset proportional relationship between the solid rectangle and the dashed rectangle, or in combination with the facial feature points, and finally the head pose frame M is obtained. The head pose frame M can also be in other shapes, which can be set according to actual needs by those skilled in the art, and the present application does not limit this. The pose detection model for extracting the head pose data can be created in advance according to the determined head pose extraction manner, the pose detection model is trained by using sample images, and the trained pose detection model is obtained.

[0095] In the present embodiment, the eye feature data can be data representing the viewing features of the user's eyes, including the size of the user's eye white, iris and pupil, the relative position relationship of the eye white, iris and pupil relative to the eye socket, etc. In step S11, the first image can be first subjected to a clipping process to intercept the eye image in the first image, and then the eye detection model trained in advance is used to recognize the eye image to obtain the eye feature data.

[0096] It should be noted that the present application exemplarily provides an implementation manner of extracting the facial feature point data, head pose data and eye feature data of the first image, and those skilled in the art can set the implementation manner of extracting the facial feature point data, head pose data and eye feature data according to actual needs, and the present application does not limit this.

[0097] In the embodiment, the first image can be captured by a terminal device with a screen on which the user gazes, and the capturing time length and frequency of the captured first image can be set. The longer the capturing time length and the higher the frequency within a certain time length range, the smaller the error of the identified user gaze area, and the higher the identification accuracy. The terminal device that captures the first image and the terminal device that executes the gaze area identification method of the present application can be the same or different. For example, the first terminal device has a screen and is used to capture the first image during the user gazes at the screen, and then the first terminal device sends the captured first image to the second terminal device. After the second terminal device receives the first image, steps S11-S14 are executed to obtain the user gaze area, and then the information of the user gaze area is sent to the first terminal device. The first terminal device performs corresponding operations based on the determined user gaze area, and the performed operations include responding to the control of the first terminal device by the user gaze area to display corresponding content for the user, and the first terminal device performs operations such as making a phone call, playing a song, closing software, etc. That is, the user can control the first terminal device to perform corresponding operations by gazing.

[0098] In step S12, the feature data is input into the trained gaze estimation model to obtain the gaze point coordinates corresponding to each first image and the gaze probability map. The gaze point coordinates represent the coordinates of the intersection point of the binocular gaze lines, and the gaze probability map includes the probability of the gaze line being in each gaze area of the screen. The gaze line represents the line connecting the eye and the gaze point, and the screen includes a plurality of pre-divided gaze areas.

[0099] In the embodiment, before the face feature point data, the head pose data, and the eye feature data are input into the gaze estimation model, the face feature point data and the head pose data can also be standardized respectively. In this way, it can be ensured that the gaze estimation model can process the feature data and ensure the accuracy of the result.

[0100] Figure 5 A schematic diagram of a user gazing at a screen according to an embodiment of the present application is shown. As shown in Figure 5 During the user gazes at the screen with both eyes, the gaze lines of the left and right eyes converge on the screen to form a gaze point. The gaze point coordinates are the coordinates (x, y) of the gaze point on the screen. Since there is a mapping relationship between the first image and the screen size, the coordinates of the gaze point on the first image can be used to determine the coordinates of the gaze point on the screen based on the mapping relationship. In addition, the screen can be divided into a plurality of gaze areas 1-1, 1-2, …, m-n according to the requirements of different use scenarios for identification accuracy. The more the number of gaze areas, the more accurate the identification accuracy. The plurality of gaze areas divided by the screen can be as shown in Figure 5 、 Figure 6As shown in the grid, the shape of each gaze region can be rectangular, triangular, square, or other regular shape, or irregular shape. The areas of the plurality of gaze regions can be the same or different, and the area of a gaze region can be set according to the distance of the gaze region from the center position of the screen, for example, the closer to the center position of the screen, the smaller the area of the gaze region. Alternatively, the area of the corresponding gaze region can be determined according to the size of the user's use frequency of different regions of the screen, for example, the higher the use frequency of a region, the smaller the area of the gaze region. In this way, adjusting the area of the gaze region according to the position of the gaze region and the corresponding use frequency can reduce the estimation error. Dividing the screen into a plurality of gaze regions can improve the reliability of the user's gaze region determination. At the same time, dividing the screen into a plurality of gaze regions can also alleviate the problem of low data reliability caused by errors in the gaze point coordinates labeled in the training images during the training of the gaze estimation model.

[0101] In this embodiment, the gaze probability map can be a probability value that indicates that the gaze point is in each gaze region in the screen, Figure 6 A schematic diagram of a gaze probability map according to an embodiment of the present application is shown. As shown in the figure, Figure 6 According to the gaze probability map, it can be determined that the probability value of the gaze point being in gaze region 1-1 is 0.000, the probability value of the gaze point being in gaze region 1-2 is 0.000,..., the probability value of the gaze point being in gaze region 2-4 is 0.700,..., and the probability value of the gaze point being in gaze region m-n is 0.000.

[0102] In a possible implementation, before step S12 is performed, the method further includes training the gaze estimation model using feature data of the training images to obtain the trained gaze estimation model. In the training, the loss of the gaze estimation model determined according to a preset loss function is used to update the model parameters of the gaze estimation model. The preset loss function is determined according to a regression loss function and a cross entropy loss function.

[0103] In this implementation, the preset loss function can be determined by weighted addition of the regression loss function and the cross entropy loss function, which can ensure the accuracy of the trained gaze estimation model and reduce the estimation error of the finally output user gaze region. The regression loss function can be a mean square error (MSE) function used in regression problems, etc. For example, the preset loss function L can be as follows:

[0104]

[0105] L = w1 * MSE + w2 * Cross Entropycross-entropy represents the cross entropy loss function. L MSE represents the mean square error function. σ1 and σ2 represent weights. logσ1σ2 represents the regularization term.

[0106] Figure 7 FIG. 1 shows a schematic diagram of the principle of a sight line estimation model according to an embodiment of the present application. Figure 7 As shown, the present application provides an example of a gaze estimation model. The implementation estimation model can be a neural network model, which includes multiple operation nodes, such as cascade 1, cascade 2, and full connections 1, 2, 3, and 4. The process of using the gaze estimation model to process facial feature point data, head pose data, and eye feature data is as follows: the terminal device inputs the facial feature point data and head pose data into the gaze estimation model after standardizing them respectively. The operation node "Cascade 1" in the gaze estimation model performs cascade processing on the standardized facial feature point data and head pose data (that is, determines the mapping relationship between the facial feature point data and the head pose data), obtains a first intermediate result, and inputs it into the operation node "Full Connection 2". Then, the operation node "Full Connection 2" performs full connection processing on the first intermediate result, obtains a second intermediate result, and inputs it into the operation node "Cascade 2". For the eye feature data, after the terminal device inputs it into the gaze estimation model, the operation node "Full Connection 1" in the gaze estimation model performs full connection processing on the eye feature data, obtains a third intermediate result, and inputs it into the operation node "Cascade 2". The "Cascade 2" operation node concatenates the third intermediate result with the second intermediate result (i.e., determines the mapping relationship between the second and third intermediate results), obtaining a fourth intermediate result, which is then fed into the "Full Connect 3" and "Full Connect 4" operation nodes, respectively. The "Full Connect 3" operation node performs a full connect on the fourth intermediate result to obtain the fixation point coordinates. The "Full Connect 4" operation node performs a full connect on the fourth intermediate result to obtain the probability value and fixation area coordinate range for each fixation area on the screen. A fixation probability map is then determined based on the probability value and fixation area coordinate range for each fixation area.

[0107] In step S13 , the gaze point coordinates and the gaze probability maps of each first image are processed to form a gaze point queue and a probability map queue.

[0108] In this embodiment, after obtaining the gaze point coordinates and gaze probability maps of multiple first images, the gaze point coordinates and gaze probability maps corresponding to the multiple first images can be sorted in order of the shooting time of the first images from early to late, and a gaze point queue and a probability map queue can be formed.

[0109] In a possible implementation, step S13 can include: determining, according to the gaze probability map, a probability value of a gaze point coordinate corresponding to the gaze probability map; screening, from each gaze point coordinate, an effective gaze point coordinate according to the probability value of the gaze point coordinate of each image, and determining an effective gaze probability map corresponding to the effective gaze point coordinate; and forming a gaze point queue and a probability map queue according to the effective gaze probability map and the effective gaze point coordinate.

[0110] In this implementation, the probability value of the gaze region where the gaze point coordinate determined in step S12 is located can be determined according to the gaze probability map, as shown in FIG. 2. As shown in FIG. 2, the gaze point (x, y) is located in the gaze region 2-4, and the probability value is 0.700. Then, each gaze point coordinate is screened according to a pre-set screening condition, to obtain an effective gaze point coordinate. The screening condition can be that the probability value corresponding to the gaze point coordinate is greater than or equal to a probability threshold value. For example, the probability threshold value can be 0.5, and the gaze point coordinate with a probability value greater than 0.5 is the effective gaze point coordinate. The gaze probability map corresponding to the effective gaze point coordinate is the effective gaze probability map. Then, a gaze point queue and a probability map queue are formed according to the effective gaze point coordinate and the effective gaze probability map. Figure 6

[0111] In this way, before the gaze point queue and the probability map queue are formed, screening is performed first, and then the gaze point queue and the probability map queue are formed according to the effective gaze point coordinate and the effective gaze probability map, so that the reliability and stability of the data in the gaze point queue and the probability map queue are improved, and the error and jitter of the user gaze region determination are reduced.

[0112] In step S14, a user gaze region is determined from the plurality of gaze regions according to the gaze point queue and the probability map queue.

[0113] In a possible implementation, step S14 can include: performing filtering processing on the gaze point queue and the probability map queue respectively to obtain a filtered gaze point queue and a probability heat map; and determining a user gaze region from the plurality of gaze regions according to the filtered gaze point queue and the probability heat map.

[0114] For filtering processing of the gaze point queue, the gaze point queue can be processed by an arithmetic average filtering method, a recursive average filtering method (also known as a sliding average filtering method), a median average filtering method (also known as an anti-impulse interference average filtering method), a de-bouncing filtering method, a Kalman filtering (non-extended Kalman), and the like, to achieve smooth output of data and increase the reliability and stability of the data in the filtered gaze point queue.

[0115] Figure 8 A schematic diagram of filtering processing of a probability map queue according to an embodiment of the present application is shown in FIG. 3. As shown in FIG. 3, the filtering processing of the probability map queue can be performed by an arithmetic average filtering method, a recursive average filtering method (also known as a sliding average filtering method), a median average filtering method (also known as an anti-impulse interference average filtering method), a de-bouncing filtering method, a Kalman filtering (non-extended Kalman), and the like.​Figure 8 As shown, the filtering processing on the probability map queue can be performed in the following manner: first, multiply all the gaze probability maps in the probability map queue to obtain a multiplied probability map; then, perform normalization processing on the multiplied probability map to obtain a probability heat map. The probability heat map can be recorded as shown, and the probability values of different gaze regions can also be intuitively distinguished by adding fillings corresponding to the probability values to the gaze regions. Figure 8

[0116] In a possible implementation, before determining the user gaze region from the plurality of gaze regions according to the filtered gaze point queue and the probability heat map, the method can further include: determining, according to the probability heat map, the probability value corresponding to the gaze point coordinate in the filtered gaze point queue; screening, according to the probability value of each gaze point coordinate, the gaze point coordinate with an abnormal probability value in the filtered gaze point queue, and updating the probability heat map according to the gaze probability map corresponding to the gaze point coordinate with the abnormal probability value.

[0117] In this implementation, the probability value of the gaze region where the corresponding gaze point coordinate is located can be determined according to the probability heat map. Then, the gaze point coordinates in the filtered gaze point queue are screened according to a pre-set screening condition, and the gaze point coordinates with abnormal probability values are removed. Then, a new probability heat map is generated according to the gaze probability map corresponding to the gaze point coordinate with the abnormal probability value and the generated probability heat map. The screening condition can be that the probability value corresponding to the gaze point coordinate is less than or equal to a probability threshold value. For example, the probability threshold value can be 0.25, and the gaze point coordinate with a probability value less than or equal to 0.25 is the gaze point coordinate with an abnormal probability value.

[0118] ​In this embodiment, the user gaze region can be determined from the gaze points in the filtered gaze point queue and the probability heat map, for example, by voting. For example, the user gaze region is determined by voting: suppose there are 100 gaze point coordinates in the filtered gaze point queue. The voting value of each gaze point coordinate for each gaze region is calculated according to whether the gaze point coordinate is in the gaze region and the probability value of the gaze region, and the total voting value of each gaze region is obtained. For example, the voting value T(i-j) of the ith gaze point coordinate for the jth gaze region can be T(i-j) = a x Pi. When the ith gaze point coordinate is in the jth gaze region, a is 1; when the ith gaze point coordinate is not in the jth gaze region, a is 0. Pi is the probability value of the ith gaze point coordinate in the jth gaze region, which is obtained from the probability heat map. Then the total voting value T(j) of the 100 gaze point coordinates for the jth gaze region can be T(j) = T(1-j) + T(2-j) +... + T(100-j). Then the gaze region with the largest total voting value is selected as the user gaze region. The voting method can increase the reliability of the determined user gaze region.

[0119] In addition, after the gaze region with the largest total voting value is determined as the user gaze region, the range of the user gaze region can be corrected according to the gaze point coordinates in the filtered gaze point queue to improve the accuracy of the user gaze region. For example, a regular or irregular gaze point region can be determined according to the gaze point coordinates in the filtered gaze point queue, and then the part of the determined user gaze region that belongs to the gaze point region is determined as the corrected user gaze region.

[0120] To further describe the gaze region identification method provided in the present application, the following describes an example of the gaze region identification method according to an embodiment of the present application. Figure 9 The gaze region identification method example includes steps S21 to S28. The example is applied to a terminal device with a screen. Figure 9 The gaze region identification method example includes steps S21 to S28. The example is applied to a terminal device with a screen.

[0121] In step S21, after determining that gaze region identification is needed, the terminal device starts the camera to take images and obtains a plurality of first images, which include the face of the user.

[0122] In step S22, after taking the first images, the terminal device extracts features from the first images to obtain face feature point data, head pose data, and eye feature data. The implementation process is described above in step S11, which is not repeated here.

[0123] In step S23, the terminal device inputs the facial feature point data, head posture data, and eye feature data into a pre-trained gaze estimation model to obtain the gaze point coordinates and gaze probability map of each first image. Figure 7 And the related text descriptions are not repeated here.

[0124] In step S24, the first outlier filtering is performed to obtain the effective gaze point coordinates and the effective gaze probability map (for the implementation process, see the above process of determining the effective gaze point coordinates and the effective gaze probability map).

[0125] In step S25 , a gaze point queue is formed using all valid gaze point coordinates, and step S26 is continued to filter the gaze point queue to obtain a filtered gaze point queue.

[0126] In step S25', all valid gaze probability maps are used to form a probability map queue, and step S26' is continued to perform superposition processing on the gaze probability maps in the probability map queue to obtain a superposition probability map, and the superposition probability map is normalized to obtain a probability heat map.

[0127] In step S27, a second outlier filtering is performed to remove the gaze point coordinates with abnormal probability values ​​in the filtered gaze point queue, and the probability heat map is updated according to the gaze probability map corresponding to the gaze point coordinates with abnormal probability values.

[0128] In step S28, the user's gaze area is determined from the multiple gaze areas using the filtered gaze point queue and the updated probability heat map obtained after executing step S27.

[0129] Figure 10 FIG. 1 is a block diagram of a gaze area recognition device according to an embodiment of the present application. Figure 10 As shown, the device includes: a feature extraction module 41, a gaze estimation module 42, a queue creation module 43 and a region determination module 44.

[0130] The feature extraction module 41 is used to extract features from at least one acquired first image to obtain feature data of each first image, wherein the feature data includes facial feature point data, head posture data and eye feature data, and the first image includes the user's face.

[0131] The gaze estimation module 42 is used to input the feature data into a trained gaze estimation model to obtain the gaze point coordinates and gaze probability map corresponding to each first image, wherein the gaze point coordinates represent the coordinates of the intersection of the binocular gazes, and the gaze probability map includes the probability values ​​of the gaze lines being in each gaze area of ​​the screen that the user is gazing at, and the gaze lines represent the line passing through the eyes and the gaze point, and the screen includes multiple pre-divided gaze areas.

[0132] The queue creating module 43 is configured to process the gaze point coordinates and the gaze probability maps of the first images to form a gaze point queue and a probability map queue.

[0133] The region determining module 44 is configured to determine a user gaze region from the plurality of gaze regions according to the gaze point queue and the probability map queue.

[0134] In a possible implementation, the queue creating module 43 can include:

[0135] The probability value determining sub-module is configured to determine a probability value of the gaze point coordinates corresponding to the gaze probability map according to the gaze probability map;

[0136] The screening sub-module is configured to screen valid gaze point coordinates from the gaze point coordinates according to the probability values of the gaze point coordinates of the first images, and determine valid gaze probability maps corresponding to the valid gaze point coordinates;

[0137] The queue forming sub-module is configured to form a gaze point queue and a probability map queue according to the valid gaze probability maps and the valid gaze point coordinates.

[0138] In a possible implementation, the region determining module 44 can include:

[0139] The filtering sub-module is configured to perform filtering processing on the gaze point queue and the probability map queue respectively to obtain a filtered gaze point queue and a probability heat map;

[0140] The region determining sub-module is configured to determine a user gaze region from the plurality of gaze regions according to the filtered gaze point queue and the probability heat map.

[0141] In a possible implementation, the filtering sub-module includes:

[0142] The first filtering sub-module is configured to perform filtering processing on the gaze point queue to obtain a filtered gaze point queue;

[0143] The second filtering sub-module is configured to multiply the gaze probability maps in the probability map queue and perform normalization processing to form a probability heat map.

[0144] In a possible implementation, the apparatus can further include:

[0145] The determining module is configured to determine a probability value of the gaze point coordinates corresponding to the filtered gaze point queue according to the probability heat map;

[0146] The screening update module is configured to screen according to the probability values of each gaze point coordinate, remove gaze point coordinates with abnormal probability values in the filtered gaze point queue, and

[0147] update the probability heat map according to a gaze probability map corresponding to the gaze point coordinates with abnormal probability values.

[0148] In a possible implementation, the device can further include:

[0149] a model training module configured to train a gaze estimation model using feature data of training images to obtain the trained gaze estimation model;

[0150] wherein a loss of the gaze estimation model determined according to a preset loss function is used to update model parameters of the gaze estimation model in training;

[0151] wherein the preset loss function is determined according to a regression loss function and a cross-entropy loss function.

[0152] The device can realize gaze region estimation based on an RGB image, so as to estimate a gaze region of a user when the user watches a screen, and the error of the estimated gaze region of the user is small and the jitter is low. The cost of the method is low, and the method can be deployed in a full scene.

[0153] Embodiments of the present application provide a gaze region identification device, including a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.

[0154] Embodiments of the present application provide a non-volatile computer-readable storage medium having computer program instructions stored thereon, the computer program instructions being executed by a processor to implement the above method.

[0155] Embodiments of the present application provide a computer program product, including computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in a processor of an electronic device, the processor in the electronic device executes the above method.

[0156] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital video disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium can be any tangible storage medium which can retain and store instructions for use by an instruction execution device.

[0157] Computer readable program instructions or code can be downloaded to a starting device, or to another computer or device from an external computer or storage device through a network, for example, the Internet, a local area network, a wide area network, and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0158] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0159] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0160] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or nonvolatile memory, or a suitable combination of the different types of computer readable storage media. The computer readable program instructions can also be downloaded to a computer, other programmable data processing apparatus, or other device from a computer readable storage medium or to an external computer or external storage device via a data signal that can be transmitted for example via a wired medium or a wireless medium such as the Internet or Wireless Application Protocol (WAP) signaling.

[0161] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0162] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions (s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts.

[0163] It is also important to note that each of the boxes in the block diagrams and / or flowchart illustrations, and combinations of boxes in the block diagrams and / or flowchart illustrations, can be implemented by hardware, for example, circuitry or an ASIC (Application Specific Integrated Circuit), or can be implemented by a combination of hardware and software, such as firmware or the like.

[0164] Although the present application has been described in connection with various embodiments thereof, it will be understood that other modifications can be made by those skilled in the art to the application described herein without departing from the scope of the application as defined by the appended claims. In the claims, the term "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The reference signs in the claims should not be construed as limiting the scope of the application.

[0165] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. It is intended that the scope of the application be defined by the scope of the patent and by the claims as allowed by the patent office, which can include adaptations based on the description, equivalents, and / or substitutions of elements individually or collectively to the entire disclosure.

Claims

1. A method for identifying a gaze area, characterized in that: The method comprises: Performing feature extraction on at least one acquired first image to obtain feature data of each first image, the feature data including facial feature point data, head posture data, and eye feature data, wherein the first image includes a user's face; Inputting the feature data into a trained gaze estimation model to obtain gaze point coordinates and a gaze probability map corresponding to each first image, wherein the gaze point coordinates represent the coordinates of the intersection of binocular gazes, and the gaze probability map includes probability values ​​of the gaze lines being in each gaze area of ​​the screen that the user is gazing at. The gaze lines represent lines passing through the eyes and the gaze points, and the screen includes multiple gaze areas that are pre-divided according to recognition accuracy requirements for different usage scenarios; Processing the gaze point coordinates and the gaze probability map of each first image to form a gaze point queue and a probability map queue; A user gaze area is determined from the multiple gaze areas according to the gaze point queue and the probability map queue.

2. The method according to claim 1, characterized in that Processing the gaze point coordinates and the gaze probability map of each first image to form a gaze point queue and a probability map queue includes: Determining, according to the gaze probability map, a probability value of a gaze point coordinate corresponding to the gaze probability map; Filtering valid gaze point coordinates from each gaze point coordinate according to the probability value of the gaze point coordinates of each first image, and determining a valid gaze probability map corresponding to the valid gaze point coordinates; A gaze point queue and a probability map queue are formed according to the effective gaze probability map and the effective gaze point coordinates.

3. The method according to claim 1, characterized in that Determining a user gaze area from the plurality of gaze areas according to the gaze point queue and the probability map queue includes: Performing filtering on the fixation point queue and the probability map queue respectively to obtain a filtered fixation point queue and a probability heat map; A user gaze area is determined from the multiple gaze areas according to the filtered gaze point queue and the probability heat map.

4. The method according to claim 3, characterized in that Filtering the fixation point queue and the probability map queue respectively to obtain a filtered fixation point queue and a probability heat map, including: Performing filtering on the gaze point queue to obtain a filtered gaze point queue; The gaze probability maps in the probability map queue are multiplied and normalized to form a probability heat map.

5. The method according to claim 3, characterized in that Before determining the user gaze area from the multiple gaze areas based on the filtered gaze point queue and the probability heat map, the method further includes: Determining probability values ​​corresponding to the coordinates of the gaze points in the filtered gaze point queue according to the probability heat map; Filtering is performed based on the probability value of each fixation point coordinate, removing fixation point coordinates with abnormal probability values ​​in the filtered fixation point queue, and The probability heat map is updated according to the gaze probability map corresponding to the gaze point coordinates with abnormal probability values.

6. The method according to claim 1, characterized in that The method further comprises: Training a sight line estimation model using feature data of the training image to obtain the trained sight line estimation model; wherein, during training, the loss of the sight line estimation model determined according to a preset loss function is used to update the model parameters of the sight line estimation model; The preset loss function is determined based on the regression loss function and the cross entropy loss function.

7. A gaze area recognition device, characterized in that: The device comprises: a feature extraction module, configured to extract features from at least one acquired first image to obtain feature data of each first image, wherein the feature data includes facial feature point data, head posture data, and eye feature data, wherein the first image includes the user's face; a gaze estimation module, configured to input the feature data into a trained gaze estimation model to obtain gaze point coordinates and a gaze probability map corresponding to each first image, wherein the gaze point coordinates represent the coordinates of the intersection of binocular gazes, and the gaze probability map includes a probability value of the gaze line being within each gaze area of ​​the screen being gazed by the user, wherein the gaze line represents a line passing through the eye and the gaze point, and the screen includes multiple gaze areas pre-divided according to the recognition accuracy requirements of different usage scenarios; a queue creation module, configured to process the gaze point coordinates and the gaze probability map of each first image to form a gaze point queue and a probability map queue; An area determination module is configured to determine a user gaze area from the multiple gaze areas based on the gaze point queue and the probability map queue.

8. The device according to claim 7, characterized in that The queue creation module includes: a probability value determination submodule, configured to determine, based on the gaze probability map, a probability value of a gaze point coordinate corresponding to the gaze probability map; a screening submodule, configured to screen out valid gaze point coordinates from each gaze point coordinate according to the probability value of the gaze point coordinates of each first image, and determine a valid gaze probability map corresponding to the valid gaze point coordinates; The queue forming submodule is used to form a gaze point queue and a probability map queue according to the effective gaze probability map and the effective gaze point coordinates.

9. The device according to claim 7, characterized in that The area determination module includes: A filtering submodule, configured to filter the fixation point queue and the probability map queue respectively to obtain a filtered fixation point queue and a probability heat map; The region determination submodule is configured to determine a user gaze region from the plurality of gaze regions based on the filtered gaze point queue and the probability heat map.

10. The device according to claim 9, characterized in that The filtering submodule includes: A first filtering submodule is configured to filter the fixation point queue to obtain a filtered fixation point queue; The second filtering submodule is used to perform normalization processing after multiplying the gaze probability maps in the probability map queue to form a probability heat map.

11. The device according to claim 9, characterized in that The device further comprises: a determination module, configured to determine probability values ​​corresponding to the coordinates of the gaze points in the filtered gaze point queue according to the probability heat map; A screening and updating module is used to screen according to the probability value of each fixation point coordinate, and remove the fixation point coordinates with abnormal probability values ​​in the filtered fixation point queue, and The probability heat map is updated according to the gaze probability map corresponding to the gaze point coordinates with abnormal probability values.

12. The device according to claim 7, characterized in that The device further comprises: A model training module, configured to train a sight line estimation model using feature data of a training image to obtain the trained sight line estimation model; wherein, during training, the loss of the sight line estimation model determined according to a preset loss function is used to update the model parameters of the sight line estimation model; The preset loss function is determined based on the regression loss function and the cross entropy loss function.

13. A gaze area recognition device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 6 when executing the instructions.

14. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

15. A computer program product, characterized in that The invention comprises a computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the method according to any one of claims 1 to 6.

Images