Wearable eye movement and facial expression synchronous acquisition device and method
By designing a wearable eye-tracking and facial expression synchronization acquisition device, and combining infrared illumination and efficient algorithms, the problems of unclear pupil recognition and non-strict synchronization acquisition in existing technologies have been solved, achieving accurate pupil recognition and image synchronization, and supporting applications in multiple fields.
Patent Information
- Application Number
- CN202110640110.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-09
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-06-09
AI Technical Summary
Existing eye-tracking and facial expression capture systems cannot clearly identify pupil edges, accurately identify pupil areas and mark fixation points, and cannot strictly synchronize eye-tracking and facial expression images.
A wearable eye-tracking and facial expression synchronization acquisition device was designed, including a synchronization acquisition system module, an ear-hook bracket module, an image acquisition module, and a recognition algorithm module. It adopts a 3D printed bracket and infrared illumination technology, combined with Gaussian blur algorithm, adaptive threshold edge detection and AVBFSC algorithm to recognize the pupil radius and center point coordinates, and achieves microsecond-level synchronization through a real-time operating system.
It achieves precise identification of the pupil region and marking of the gaze point, ensuring strict synchronization between eye movement images and facial expression images, and supports applications such as affective computing, cognitive research, diagnosis of mental disorders, and monitoring of driver fatigue.
Smart Images

Figure CN113269122B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of facial expression acquisition, and in particular to a wearable eye movement and facial expression synchronous acquisition device and method. Background Art
[0002] Eye movement and expression, as a spontaneous behavior of people, can directly or indirectly reflect the current psychological state of the subject. As an important measuring instrument for eye movement behavior, eye movement analyzer is often used in the field of psychology. Its application in psychological experiments is mainly to infer the cognitive activities reflected by the accurate description of eye movement. Among them, human eye movement behavior is mainly described from three aspects: fixation point, that is, the human eye stops and looks at a certain point; saccade, that is, the movement process of the human eye from one fixation point to the next; pupil size. This non-invasive device can easily collect data and is easy to operate. Expressions are fast and difficult to detect, and can express real emotions and intentions. The fusion analysis of expressions and eye movements can capture the modal representation and cross-modal complementary correlation of multimodal data. The time synchronization of fusion analysis is very important, because human eye movement behavior can be completed at the millisecond level, and expression changes are weak and rapid. Therefore, accurate time synchronization is the prerequisite for multimodal fusion analysis.
[0003] Eye movement and facial expression fusion analysis is gradually being applied in various fields. In fatigue detection, by observing the driver's eye movement behavior and facial expression changes, it is possible to determine whether he is in a state of fatigue; in clinical practice, eye trackers can be used as a diagnostic tool to diagnose corresponding mental illnesses by testing the patient's eye movement function; in the diagnosis and research of mental disorders such as anxiety, depression, and autism, researchers use the subject's eye movement indicators and facial expression changes under psychological stimulation paradigms or cognitive tasks to make an objective and quantitative assessment of their mental state. Existing eye movement and facial expression acquisition systems often cannot clearly identify the pupil edge, accurately identify the pupil area and mark the gaze point, and cannot strictly synchronize the acquisition of eye movement images and facial expression images. Summary of the invention
[0004] The present application provides a wearable eye movement and facial expression synchronous acquisition device, system and algorithm to solve the problem that the existing eye movement and facial expression acquisition systems are often unable to clearly identify the pupil edge, accurately identify the pupil area and mark the gaze point, and are unable to strictly synchronize the acquisition of eye movement images and expression images.
[0005] On the one hand, the present application provides a wearable eye movement and facial expression synchronous acquisition device, including a synchronous acquisition system module, an ear-hook bracket module, an image acquisition module and a recognition algorithm module, wherein:
[0006] The synchronous acquisition system module is connected to the ear-hook bracket module via a USB interface, the ear-hook bracket module is connected to the image acquisition module via a USB interface, and the image acquisition module and the recognition algorithm module are respectively connected to the synchronous acquisition system module via signals;
[0007] The synchronous acquisition system module is configured to synchronously acquire expression images and eye movement images in real time;
[0008] The ear-hook bracket module is configured to mount an eye-tracking camera and a foreground camera, and includes an ear-hook bracket, an extension arm, a spherical connection arm, a foreground camera, and a pupil camera;
[0009] The number of the extension arm, the spherical connecting arm and the pupil camera is two; the ear-hook bracket is connected to the extension arm via a sawtooth-shaped slide buckle structure provided on the inner side; one end of the spherical connecting arm is connected to the extension arm, and the other end of the spherical connecting arm is connected to the pupil camera; one side of the ear-hook bracket is connected to the foreground camera;
[0010] The image acquisition module is configured to acquire the expression image, eye movement image and foreground image of the subject;
[0011] The recognition algorithm module is configured to calculate the radius size and center point position of the pupil based on image denoising, edge detection and pupil segmentation algorithms.
[0012] Optionally, the synchronous acquisition system module includes a real-time process sub-module and a non-real-time process sub-module, adopts a memory file system, and applies a real-time synchronization mechanism. The synchronous acquisition system module is also connected to a process preloading unit, a real-time synchronization signal unit and a multi-processor unit.
[0013] Optionally, the 3D printed bracket is made of elastic nylon material, including an ear-hook bracket, an extension arm and a rear bracket, the ear-hook bracket includes an ear-hook structure, an extension arm and a balancing device, the number of the ear-hook structures and the extension arms are two, the two ear-hook structures are directly connected to the two ends of the balancing device, the connection between the ear-hook structure and the balancing device is also connected to an extension arm, and the extension arm is connected to the extension arm through a serrated slide buckle structure provided on the inner side.
[0014] Optionally, the rear section of the left extension arm is provided with an empty slot, and a hub is provided in the empty slot.
[0015] Optionally, the extension arm includes a first spherical interface and a plug, one end of the first spherical interface is directly connected to one end of the plug, the other end of the plug is provided with two protrusions, a serrated slide buckle structure connected to the extension arm, the other end of the first spherical interface is connected to the large end of the spherical connecting arm, and the connection between the two can adjust the direction of the spherical connecting arm by rotation.
[0016] Optionally, the spherical connecting arm includes a circular tube, a first spherical joint and a second spherical joint, the first spherical joint and the second spherical joint are respectively connected to two ends of the circular tube, the first spherical joint is larger than the second spherical joint, the first spherical joint is connected to an extension arm, and the second spherical joint is connected to a pupil camera.
[0017] Optionally, the foreground camera is also provided with a foreground camera housing, and the foreground camera housing is connected to the foreground camera via a third spherical joint.
[0018] Optionally, each pupil camera is also provided with a pupil camera housing, and the pupil camera housing is connected to the second spherical joint via a second spherical interface.
[0019] Optionally, the pupil camera housing is further provided with two notches, in which lens modules and infrared fill light modules are provided.
[0020] Optionally, the image acquisition module includes an expression camera submodule, an eye movement camera submodule and a foreground camera submodule, and the expression camera submodule, the eye movement camera submodule and the foreground camera submodule are respectively connected to an expression image acquisition unit, an eye movement image acquisition unit and a foreground image acquisition unit, and the eye movement image acquisition unit is also connected to an infrared fill light processing unit and a filtering processing unit.
[0021] Optionally, the recognition algorithm module includes an image denoising technology submodule, a corrosion edge detection submodule and a pupil segmentation submodule. The image denoising technology submodule is based on a Gaussian blur algorithm, which is used to suppress noise in the image, smooth the image, and retain the pupil edge; the corrosion edge detection submodule filters redundant edge branches based on an adaptive threshold edge detection algorithm; and the pupil segmentation submodule segments the pupil based on an AVBFSC algorithm.
[0022] On the other hand, the present application provides an algorithm for identifying pupil radius size and center point coordinates, including:
[0023] The original eye movement image collected by the eye movement camera submodule is blurred based on the Gaussian blur algorithm to suppress image noise and obtain a denoised image;
[0024] Optimizing the edge of the denoised image based on an adaptive threshold method to obtain an edge-optimized image;
[0025] The pupil in the optimized image is segmented based on the AVBFSC algorithm to determine the pupil radius and center point coordinates.
[0026] Optionally, the step of blurring the original eye movement image collected by the eye movement camera submodule based on a Gaussian blur algorithm includes:
[0027] The original eye movement image collected by the eye movement camera submodule is blurred by the Gaussian blur algorithm to suppress image noise, smooth the image, and obtain a denoised image.
[0028] Optionally, the step of optimizing the edge of the denoised image based on an adaptive threshold method includes:
[0029] Set adaptive threshold: take the average pixel intensity of each area of the denoised image as the threshold Thread;
[0030] Image binarization processing: Use Thread to convert the denoised image into a binary digital image. The formula is as follows:
[0031]
[0032] Erosion optimization: Use a structural element of fixed size to scan each pixel of the binary digital image, and perform an "AND" operation on the structural element and the binary digital image it covers. If both are 1, the pixel of the structural image is 1, otherwise it is 0. Filter out redundant edge branches to obtain an edge-optimized image.
[0033] Optionally, the step of segmenting the pupil in the optimized image based on the AVBFSC algorithm includes:
[0034] Input the edge optimized image, the pixel intensity of the background is 0, and the pixel intensity of the image target is 1; use CHT to calculate the rough pupil center (Ox, Oy) and radius Or, select the area with (Ox, Oy) as the initial center and Or±δ*Or as the radius and the pixel points with intensity 1 in the area;
[0035] Set the total number of random searches N, randomly search three non-collinear pixels, perform outlier detection on the selected pixels, and use them to fit the circle model if they are not outliers;
[0036] Substitute all non-outlier pixels into the circle model and calculate the distance to the edge of the circle. If the distance is less than a given threshold, these pixels are regarded as inliers and the number of inliers is calculated.
[0037] If outliers appear among the non-collinear pixels, give up fitting the circular model and repeat the above steps until the search times reaches N and the iterative search is stopped. The parameters of the circular model with the largest number of inner points are returned. Finally, the pupil area, radius size and center point coordinates in the eye movement image are determined.
[0038] It can be seen from the above technical solutions that the present application provides a wearable eye movement and facial expression synchronous acquisition device, system and algorithm, including a wearable eye movement and facial expression synchronous acquisition device and a pupil radius size and center point coordinate recognition algorithm, wherein the recognition algorithm module in the wearable eye movement and facial expression synchronous acquisition device includes an image denoising technology submodule, a corrosion edge detection submodule and a pupil segmentation submodule, and the pupil radius size and center point coordinate recognition algorithm includes a step of blurring the original eye movement image collected by the eye movement camera submodule, a step of optimizing the edge of the denoised image, and a step of segmenting the pupil in the optimized image. The image denoising technology submodule runs the step of blurring the original eye movement image collected by the eye movement camera submodule, the corrosion edge detection submodule runs the step of optimizing the edge of the denoised image, and the pupil segmentation submodule runs the step of segmenting the pupil in the optimized image.
[0039] This application innovatively designs the structure of the acquisition equipment and the camera hardware, which avoids occlusion of key facial areas; introduces infrared fill light and filtering technology into the eye movement camera to make the pupil edge clearer; designs a new pupil recognition algorithm AVBFSC, which can more accurately identify the pupil area and mark the gaze point; based on the real-time operating system, designs a synchronization mechanism with microsecond delay to achieve strict synchronization of eye movement images and expression images; this application can provide equipment support for research in subjects such as affective computing, cognition, and psychology, and can also be used for applications such as diagnosis of mental disorders, fatigue driving monitoring, and attention assessment, and has broad application value. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0041] Figure 1 A schematic diagram of the overall framework of a wearable eye movement and facial expression synchronous acquisition device, system and algorithm provided in this application;
[0042] Figure 2 A timing diagram of the synchronous acquisition mechanism of a wearable eye movement and facial expression synchronous acquisition device, system and algorithm provided in this application;
[0043] Figure 3 A schematic diagram of the structure of an ear-hook bracket module for a wearable eye movement and facial expression synchronous acquisition device, system and algorithm provided in this application;
[0044] Figure 4 A schematic diagram of the structure of an ear-hook bracket module for a wearable eye movement and facial expression synchronous acquisition device, system and algorithm provided in this application;
[0045] Figure 5 A schematic diagram of the structure of an extension arm of an ear-hook bracket module for a wearable eye movement and facial expression synchronous acquisition device, system and algorithm provided in the present application;
[0046] Figure 6 A schematic diagram of the structure of a spherical connecting arm of an ear-hook bracket module for a wearable eye movement and facial expression synchronous acquisition device, system and algorithm provided in the present application;
[0047] Figure 7 A schematic diagram of the structure of the foreground camera housing of an ear-hook bracket module of a wearable eye movement and facial expression synchronous acquisition device, system and algorithm provided in the present application;
[0048] Figure 8 A schematic diagram of the structure of the pupil camera housing of an ear-hook bracket module for a wearable eye movement and facial expression synchronous acquisition device, system and algorithm provided in this application;
[0049] Fig. 9 A schematic diagram of the pupil recognition algorithm flow of a wearable eye movement and facial expression synchronous acquisition device, system and algorithm provided in this application. DETAILED DESCRIPTION
[0050] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.
[0051] Facial expression recognition refers to separating specific expression states from a given static image or dynamic video sequence, thereby determining the psychological emotions of the identified object, realizing the computer's understanding and recognition of facial expressions, fundamentally changing the relationship between people and computers, and achieving better human-computer interaction. Therefore, facial expression recognition has great potential application value in the fields of psychology, intelligent robots, intelligent monitoring, virtual reality, and synthetic animation. Facial expression recognition technology has only gradually developed in recent decades. Due to the diversity and complexity of facial expressions, and involving physiology and psychology, expression recognition is more difficult. Therefore, compared with other biometric technologies such as fingerprint recognition, iris recognition, and face recognition, it has developed relatively slowly and is not widely used. However, expression recognition is of great value to human-computer interaction. Therefore, many research institutions and scholars at home and abroad are committed to research in this area and have achieved certain results.
[0052] See also Figure 1 , Figure 2 , Figure 3 and Fig. 9 , a wearable eye movement and facial expression synchronous acquisition device, comprising a synchronous acquisition system module, an ear-hook bracket module, an image acquisition module and a recognition algorithm module, wherein:
[0053] The synchronous acquisition system module is connected to the ear-hook bracket module via a USB interface, the ear-hook bracket module is connected to the image acquisition module via a USB interface, and the image acquisition module and the recognition algorithm module are respectively connected to the synchronous acquisition system module via signals;
[0054] The synchronous acquisition system module is configured to synchronously acquire expression images and eye movement images in real time;
[0055] The ear-hook bracket module includes a 3D printed bracket configured to mount an eye-tracking camera and a foreground camera;
[0056] The image acquisition module is configured to acquire the expression image, eye movement image and foreground image of the subject;
[0057] The recognition algorithm module is configured to calculate the radius size and center point position of the pupil based on image denoising, edge detection and pupil segmentation algorithms.
[0058] The synchronous acquisition system module includes a real-time process submodule and a non-real-time process submodule, adopts a memory file system, and applies a real-time synchronization mechanism. The synchronous acquisition system module is also connected to a process preloading unit, a real-time synchronization signal unit and a multi-processor unit.
[0059] The processes of the synchronous acquisition system are divided into two categories: real-time processes and non-real-time processes. This application designs a real-time synchronization mechanism based on a real-time operating system and a memory file system. It uses process preloading, real-time synchronization signals, and multi-processor binding of real-time processes to perform real-time synchronization of multiple acquisition tasks, achieving microsecond-level synchronization delays for facial expressions and eye movement images, and meeting the requirements for time synchronization for multi-modal fusion analysis in scenarios such as emotional computing and cognitive experiments.
[0060] The 3D printed bracket is made of elastic nylon material and includes an ear-hook bracket, an extension arm and a rear bracket.
[0061] The ear-hook bracket module is designed with a 3D printed structure adapted to the acquisition system, which is used to install the eye movement camera and the foreground camera. While collecting eye movements, this application also needs to capture facial expression images from the front of the subject. Therefore, this module has made structural innovations. Traditional wearable eye movement acquisition devices are mostly frame-type, which will block most of the eyebrows and eyes when worn. These areas are key areas for facial expression recognition. If blocked, it will affect the preprocessing and interest area positioning algorithm. In response to this problem, the application has designed an ear-hook structure to avoid blocking the eyebrows and eyes. It is made of 3D printing and is printed with nylon material, which has high fineness, high elasticity and low weight.
[0062] The image acquisition module includes an expression camera submodule, an eye movement camera submodule and a foreground camera submodule. The expression camera submodule, the eye movement camera submodule and the foreground camera submodule are respectively connected to an expression image acquisition unit, an eye movement image acquisition unit and a foreground image acquisition unit. The eye movement image acquisition unit is also connected to an infrared fill light processing unit and a filtering processing unit.
[0063] The image acquisition module includes an expression camera, an eye movement camera and a foreground camera. The eye movement camera uses infrared fill light and filtering technology to provide an additional light source for the high frame rate camera to distinguish the pupil and iris, and filters out the natural light that affects the eye movement image shooting through filtering technology. The eye movement camera is used to collect the pupil image of the subject, with two left and right cameras, with a resolution of 300×300 and a frame rate of 400 frames. The expression camera is used to collect the expression image of the subject from the front, using a desktop camera with a resolution of 1920×1080 and a frame rate of 30 frames. The foreground camera is used to simulate the subject's field of view image, on which the subject's gaze point can be determined, with a resolution of 1920×1080 and a frame rate of 30 frames; the four cameras are connected to the host computer via a USB cable.
[0064] The recognition algorithm module includes an image denoising technology submodule, a corrosion edge detection submodule and a pupil segmentation submodule. The recognition algorithm module consists of the following three steps: image denoising technology based on Gaussian blur algorithm, corrosion edge detection based on adaptive threshold, and pupil segmentation based on AVBFSC algorithm. It completes the recognition of pupil area and calculation of pupil radius and center point coordinates from the eye movement image, and marks the user's gaze point in the foreground image.
[0065] The image denoising technology submodule is based on the Gaussian blur algorithm. The Gaussian blur algorithm is applied to the image to blur the image, which can effectively suppress noise and smooth the image.
[0066] The corrosion edge detection submodule is based on an edge detection algorithm of an adaptive threshold, and its specific steps are as follows:
[0067] Set adaptive threshold: take the average value of pixel intensity of each area of the denoised image as the threshold Thread;
[0068] Image binarization processing: Use Thread to convert the denoised image into a binary digital image. The formula is as follows:
[0069]
[0070] Erosion optimization: Use a structural element of a certain size to scan each pixel of the binary digital image, and perform an "AND" operation on the structural element and the binary digital image it covers. If both are 1, the pixel of the structural image is 1, otherwise it is 0; finally, some redundant edge branches are filtered out.
[0071] The pupil segmentation submodule is based on the AVBFSC algorithm, and its specific steps are:
[0072] Input the edge optimized image, the pixel intensity of the background is 0, and the pixel intensity of the image target is 1; use CHT to calculate the rough pupil center (Ox, Oy) and radius Or, select the area with (Ox, Oy) as the initial center and Or±δ*Or as the radius and the pixel points with intensity 1 in the area;
[0073] Set the total number of random searches N, randomly search three non-collinear pixels, perform outlier detection on the selected pixels, and use these three points to fit a circle model if they are not outliers;
[0074] Then, all non-outlier pixels are substituted into the above circle model to calculate the distance to the edge of the circle. If the distance is less than a given threshold, these pixels are regarded as inliers and the number of inliers is calculated.
[0075] If outliers appear among these three pixels, give up fitting the circular model and repeat the above steps until the search times reaches N. Then, the iterative search is stopped and the parameters of the circular model with the largest number of inner points are returned. Finally, the pupil area, radius size and center point coordinates in the eye movement image are determined.
[0076] See also Figure 3The present invention proposes a wearable eye movement and facial expression synchronous acquisition device, which is connected to an ear-hook bracket module. The synchronous acquisition device is made of 3D printing and made of nylon, which has high precision, high elasticity and low mass. The main structure is improved to not block the facial expression interest area. The eye movement acquisition device consists of a 3D printed bracket and a camera. The bracket mainly consists of left and right ear-hook brackets, a rear bracket and an extension arm. The structure wears the device to the user's ear through the left and right ear-hook brackets. The eye movement camera extends forward to capture the eye area image. The foreground camera is located at the right temple position to capture the foreground image simulating the user's field of view. The rear bracket plays a balancing role. The main structure of the device and the camera part are both located at the front end of the device, and the wearing position is biased backward, requiring a structure to maintain the center of gravity. The extension arm part is used to adapt to different people, and the camera can be stretched out appropriately. This structure avoids the occlusion of important areas for expression recognition such as eyebrows and eyes by traditional frame structures. The structural design of the eyes avoids occluding key parts such as eyebrows, glasses, and mouth as much as possible. Therefore, the overall ear-hook structure is chosen, and the pupil camera bracket will stretch outward according to the different face shapes of the subjects when worn, so that the camera shooting the pupil will not block the cheek part. At the same time, the camera direction is adjusted so that the camera can shoot the pupil diagonally upwards, which neither blocks the face nor can clearly capture the pupil image.
[0077] See also Figure 3 , is a structural schematic diagram of a wearable ear-hook pupil collection and gaze point prediction eye tracker provided by the present application. The main structure of the present invention is divided into an ear-hook bracket 1, an extension arm 2, a spherical connecting arm 3, a foreground camera 4 and two pupil cameras 5. The ear-hook bracket 1 is the main part, and various components are connected to it: the two extension arms 2 are connected to the bracket through a serrated groove buckle structure provided on the inner side of the bracket; the large end of the spherical connecting arm 3 is connected to the extension arm 2, and the small end is connected to the pupil camera 5; the foreground camera 4 is connected to the ear hook on the right side of the bracket through a spherical joint.
[0078] The present invention adopts 3D printing technology and is made of elastic nylon material and nylon material, and has strong elastic deformation ability and low density.
[0079] See also Figure 4 , which is a schematic structural diagram of an ear-hook bracket of a wearable ear-hook pupil collection and gaze point prediction eye tracker provided in the present application. The ear-hook bracket 1 structure of the present invention is composed of three parts: an upper ear-hook structure 6, a front extension arm 7 and a rear balancing device 8.
[0080] The wearer wears the eye movement acquisition device through the ear-hook structure 6, which fits the human ear structure and bends inward to fit the wearer's auricle. At the same time, in order to adapt to wearers with different face shapes and feel comfortable when wearing, a layer of silicone pad is attached to the inside of the ear-hook structure; the front extension arm 7 is connected to the extension arm 2, and a serrated slide buckle structure is designed on the inner side thereof, and a rectangular shell structure with an inner diameter of 30mm×15mm×7mm is opened at the rear section of the left extension arm 7 for installing a hub 9 for three cameras. The data cables of the three cameras are gathered along the inside of the bracket to the hub 9, and then connected to the host computer by a typeC-usb2.0 connection cable; the rear balancing device 8 is used to connect and balance the structures on both sides. Most of the weight of the present invention is concentrated in the front half, and a balancing device needs to be designed backward to achieve the purpose of stable wearing. The hollow design ensures the lightness of the overall structure and the balance between the left and right.
[0081] See also Figure 5 , which is a structural schematic diagram of an extension arm of a wearable ear-hook pupil collection and gaze point prediction eye tracker provided in the present application. The extension arm 2 of the present invention consists of two parts: a first spherical interface 10 at the front end and a rectangular plug 11 at the rear end.
[0082] Two protrusions are designed at the end of the rear end plug 11, and the serrated slot snap-in structure of the connecting bracket extension arm 7 can freely extend the extension arm 2 and fix it in a certain position at the same time; the first spherical interface 10 at the front end is connected to the first spherical joint 13 at the large end of the spherical connecting arm 3, and the connection between the two can adjust the direction of the spherical connecting arm 3 by rotation.
[0083] See also Figure 6 , which is a structural schematic diagram of a spherical connecting arm of a wearable ear-hook pupil collection and gaze point prediction eye tracker provided in the present application. The spherical connecting arm 3 of the present invention is composed of a middle circular tube 12 and two spherical joints, one large and one small. The large first spherical joint 13 is connected to the extension arm 2, and the small second spherical joint 14 is connected to the pupil camera 5.
[0084] The spherical connecting arm 3 and the extension arm 2 work together to adjust the shooting angle of the pupil camera 5 to accommodate most wearers.
[0085] See also Figure 7 , which is a structural schematic diagram of the foreground camera housing of a wearable ear-hook pupil collection and gaze point prediction eye tracker provided in this application, and this structure is used to install the foreground camera 4.
[0086] The main body of the foreground camera housing 15 is a rectangular parallelepiped with an inner diameter of 32mm×13mm×7mm, and a 1080p, 60Hz high-definition camera is installed inside. The connection part uses a third ball joint 16, which can adjust the shooting angle of the foreground camera 4 to adapt to the field of view of most wearers.
[0087] See also Figure 8 , which is a structural schematic diagram of the pupil camera shell of a wearable ear-hook pupil collection and gaze point prediction eye tracker provided in this application, and this structure is used to install two left and right pupil cameras 5.
[0088] The main body of the pupil camera housing 17 is also a rectangular parallelepiped with an inner diameter of 53mm×10mm×5mm. A high-definition camera with a resolution of 300×300 and a frequency of 400Hz is installed inside the housing for capturing pupil images. The pupil camera housing 17 is connected to the second spherical joint 14 of the spherical connecting arm 3 via the second spherical interface 18. Two notches are opened on one side of the pupil camera housing 17, wherein the circular notch is used to install the lens module of the pupil camera 5, and the square notch is used to install the infrared fill light module. According to the different absorbances of the pupil and iris, an infrared fill light module is added to the camera module, and the wavelength of the infrared fill light is 850nm. At the same time, an 850nm narrow-band filter is installed on the camera to avoid interference from natural light, so as to finally record a clear pupil image.
[0089] See also Figure 1 , Figure 2 and Figure 3 , Figure 1 It is a schematic diagram of the overall system framework of the present invention, a wearable eye movement and facial expression synchronous acquisition system, which is divided into four modules, including a synchronous acquisition system module, an ear-hook bracket module, an image acquisition module, and a recognition algorithm module.
[0090] Synchronous acquisition system modules such as Figure 2 As shown, the acquisition part of the system needs to process the video stream in real time. The general operating system has large delay and jitter, which leads to clock drift and frame loss in the video recording process, resulting in the eye movement image and the expression image being out of sync on the time axis. The present invention is based on a real-time operating system and a memory file system, and designs a real-time synchronization mechanism. A preloading mechanism is adopted, and the main process is started first. The main process initializes relevant resources and starts the eye movement acquisition process, the expression acquisition process, the foreground camera process and the stimulation paradigm process (the stimulation paradigm is required in scenes such as emotional computing and cognitive experiments). After the sub-process is started, the process priority is increased respectively, and the synchronization signal of the main process is waited for. When the user instruction is issued, the main process sends a synchronization signal to each sub-process and starts the acquisition task. In order to reduce the delay, the collected image files are temporarily stored in the memory file system (RAMFS), and the pupil recognition program directly reads the eye movement image files from the memory file system to realize real-time pupil recognition and gaze point prediction. After the acquisition is completed, the eye movement image is encoded into a video file and written into the disk file system.
[0091] The image acquisition module is divided into three parts: foreground camera, expression camera and eye movement camera; the resolution of foreground camera and expression camera is 1920×1080, and the frame rate is 30 frames; the eye movement camera is used to collect eye movement images and the eye movement is completed within a few milliseconds, so the eye movement camera needs a high frame rate and additional infrared fill light; there are two left and right eye movement cameras, with a resolution of 300×300 and a frame rate of 400 frames; in order to enable the camera to clearly capture the pupil, 850nm infrared fill light is used, and in order to avoid interference caused by other light sources in the environment, the camera lens uses an 850nm narrow-band filter to filter other bands of visible light. In actual use scenarios, the ambient light may vary greatly. Too low or too high illumination will cause insufficient or excessive fill light. This module uses a fixed exposure mode. Through port mapping, the "gain" interface in the UVC protocol is changed to the infrared light brightness adjustment interface. By adjusting the "gain", the LED brightness change can be controlled. The port in the main control chip is connected to the MCU input, and the output PWM signal is adjusted by software to adjust the brightness of the infrared light and achieve stable picture brightness. The specific principle is as follows Figure 3 shown.
[0092] The recognition algorithm module refers to identifying and segmenting the pupil area from the original eye movement image, and calculating the pupil radius and center point coordinates. Pupil segmentation consists of three parts: image denoising, edge detection and pupil circle detection. The recognition algorithm only targets the pupil image, that is, the image collected by the pupil camera, identifies the pupil area in the image, and calculates the pupil radius, center point and other eye movement indicators. Finally, the gaze point is predicted and projected onto the image captured by the foreground camera (the foreground camera is used to simulate the wearer's field of view). The expression acquisition module does not use a recognition algorithm, but only collects and saves images through a real-time synchronization system. Because it is necessary to perform fusion analysis on the pupil image and the expression image, an expression acquisition module is added to the system of this application. The fusion analysis requires that the images correspond one to one on the time axis, so a real-time synchronization system is used.
[0093] The present application provides a wearable eye movement and facial expression synchronous acquisition device, system and algorithm, wherein the synchronous acquisition system module is connected to an ear-hook bracket module, an image acquisition module and a recognition algorithm module, wherein the synchronous acquisition system module includes a real-time process submodule and a non-real-time process submodule; the ear-hook bracket module includes a 3D printing bracket and a camera device; the image acquisition module includes an expression camera submodule, an eye movement camera submodule and a foreground camera submodule; the recognition algorithm module includes an image denoising technology submodule, a corrosion edge detection submodule and a pupil segmentation submodule.
[0094] This application innovatively designs the structure of the acquisition equipment and the camera hardware, which avoids occlusion of key facial areas; introduces infrared fill light and filtering technology into the eye movement camera to make the pupil edge clearer; designs a new pupil recognition algorithm AVBFSC, which can more accurately identify the pupil area and mark the gaze point; based on the real-time operating system, designs a synchronization mechanism with microsecond delay to achieve strict synchronization of eye movement images and expression images; this application can provide equipment support for research in subjects such as affective computing, cognition, and psychology, and can also be used for applications such as diagnosis of mental disorders, fatigue driving monitoring, and attention assessment, and has broad application value.
[0095] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. For those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present application is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application.
[0096] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
[0097] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.
Claims
1. A wearable device for synchronously collecting eye movements and facial expressions. It is characterized in that include: Synchronous acquisition system module, ear-hanging bracket module, image acquisition module and recognition algorithm module, wherein: The synchronous acquisition system module is connected to the ear-hook bracket module via a USB interface, the ear-hook bracket module is connected to the image acquisition module via a USB interface, and the image acquisition module and the recognition algorithm module are respectively connected to the synchronous acquisition system module via signals; The synchronous acquisition system module is configured to synchronously acquire expression images and eye movement images in real time; The ear-hook bracket module includes a 3D printed bracket configured to mount an eye-tracking camera and a foreground camera; The image acquisition module is configured to acquire the expression image, eye movement image and foreground image of the subject; The recognition algorithm module is configured to calculate the radius size and center point position of the pupil based on image denoising, edge detection and pupil segmentation algorithm; The recognition algorithm module includes an image denoising technology submodule, a corrosion edge detection submodule and a pupil segmentation submodule. The image denoising technology submodule is based on a Gaussian blur algorithm and is used to suppress noise in an image, smooth the image and retain the pupil edge; the corrosion edge detection submodule filters redundant edge branches based on an edge detection algorithm based on an adaptive threshold; and the pupil segmentation submodule segments the pupil based on an AVBFSC algorithm. The pupil segmentation submodule is configured as follows: Input the edge optimized image of the corrosion edge detection submodule, the pixel intensity of the background is 0, and the pixel intensity of the image target is 1; use CHT to calculate the rough pupil center (Ox, Oy) and radius Or, select the area with (Ox, Oy) as the initial center and Or±δ*Or as the radius and the pixel points with intensity 1 in the area; Set the total number of random searches N, randomly search three non-collinear pixels, perform outlier detection on the selected pixels, and use them to fit the circle model if they are not outliers; Substitute all non-outlier pixels into the circle model and calculate the distance to the edge of the circle. If the distance is less than a given threshold, these pixels are regarded as inliers and the number of inliers is calculated. If outliers appear among the non-collinear pixels, give up fitting the circular model and repeat the above steps until the search times reaches N and the iterative search is stopped. The parameters of the circular model with the largest number of inner points are returned. Finally, the pupil area, radius size and center point coordinates in the eye movement image are determined.
2. A wearable eye movement and facial expression synchronous acquisition device according to claim 1, It is characterized in that The synchronous acquisition system module includes a real-time process submodule and a non-real-time process submodule, adopts a memory file system, and applies a real-time synchronization mechanism. The synchronous acquisition system module is also connected to a process preloading unit, a real-time synchronization signal unit and a multi-processor unit.
3. A wearable eye movement and facial expression synchronous acquisition device according to claim 1, It is characterized in that The 3D printed bracket is made of elastic nylon material and includes an ear-hook bracket, an extension arm and a rear bracket.
4. A wearable eye movement and facial expression synchronous acquisition device according to claim 1, It is characterized in that The image acquisition module includes an expression camera submodule, an eye movement camera submodule and a foreground camera submodule. The expression camera submodule, the eye movement camera submodule and the foreground camera submodule are respectively connected to an expression image acquisition unit, an eye movement image acquisition unit and a foreground image acquisition unit. The eye movement image acquisition unit is also connected to an infrared fill light processing unit and a filtering processing unit.
5. A method for identifying pupil radius and center point coordinates, It is characterized in that A synchronous acquisition device according to any one of claims 1 to 4, comprising: The original eye movement image collected by the eye movement camera submodule is blurred based on the Gaussian blur algorithm to suppress image noise and obtain a denoised image; Optimizing the edge of the denoised image based on an adaptive threshold method to obtain an edge-optimized image; Segment the pupil in the optimized image based on the AVBFSC algorithm to determine the pupil radius and center point coordinates; The step of segmenting the pupil in the optimized image based on the AVBFSC algorithm comprises: Input the edge optimized image, the pixel intensity of the background is 0, and the pixel intensity of the image target is 1; use CHT to calculate the rough pupil center (Ox, Oy) and radius Or, select the area with (Ox, Oy) as the initial center and Or±δ*Or as the radius and the pixel points with intensity 1 in the area; Set the total number of random searches N, randomly search three non-collinear pixels, perform outlier detection on the selected pixels, and use them to fit the circle model if they are not outliers; Substitute all non-outlier pixels into the circle model and calculate the distance to the edge of the circle. If the distance is less than a given threshold, these pixels are regarded as inliers and the number of inliers is calculated. If outliers appear among the non-collinear pixels, give up fitting the circular model and repeat the above steps until the search times reaches N and the iterative search is stopped. The parameters of the circular model with the largest number of inner points are returned. Finally, the pupil area, radius size and center point coordinates in the eye movement image are determined.
6. A method for identifying pupil radius and center point coordinates according to claim 5, It is characterized in that The step of blurring the original eye movement image collected by the eye movement camera submodule based on the Gaussian blur algorithm comprises: The original eye movement image collected by the eye movement camera submodule is blurred by the Gaussian blur algorithm to suppress image noise, smooth the image, and obtain a denoised image.
7. A method for identifying pupil radius and center point coordinates according to claim 5, It is characterized in that The step of optimizing the edge of the denoised image based on the adaptive threshold method comprises: Set adaptive threshold: take the average pixel intensity of each area of the denoised image as the threshold Thread; Image binarization processing: Use Thread to convert the denoised image into a binary digital image. The formula is as follows: Erosion optimization: Use a structural element of fixed size to scan each pixel of the binary digital image, and perform an "AND" operation on the structural element and the binary digital image it covers. If both are 1, the pixel of the structural image is 1, otherwise it is 0. Filter out redundant edge branches to obtain an edge-optimized image.
