Smart glasses for eye tracking interaction and method thereof

Abstract

The application discloses an eye movement tracking interaction intelligent glasses and a method thereof, and belongs to the technical field of intelligent glasses, and comprises an intelligent interaction glasses body, left and right lenses are arranged on the intelligent interaction glasses body and are symmetrically distributed, an information acquisition module and an interaction server are arranged on the intelligent interaction glasses body, and the information acquisition module and the interaction server perform data interaction through a wireless network. In order to solve the problems that the user still needs to use external equipment and operate with hands, cannot free the hands, the interaction is inconvenient, and the intelligent glasses service is inconvenient, the eye movement tracking interaction intelligent glasses and the method thereof are used for intelligent interaction by using the eye movement state of the user, the intelligent interaction can be realized without the help of external equipment, the intelligent glasses service is more convenient, the eye movement tracking intelligent interaction is carried out according to different response strategies, the intelligent interaction is convenient, and the hands are free.

Description

Technical Field

[0001] This invention relates to the field of smart glasses technology, and in particular to a smart glasses and method for eye-tracking interaction. Background Technology

[0002] Smart glasses devices, including AR and MR glasses, offer interaction methods that can improve the efficiency of human interaction with the glasses and reduce the intensity of operation.

[0003] Chinese Patent Publication No. CN111475017A discloses a smart glasses device and a human-computer interaction method. The smart glasses device includes: smart interactive glasses, an eye-tracking camera, a server, a touch keyboard glasses case, and a touch button ring. The eye-tracking camera and the server are mounted on the smart interactive glasses. The server is connected to both the smart interactive glasses and the eye-tracking camera. The server is wirelessly connected to the touch keyboard glasses case and the touch button ring. Buttons are provided on the inner wall of the touch keyboard glasses case. The inner walls of the touch keyboard glasses case with buttons are hinged together. The touch keyboard glasses case is used to input information into the smart interactive glasses. This allows users to interact with the smart glasses device naturally, efficiently, and accurately without excessive physical movement that could cause fatigue.

[0004] Chinese patent CN103336575B discloses a human-computer interaction smart glasses system, comprising: smart glasses, which are see-through smart glasses, allowing visible light to pass through the lenses of the smart glasses, while simultaneously superimposing image information onto the user's actual field of vision; two cameras and an infrared LED are installed on the smart glasses, with the two cameras symmetrically mounted on the left and right ends of the smart glasses, and the infrared LED installed at the center of the smart glasses; the two cameras and the infrared LED of the smart glasses form a three-dimensional motion capture system for capturing the motion trajectory and coordinates of objects within a certain three-dimensional space. It captures and judges the user's finger movement information to achieve human-computer interaction with the smart glasses. However, the above patent has the following drawbacks:

[0005] When using smart glasses for data interaction, external devices and user hand operation are still required, which cannot free the user's hands, making the interaction inconvenient and the smart glasses service inconvenient. Summary of the Invention

[0006] The purpose of this invention is to provide an eye-tracking interactive smart glasses and method, which utilizes the user's eye movement state for intelligent interaction, achieving intelligent interaction without the need for external devices, making smart glasses services more convenient. It performs eye-tracking intelligent interaction according to different response strategies, making intelligent interaction convenient and freeing up the user's hands, thereby solving the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An eye-tracking interactive smart glasses includes a smart interactive glasses body with a left lens and a right lens symmetrically distributed on the smart interactive glasses body. The smart interactive glasses body is equipped with an information acquisition module and an interaction server. The information acquisition module interacts with the interaction server via a wireless network.

[0009] Furthermore, the information acquisition module includes an inertial measurement unit, a high-definition acquisition camera, a ranging sensor, and a timing sensor, wherein...

[0010] The inertial measurement unit is installed on the smart interactive glasses and is used to measure the three-axis attitude angle and acceleration of the user's eyeball relative to the user's head. The inertial measurement unit includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometers are used to detect the acceleration signals of the user's eyeball in the independent three axes of the carrier coordinate system during the movement. The gyroscopes are used to detect the angular velocity signals of the user's eyeball relative to the navigation coordinate system during the movement. The angular velocity and acceleration of the user's eyeball in three-dimensional space are measured during the movement, and the attitude of the user's eyeball during the movement is calculated accordingly.

[0011] The high-definition acquisition cameras are two in number and symmetrically mounted on the smart interactive glasses. The two high-definition acquisition cameras are located above the left and right lenses, respectively, and are used to capture the user's eye movements in real time and sense the user's eye movement state.

[0012] The ranging sensor is installed in the middle of the smart interactive glasses. The ranging sensor is a laser ranging sensor and is used to measure the straight-line distance from the user's head to an obstacle directly in front of them.

[0013] The timing sensor is installed on the smart interactive glasses and is used to time the user's eye movements and viewing time.

[0014] Furthermore, the interactive server has a built-in information processing module, an information analysis module, and an interactive execution module, wherein...

[0015] The information processing module is used to process the eye-tracking data collected in real time by the information acquisition module. First, it extracts features from the real-time eye-tracking data and automatically acquires the eye-tracking data. Then, it identifies and judges the acquired eye-tracking data and generates corresponding three-dimensional coordinates, finally forming an eye-tracking coordinate system. Finally, it transmits the processed eye-tracking data to the information analysis module.

[0016] The information analysis module is used to analyze the eye-tracking data processed in real time by the information processing module. First, it finds the ROI area that the user wants to interact with based on the formed eye-tracking coordinate system. Based on the user's eye-tracking viewing time recorded by the timing sensor and referring to the access interaction threshold stored in the access storage unit, it automatically analyzes the ROI area that the user wants to interact with to determine whether the user needs to interact and feeds back the analysis results to the interaction execution module.

[0017] The interactive execution module is used to interactively execute the analysis results transmitted in real time by the information analysis module. First, it reads the analysis results of the ROI area that the user wants to interact with. After reading the analysis results, it intelligently executes different response strategies according to different analysis results, and performs intelligent interaction with eye tracking according to different response strategies.

[0018] Furthermore, the information processing module processes the real-time acquired eye-tracking data and performs the following operations:

[0019] After the information acquisition module collects eye movement data in real time, it transmits the real-time eye movement data to the feature extraction unit. The feature extraction unit uses the real-time collected three-axis attitude angle, acceleration, user eye movement state, straight-line distance from the user's head to the obstacle in front, and user eye movement viewing time to extract useful user eye movement data, and transmits the extracted useful user eye movement data to the recognition and judgment unit.

[0020] After receiving the extracted data from the feature extraction unit, the recognition and judgment unit uses the recognition and judgment unit to identify and judge the extracted useful user eye movement data, find the interrelationships between the user eye movement data, and generate corresponding three-dimensional coordinates based on the three-axis attitude angle, acceleration and user eye movement state, and finally form an eye movement tracking coordinate system.

[0021] Furthermore, the information analysis module analyzes the eye-tracking data and performs the following operations:

[0022] The identification and judgment unit transmits the final eye-tracking coordinate system data to the region search unit. The region search unit uses rectangles, circles, ellipses, irregular polygons, etc., to delineate the region that the user wants to interact with from the received eye-tracking coordinate system data, finds the ROI region that the user wants to interact with, and transmits the found ROI region that the user wants to interact with to the automatic analysis unit.

[0023] After receiving the ROI area data transmitted by the identification and judgment unit, the automatic analysis unit automatically analyzes the ROI area that the user wants to interact with and determines whether the user needs to perform intelligent interaction.

[0024] When performing automated analysis of the ROI region, the automatic analysis unit sends instructions to the timing sensor, records the user's eye movement viewing time in real time based on the timing sensor, and transmits the real-time recorded user eye movement viewing time to the automatic analysis unit.

[0025] After receiving the user's eye movement viewing time transmitted by the timing sensor, the automatic analysis unit analyzes the user's eye movement viewing time recorded by the timing sensor and refers to the access interaction threshold stored in the access storage unit, and feeds back the analysis results to the interaction execution module.

[0026] Furthermore, the region search unit includes:

[0027] The sequence determination subunit is used to determine the eye trajectory based on the eye tracking coordinate system data, and determine the coordinate value of each eye trajectory point. Based on the point sequence of the eye trajectory points in the eye trajectory, the horizontal coordinate value of each eye trajectory point is sorted to obtain the horizontal coordinate sequence. At the same time, based on the point sequence, the vertical coordinate value of each eye trajectory point is sorted to obtain the vertical coordinate sequence.

[0028] The first determining subunit is used to determine the difference between each horizontal coordinate value in the horizontal coordinate sequence and the corresponding previous horizontal coordinate value, to calculate the first total number of positive numbers and the second total number of negative numbers contained in the horizontal coordinate difference, and to use the ratio of the smaller of the first total number and the second total number to the first total number of eye movement trajectory points contained in the eye movement trajectory as the first round-trip coefficient of the eye movement trajectory.

[0029] The second determining subunit is used to determine the difference between each ordinate value in the vertical coordinate sequence and the corresponding previous ordinate value, to calculate the third total number of positive numbers and the fourth total number of negative numbers contained in the ordinate difference, and to use the ratio of the smaller of the third total number and the fourth total number to the total number of the first eye movement trajectory points as the second round-trip coefficient of the eye movement trajectory.

[0030] The weight determination subunit is used to take the sum of the first round-trip coefficient and the second round-trip coefficient as the comprehensive round-trip coefficient of the eye track, take the ratio of the point ordinal number of the eye track point in the eye track to the total number of first eye track points included in the eye track as the tracking weight of the eye track point, and determine the interest weight of the eye track point based on the tracking weight and the comprehensive round-trip coefficient.

[0031] The region determination subunit is used to determine the tracking region of the eye track point with the eye track point as the center of the field of vision and based on the corresponding interest weight, and to summarize the tracking regions corresponding to all eye track points to obtain the corresponding comprehensive tracking region.

[0032] The final determination subunit is used to determine the matching degree between the comprehensive tracking area and the preset shapes contained in the preset shape list. Based on the preset shape corresponding to the maximum matching degree, the minimum area containing the comprehensive tracking area is determined. The minimum area is taken as the ROI area that the user wants to interact with and access, and the determined ROI area that the user wants to interact with and access is transmitted to the automatic analysis unit.

[0033] The preset shape list includes rectangles, circles, ellipses, and irregular polygons.

[0034] Further, the automatic analysis unit is characterized by comprising:

[0035] The region reading subunit is used to read information from the ROI region and obtain the information of interest and the image of interest contained in the ROI region.

[0036] The instruction reading subunit is used to obtain the latest interaction instruction, retrieve the interaction ROI area corresponding to the latest interaction instruction, read information from the interaction ROI area, and obtain the latest interaction information.

[0037] The vocabulary sorting subunit is used to identify the first core vocabulary in the information of interest, sort the first core vocabulary according to the order of appearance of the first core vocabulary in the information of interest, and obtain a first core vocabulary sequence. At the same time, it identifies the second core vocabulary in the latest interaction information, sorts the second core vocabulary according to the order of appearance of the second core vocabulary in the latest interaction information, and obtains a second core vocabulary sequence.

[0038] The third determining subunit is used to determine the degree of correlation between the information of interest and the latest interaction information based on the first core vocabulary sequence and the second core vocabulary sequence, determine the predicted degree of interest of the interest shape based on the preset interest shape list, and determine the first interaction degree coefficient based on the predicted degree of interest and the degree of correlation.

[0039] The trajectory segmentation subunit is used to retrieve the corresponding eye movement trajectory based on the ROI region, determine the inflection point contained in the eye movement trajectory, and divide the eye movement trajectory into multiple eye movement sub-trajectories based on the inflection point.

[0040] The coefficient determination subunit is used to determine the trajectory vibration coefficient of the eye movement trajectory based on the ratio of the total number of inflection points in the eye movement trajectory to the total number of the first eye movement trajectory points, and to determine the sub-interaction degree coefficient of the eye movement sub-trajectory based on the ratio of the total number of the second eye movement trajectory points in the eye movement sub-trajectory to the total number of the first eye movement trajectory points.

[0041] The fourth determining subunit is used to determine the second interaction degree coefficient based on the maximum sub-interaction degree coefficient and the trajectory vibration coefficient;

[0042] The final judgment subunit is used to determine a comprehensive interaction coefficient based on the first interaction degree coefficient and the second interaction degree coefficient. When the comprehensive interaction coefficient is greater than the interaction coefficient threshold, it is determined that the user needs to perform intelligent interaction; otherwise, it is determined that the user does not need to perform intelligent interaction.

[0043] Furthermore, the interactive execution module executes the analysis results and performs the following operations:

[0044] The automatic analysis unit transmits the final analysis results to the information reading unit, which then reads the analysis results of the ROI area that the user wants to interact with and transmits the information reading results to the strategy execution unit.

[0045] After receiving the information reading results transmitted by the information reading unit, the strategy execution unit intelligently executes different response strategies for different analysis results, and performs intelligent eye-tracking interaction based on different response strategies.

[0046] Furthermore, the policy execution unit performs the following operations:

[0047] If the analysis result shows that the user's eye movement viewing time value recorded by the timing sensor does not exceed the access interaction threshold stored in the access storage unit, the strategy execution unit will execute the first-level response strategy based on the analysis result. At this time, the user's viewing page does not change, so the user does not interact with the viewing page.

[0048] If the analysis result shows that the user's eye-tracking viewing time value recorded by the timing sensor exceeds the access interaction threshold stored in the access storage unit, the strategy execution unit will execute a secondary response strategy based on the analysis result. At this time, the user's viewing page will change, enabling the user to perform eye-tracking intelligent interaction with the viewing page.

[0049] According to another aspect of the present invention, an eye-tracking interaction method for smart glasses is provided, comprising the following steps:

[0050] S10: The user wears smart glasses, and the information acquisition module collects the user's three-axis attitude angle and acceleration in real time, captures the user's eye movement in real time and senses the user's eye movement state, measures the straight-line distance from the user's head to the obstacle in front of them and records the user's eye movement viewing time. After the above eye movement data information is collected, the eye movement data information collected in real time is transmitted to the information processing module.

[0051] S20: The information processing module performs feature extraction on the real-time collected eye movement data information, extracts useful user eye movement data, identifies and judges the extracted useful user eye movement data, finds the interrelationship between user eye movement data, generates corresponding three-dimensional coordinates based on three-axis attitude angles, acceleration and user eye movement state, and finally forms an eye movement tracking coordinate system, and transmits the eye movement tracking coordinate system data to the information analysis module.

[0052] S30: The information analysis module delineates the area that the user wants to interact with from the received eye-tracking coordinate system data using squares, circles, ellipses, irregular polygons, etc., finds the ROI area that the user wants to interact with, automatically analyzes the ROI area that the user wants to interact with based on the user's eye-tracking viewing time recorded by the timing sensor and with reference to the access interaction threshold stored in the access storage unit, determines whether the user needs to perform intelligent interaction, and feeds back the analysis results to the interaction execution module;

[0053] S40: The interaction execution module reads the analysis results of the ROI area that the user wants to interact with, and intelligently executes different response strategies for different analysis results, and performs intelligent eye-tracking interaction based on different response strategies.

[0054] Compared with the prior art, the beneficial effects of the present invention are:

[0055] 1. The present invention relates to an eye-tracking interactive smart glasses and method. The user wears the smart glasses, and an information acquisition module collects the user's three-axis attitude angles and acceleration in real time. It also captures real-time images of the user's eye movements and senses the user's eye movement state, measures the straight-line distance from the user's head to an obstacle in front of them, and records the user's eye movement viewing time. After the eye movement data is collected, an information processing module extracts features from the real-time collected eye movement data, extracts useful user eye movement data, identifies and judges the extracted useful user eye movement data, finds the interrelationships between user eye movement data, and generates corresponding three-dimensional coordinates based on the three-axis attitude angles, acceleration, and user eye movement state. This ultimately forms an eye-tracking coordinate system, facilitating understanding of the user's eye movement state and enabling intelligent interaction using the user's eye movement state. Intelligent interaction can be achieved without the aid of external devices, making smart glasses services more convenient.

[0056] 2. The eye-tracking interactive smart glasses and method of the present invention, wherein the information analysis module delineates the area that the user wants to interact with from the received eye-tracking coordinate system data in the form of squares, circles, ellipses, irregular polygons, etc., finds the ROI area that the user wants to interact with, and automatically analyzes the ROI area that the user wants to interact with based on the user's eye-tracking viewing time recorded by the timing sensor 5 and with reference to the access interaction threshold stored in the access storage unit, and determines whether the user needs to perform smart interaction. The interaction execution module reads the analysis results of the ROI area that the user wants to interact with accordingly, and intelligently executes different response strategies for different analysis results, and performs eye-tracking smart interaction according to different response strategies, making smart interaction convenient and freeing up the hands.

[0057] 3. The eye-tracking interactive smart glasses and method of the present invention determine the corresponding coordinate value sequence through the sequence determination subunit included in the region search unit, providing an important data foundation for subsequently determining the comprehensive round-trip coefficient of the eye track; based on the first determination subunit and the second determination subunit, the comprehensive round-trip coefficient representing the number of round trips in the eye track is determined based on the number of positive and negative numbers in the difference between each horizontal coordinate and the previous horizontal coordinate, and the number of positive and negative numbers in the difference between each vertical coordinate and the previous vertical coordinate, providing a basis for accurately determining the user's region of interest; based on the weight determination subunit, the interest weight of the eye tracking point is determined by considering the influence of the number of points in the eye track on the region determination and the coefficient of the number of round trips in the eye track; based on the region determination subunit and the final determination subunit, the region of interest that the user wants to interact with is accurately divided based on the interest weight, realizing the determination of the region division weight based on the tracking analysis of the eye track, making the divided region of interest closer to the user's interaction needs.

[0058] 4. The eye-tracking interactive smart glasses and method of the present invention, through the region reading subunit and instruction reading subunit included in the automatic analysis unit, realizes the reading of information and graphics contained in the currently determined region of interest and the information and graphics contained in the region of interest corresponding to the last executed interaction instruction. This provides an important information foundation for subsequently determining the correlation between the current region of interest and the region of interest corresponding to the last executed interaction instruction. Then, based on the word sorting subunit and the third determination subunit, it realizes the determination of the degree coefficient representing the user's desire to interact by comprehensively determining the correlation between the currently determined region of interest and the region of interest corresponding to the last executed interaction instruction and the predicted interest weight of the interest graphics. Based on the trajectory division subunit, coefficient determination subunit, and fourth determination subunit, it realizes the determination of the second interaction degree coefficient by comprehensively considering the inflection point statistics of the eye movement trajectory and the influence of the longest eye movement sub-trajectory without inflection point. Finally, by comparing the first interaction degree coefficient and the second interaction degree coefficient with the interaction degree coefficient threshold through the final judgment subunit, it determines whether the user needs to perform intelligent interaction. This realizes the determination of the user's interaction needs by comprehensively considering factors such as the number of times the user travels back and forth in the region of interest and the length of the trajectory of continuous gaze. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the eye-tracking interactive smart glasses of the present invention;

[0060] Figure 2 This is a schematic diagram of the smart glasses with eye-tracking interaction according to the present invention;

[0061] Figure 3 This is a block diagram of the eye-tracking interactive smart glasses of the present invention;

[0062] Figure 4 This is an algorithm diagram showing the execution analysis results of the interactive execution module of the present invention.

[0063] Figure 5 This is a flowchart of the eye-tracking interaction method of the present invention;

[0064] Figure 6 This is a schematic diagram of the region search unit of the present invention;

[0065] Figure 7 This is a schematic diagram of the automatic analysis unit of the present invention.

[0066] In the diagram: 1. Smart interactive glasses; 11. Left lens; 12. Right lens; 2. Inertial measurement unit; 3. High-definition acquisition camera; 4. Distance sensor; 5. Timing sensor. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Example 1

[0069] See Figures 1-3 A smart glasses with eye-tracking interaction includes a smart interactive glasses body 1, on which a left lens 11 and a right lens 12 are disposed and symmetrically distributed.

[0070] The smart interactive glasses 1 are equipped with an information collection module and an interactive server. The information collection module interacts with the interactive server via a wireless network.

[0071] The information acquisition module includes an inertial measurement unit 2, a high-definition acquisition camera 3, a ranging sensor 4, and a timing sensor 5, among which...

[0072] An inertial measurement unit 2 is installed on the smart interactive glasses body 1, and the inertial measurement unit 2 is used to measure the three-axis attitude angle and acceleration of the user's eyeball relative to the user's head movement.

[0073] The inertial measurement unit 2 includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometers are used to detect the acceleration signals of the user's eyeballs on three independent axes of the carrier coordinate system during movement. The gyroscopes are used to detect the angular velocity signals of the user's eyeballs relative to the navigation coordinate system during movement. The angular velocity and acceleration of the user's eyeballs in three-dimensional space are measured during movement, and the attitude of the user's eyeballs during movement is calculated accordingly.

[0074] The inertial measurement unit (IMU) is a strapdown inertial navigation system. The system consists of three accelerometers and three angular velocity sensors. The accelerometers are used to sense the acceleration component of the user's eye movement relative to the vertical line, and the angular velocity sensors are used to sense the angle information of the user's eye movement. This sub-component mainly consists of two A / D converters AD7716BS and a 64K E / EPROM memory X25650. The A / D converters use the analog variables from the IMU's sensors, convert them into digital information, and then the CPU calculates and finally outputs the user's eye movement pitch and tilt angles.

[0075] Two high-definition acquisition cameras 3 are symmetrically installed on the smart interactive glasses body 1. The two high-definition acquisition cameras 3 are located above the left lens 11 and the right lens 12 respectively. The high-definition acquisition cameras 3 are used to capture the user's eye movement in real time and sense the user's eye movement state.

[0076] The distance sensor 4 is installed in the middle of the smart interactive glasses body 1. The distance sensor 4 is a laser distance sensor and is used to measure the straight distance from the user's head to the obstacle directly in front of him.

[0077] A timing sensor 5 is installed on the smart interactive glasses body 1. The timing sensor 5 is used to time the user's eye movement and viewing time.

[0078] Example 2

[0079] The interactive server has a built-in information processing module, information analysis module, and interactive execution module. The steps for data transmission between the information processing module, information analysis module, and interactive execution module are as follows:

[0080] S1: Extract features from real-time eye-tracking data and automatically acquire eye-tracking data. Identify and judge the acquired eye-tracking data and generate corresponding three-dimensional coordinates. Finally, form an eye-tracking coordinate system and transmit the processed eye-tracking data to the information analysis module.

[0081] It should be noted that the information processing module processes the real-time eye-tracking data in the following steps:

[0082] S11: After the information acquisition module collects eye movement data in real time, it transmits the real-time eye movement data to the feature extraction unit. The feature extraction unit uses the real-time collected three-axis attitude angle, acceleration, user eye movement state, straight-line distance from the user's head to the obstacle in front, and user eye movement viewing time to extract useful user eye movement data, and transmits the extracted useful user eye movement data to the recognition and judgment unit.

[0083] S12: After receiving the extracted data transmitted by the feature extraction unit, the identification and judgment unit uses the identification and judgment unit to identify and judge the extracted useful user eye movement data, find the interrelationship between the user eye movement data, generate corresponding three-dimensional coordinates based on the three-axis attitude angle, acceleration and user eye movement state, and finally form an eye movement tracking coordinate system.

[0084] S2: Based on the established eye-tracking coordinate system, find the ROI area that the user wants to interact with. Based on the user's eye-tracking viewing time recorded by the timing sensor 5 and referring to the access interaction threshold stored in the access storage unit, automatically analyze the ROI area that the user wants to interact with, determine whether the user needs to interact, and feed back the analysis results to the interaction execution module.

[0085] ROI is a type of IVE (Intelligent Video Encoding) technology. IVE technology can intelligently encode video according to customer requirements, optimizing video encoding performance without sacrificing image quality. This ultimately reduces network bandwidth usage and storage space. In surveillance footage, some areas are not needed for monitoring or are irrelevant, such as the sky, walls, and grass. Ordinary network surveillance cameras encode and transmit the entire area, putting pressure on network bandwidth and video storage. ROI intelligent video encoding technology effectively solves this problem. Cameras with ROI functionality allow users to select areas of interest in the frame. After enabling ROI, important or moving areas will be encoded with high-quality lossless encoding, while stationary or unselected areas will have their bitrate and image quality reduced, undergo standard definition video compression, or even be omitted from transmission, thus saving network bandwidth and video storage space.

[0086] It should be noted that the information analysis module analyzes eye-tracking data in the following steps:

[0087] S21: The recognition and judgment unit transmits the final eye-tracking coordinate system data to the region search unit. The region search unit uses squares, circles, ellipses, irregular polygons, etc., to delineate the region that the user wants to interact with from the received eye-tracking coordinate system data, finds the ROI region that the user wants to interact with, and transmits the found ROI region that the user wants to interact with to the automatic analysis unit.

[0088] S22: After receiving the ROI area data transmitted by the identification and judgment unit, the automatic analysis unit automatically analyzes the ROI area that the user wants to interact with and determines whether the user needs to perform intelligent interaction.

[0089] See Figure 6 The region search unit in S21 includes:

[0090] The sequence determination subunit is used to determine the eye trajectory based on the eye tracking coordinate system data, and determine the coordinate value of each eye trajectory point. Based on the point sequence of the eye trajectory points in the eye trajectory, the horizontal coordinate value of each eye trajectory point is sorted to obtain the horizontal coordinate sequence. At the same time, based on the point sequence, the vertical coordinate value of each eye trajectory point is sorted to obtain the vertical coordinate sequence.

[0091] The first determining subunit is used to determine the difference between each horizontal coordinate value in the horizontal coordinate sequence and the corresponding previous horizontal coordinate value, to calculate the first total number of positive numbers and the second total number of negative numbers contained in the horizontal coordinate difference, and to use the ratio of the smaller of the first total number and the second total number to the first total number of eye movement trajectory points contained in the eye movement trajectory as the first round-trip coefficient of the eye movement trajectory.

[0092] The second determining subunit is used to determine the difference between each ordinate value in the vertical coordinate sequence and the corresponding previous ordinate value, to calculate the third total number of positive numbers and the fourth total number of negative numbers contained in the ordinate difference, and to use the ratio of the smaller of the third total number and the fourth total number to the total number of the first eye movement trajectory points as the second round-trip coefficient of the eye movement trajectory.

[0093] The weight determination subunit is used to take the sum of the first round-trip coefficient and the second round-trip coefficient as the comprehensive round-trip coefficient of the eye track, take the ratio of the point ordinal number of the eye track point in the eye track to the total number of first eye track points included in the eye track as the tracking weight of the eye track point, and determine the interest weight of the eye track point based on the tracking weight and the comprehensive round-trip coefficient.

[0094] The region determination subunit is used to determine the tracking region of the eye track point with the eye track point as the center of the field of vision and based on the corresponding interest weight, and to summarize the tracking regions corresponding to all eye track points to obtain the corresponding comprehensive tracking region.

[0095] The final determination subunit is used to determine the matching degree between the comprehensive tracking area and the preset shapes contained in the preset shape list. Based on the preset shape corresponding to the maximum matching degree, the minimum area containing the comprehensive tracking area is determined. The minimum area is taken as the ROI area that the user wants to interact with and access, and the determined ROI area that the user wants to interact with and access is transmitted to the automatic analysis unit.

[0096] The preset shape list includes rectangles, circles, ellipses, and irregular polygons.

[0097] In this embodiment, the eye trajectory is the trajectory of the user's eye gaze point determined based on eye tracking coordinate system data.

[0098] In this embodiment, the horizontal coordinate sequence is the sequence of horizontal coordinates obtained by sorting the horizontal coordinate values ​​of each eye track point based on the point sequence of eye track points in the eye track.

[0099] In this embodiment, the eye-tracking point is the coordinate point contained in the eye-tracking trajectory.

[0100] In this embodiment, the vertical coordinate sequence is the sequence of horizontal coordinates obtained by sorting the vertical coordinate values ​​of each eye track point based on the point sequence of eye track points in the eye track.

[0101] In this embodiment, the first total number is the total number of positive numbers contained in the difference between the horizontal coordinate value and the corresponding previous horizontal coordinate value in the horizontal coordinate sequence.

[0102] In this embodiment, the second total number is the total number of negative numbers contained in the difference between the horizontal coordinate value and the corresponding previous horizontal coordinate value in the horizontal coordinate sequence.

[0103] In this embodiment, the first round-trip coefficient is a coefficient that represents the number of lateral round trips of the eye movement trajectory, determined based on the abscissa of the eye movement trajectory point.

[0104] In this embodiment, the third total number is the total number of positive numbers contained in the difference between the vertical coordinate value and the corresponding previous vertical coordinate value in the vertical coordinate sequence.

[0105] In this embodiment, the fourth total number is the total number of negative numbers contained in the difference between the vertical coordinate value and the corresponding previous vertical coordinate value in the vertical coordinate sequence.

[0106] In this embodiment, the second round-trip coefficient is a coefficient that represents the number of longitudinal round trips of the eye movement trajectory, determined based on the horizontal coordinate of the eye movement trajectory point.

[0107] In this embodiment, the comprehensive round-trip coefficient is the sum of the first round-trip coefficient and the second round-trip coefficient.

[0108] In this embodiment, the tracking weight is the ratio of the ordinal number of the eye-tracking point in the eye-tracking trajectory to the total number of the first eye-tracking points contained in the eye-tracking trajectory.

[0109] In this embodiment, the total number of the first eye movement trajectory points is the total number of eye movement trajectory points contained in the eye movement trajectory.

[0110] In this embodiment, the weight of interest is the product of the tracking weight and the overall round-trip coefficient.

[0111] In this embodiment, the tracking area of ​​the eye-tracking point is determined based on the corresponding interest weight, using the eye-tracking point as the center of the visual field, including:

[0112] The circular region defined by taking the eye-tracking trajectory point as the center and the product of the preset division length and the interest weight as the radius is the tracking area of ​​the eye-tracking trajectory point.

[0113] In this embodiment, the center of the field of view is the center of the tracking area.

[0114] In this embodiment, the comprehensive tracking area is the area obtained by summing up the tracking areas corresponding to all eye movement trajectory points.

[0115] In this embodiment, the preset shape list is a list that includes squares, circles, ellipses, and irregular polygons.

[0116] In this embodiment, the preset shape is a square, a circle, an ellipse, or an irregular polygon.

[0117] In this embodiment, determining the matching degree between the comprehensive tracking area and the preset shapes included in the preset shape list includes:

[0118] Based on the coordinates of the first contour point of the integrated tracking area and the coordinates of the second contour point of the preset shape, the matching degree between the integrated tracking area and the preset shape is calculated, including:

[0119]

[0120] In the formula, ρ represents the matching degree between the comprehensive tracking area and the preset shape, i represents the i-th first contour point in the comprehensive tracking area, n represents the total number of first contour points contained in the comprehensive tracking area, j represents the j-th second contour point in the preset shape, m represents the total number of second contour points contained in the preset shape, and x represents the total number of second contour points contained in the preset shape. i Let x be the x-coordinate of the i-th first contour point. j Let y be the x-coordinate value of the j-th second contour point. i Let x be the ordinate value of the i-th first contour point. j The ordinate value of the j-th second contour point;

[0121] In the formula, the coordinates of the first contour point are (3,4), (5,12), and (6,8), and the coordinates of the second contour point are (4,4), (5,5), and (6,6), so ρ is 0.868;

[0122] Based on the difference between the coordinate difference between each first contour point in the comprehensive tracking area and each second contour point in the preset shape and the ratio of the difference between the corresponding first contour point and the origin, the matching degree between the comprehensive tracking area and the preset shape is accurately determined.

[0123] In this embodiment, the smallest region is the region with the smallest area containing the comprehensive tracking region, which is the preset shape corresponding to the maximum matching degree.

[0124] The beneficial effects of the above technologies are as follows: The smart glasses and method for eye-tracking interaction of the present invention determine the corresponding coordinate value sequence through the sequence determination subunit included in the region search unit, providing an important data foundation for subsequently determining the comprehensive round-trip coefficient of the eye-tracking trajectory; based on the first determination subunit and the second determination subunit, the comprehensive round-trip coefficient representing the number of round trips in the eye-tracking trajectory is determined based on the number of positive and negative numbers in the difference between each horizontal coordinate and the previous horizontal coordinate, and the number of positive and negative numbers in the difference between each vertical coordinate and the previous vertical coordinate, providing a basis for accurately determining the user's region of interest; based on the weight determination subunit, the interest weight of the eye-tracking point is determined by considering the influence of the number of points in the eye-tracking trajectory on the region determination and the coefficient of the number of round trips in the eye-tracking trajectory; based on the region determination subunit and the final determination subunit, the region of interest that the user wants to interact with is accurately divided based on the interest weight, realizing the determination of the region division weight based on the tracking analysis of the eye-tracking trajectory, making the divided region of interest closer to the user's interaction needs.

[0125] See Figure 7 The automatic analysis unit in S22 includes:

[0126] The region reading subunit is used to read information from the ROI region and obtain the information of interest and the image of interest contained in the ROI region.

[0127] The instruction reading subunit is used to obtain the latest interaction instruction, retrieve the interaction ROI area corresponding to the latest interaction instruction, read information from the interaction ROI area, and obtain the latest interaction information.

[0128] The vocabulary sorting subunit is used to identify the first core vocabulary in the information of interest, sort the first core vocabulary according to the order of appearance of the first core vocabulary in the information of interest, and obtain a first core vocabulary sequence. At the same time, it identifies the second core vocabulary in the latest interaction information, sorts the second core vocabulary according to the order of appearance of the second core vocabulary in the latest interaction information, and obtains a second core vocabulary sequence.

[0129] The third determining subunit is used to determine the degree of correlation between the information of interest and the latest interaction information based on the first core vocabulary sequence and the second core vocabulary sequence, determine the predicted degree of interest of the interest shape based on the preset interest shape list, and determine the first interaction degree coefficient based on the predicted degree of interest and the degree of correlation.

[0130] The trajectory segmentation subunit is used to retrieve the corresponding eye movement trajectory based on the ROI region, determine the inflection point contained in the eye movement trajectory, and divide the eye movement trajectory into multiple eye movement sub-trajectories based on the inflection point.

[0131] The coefficient determination subunit is used to determine the trajectory vibration coefficient of the eye movement trajectory based on the ratio of the total number of inflection points in the eye movement trajectory to the total number of the first eye movement trajectory points, and to determine the sub-interaction degree coefficient of the eye movement sub-trajectory based on the ratio of the total number of the second eye movement trajectory points in the eye movement sub-trajectory to the total number of the first eye movement trajectory points.

[0132] The fourth determining subunit is used to determine the second interaction degree coefficient based on the maximum sub-interaction degree coefficient and the trajectory vibration coefficient;

[0133] The final judgment subunit is used to determine a comprehensive interaction coefficient based on the first interaction degree coefficient and the second interaction degree coefficient. When the comprehensive interaction coefficient is greater than the interaction coefficient threshold, it is determined that the user needs to perform intelligent interaction; otherwise, it is determined that the user does not need to perform intelligent interaction.

[0134] In this embodiment, the information of interest is the information contained in the ROI region.

[0135] In this embodiment, the graphic of interest is the graphic contained within the ROI region.

[0136] In this embodiment, the latest interaction instruction is the interaction instruction executed the most recent time it was determined that the user needs to perform intelligent interaction.

[0137] In this embodiment, the interaction ROI region is the ROI region at the latest time when it is determined that the user needs to perform intelligent interaction.

[0138] In this embodiment, the latest interaction information is the information read from the interaction ROI region.

[0139] In this embodiment, the first core vocabulary is the core vocabulary contained in the information of interest (vocabulary in a preset core vocabulary library).

[0140] In this embodiment, the first core vocabulary sequence is the vocabulary sequence obtained by sorting the first core vocabulary based on the order in which the first core vocabulary appears in the information of interest.

[0141] In this embodiment, the second core vocabulary is the core vocabulary contained in the latest interactive information.

[0142] In this embodiment, the second core vocabulary sequence is the vocabulary sequence obtained by sorting the second core vocabulary based on the order in which the second core vocabulary appears in the latest interactive information.

[0143] In this embodiment, the degree of correlation between the information of interest and the latest interaction information is determined based on the first core vocabulary sequence and the second core vocabulary sequence, including:

[0144] The first core vocabulary word's first ordinal number and the second core vocabulary word's second ordinal number within the first core vocabulary word sequence are determined. Based on the core vocabulary association coefficient list (i.e., containing association coefficients between different core vocabulary words), the association coefficient between each first core vocabulary word and each second core vocabulary word is determined. Based on the first and second word ordinal numbers and the association coefficients, the degree of association between information of interest and the latest interactive information is calculated.

[0145]

[0146] In the formula, ε represents the degree of correlation between information of interest and the latest interactive information, p is the p-th first core word in the first core word sequence, a is the total number of first core words in the first core word sequence, q is the q-th second core word in the second core word sequence, b is the total number of second core words in the second core word sequence, and g... pq This represents the degree of association between the p-th primary core word and the q-th secondary core word.

[0147] For example, a is 2, b is 2, g 11 It is 0.5 g 12 It is 0.8, g 21 For 0.1, g 22 The value is 0.6, and ε is 0.175;

[0148] Based on the first word ordinal number of the first core word in the first core word sequence, the second word ordinal number of the second core word in the second core word sequence, and the correlation coefficient between each first core word and each second core word, the degree of correlation between information of interest and the latest interactive information is accurately calculated.

[0149] In this embodiment, the preset list of shapes of interest is a list containing the predicted degree of interest for each shape of interest.

[0150] In this embodiment, the first interaction degree coefficient is the product of the predicted degree of interest and the degree of association.

[0151] In this embodiment, the inflection point is the point where the eye movement trajectory turns.

[0152] In this embodiment, the eye-tracking sub-trajectory is a number of sub-trajectories obtained by dividing the eye-tracking trajectory based on the inflection point.

[0153] In this embodiment, the trajectory vibration coefficient is the ratio of the total number of inflection points in the eye movement trajectory to the total number of points in the first eye movement trajectory.

[0154] In this embodiment, the total number of the second eye-track points is the total number of eye-track points contained in the eye-track sub-track.

[0155] In this embodiment, the sub-interaction degree coefficient is the ratio of the total number of second eye movement trajectory points to the total number of first eye movement trajectory points contained in the eye movement sub-trajectory.

[0156] In this embodiment, the second interaction degree coefficient is the product of the maximum sub-interaction degree coefficient and the trace vibration coefficient.

[0157] In this embodiment, the comprehensive interaction coefficient is the sum of the first interaction degree coefficient and the second interaction degree coefficient.

[0158] In this embodiment, the interaction coefficient threshold is the minimum interaction coefficient threshold corresponding to the determination that the user needs to perform intelligent interaction.

[0159] The beneficial effects of the above technologies are as follows: The smart glasses and method for eye-tracking interaction of the present invention determine the corresponding coordinate value sequence through the sequence determination subunit included in the region search unit, providing an important data foundation for subsequently determining the comprehensive round-trip coefficient of the eye-tracking trajectory; based on the first determination subunit and the second determination subunit, the comprehensive round-trip coefficient representing the number of round trips in the eye-tracking trajectory is determined based on the number of positive and negative numbers in the difference between each horizontal coordinate and the previous horizontal coordinate, and the number of positive and negative numbers in the difference between each vertical coordinate and the previous vertical coordinate, providing a basis for accurately determining the user's region of interest; based on the weight determination subunit, the interest weight of the eye-tracking point is determined by considering the influence of the number of points in the eye-tracking trajectory on the region determination and the coefficient of the number of round trips in the eye-tracking trajectory; based on the region determination subunit and the final determination subunit, the region of interest that the user wants to interact with is accurately divided based on the interest weight, realizing the determination of the region division weight based on the tracking analysis of the eye-tracking trajectory, making the divided region of interest closer to the user's interaction needs.

[0160] S23: When performing automated analysis of the ROI area, the automatic analysis unit sends instructions to the timing sensor 5, records the user's eye movement viewing time in real time according to the timing sensor 5, and transmits the real-time recorded user's eye movement viewing time to the automatic analysis unit.

[0161] S24: After receiving the user's eye movement viewing time transmitted by the timing sensor 5, the automatic analysis unit analyzes the user's eye movement viewing time recorded by the timing sensor 5 and refers to the access interaction threshold stored in the access storage unit, and feeds back the analysis results to the interaction execution module.

[0162] S3: Read the analysis results of the ROI area that the user wants to interact with. After reading the analysis results, execute different response strategies intelligently according to different analysis results, and perform intelligent eye-tracking interaction according to different response strategies.

[0163] See Figure 4 It should be noted that the steps for the interactive execution module to execute the analysis results are as follows:

[0164] S31: The automatic analysis unit transmits the final analysis results to the information reading unit, which reads the analysis results of the ROI area that the user wants to interact with, and transmits the information reading results to the strategy execution unit.

[0165] S32: After receiving the information reading result transmitted by the information reading unit, the strategy execution unit uses the strategy execution unit to intelligently execute different response strategies for different analysis results, and performs intelligent eye-tracking interaction according to different response strategies.

[0166] It should be noted that the steps executed by the strategy execution unit are as follows:

[0167] S321: If the analysis result of the user's eye movement viewing time value recorded by the timing sensor 5 does not exceed the access interaction threshold stored in the access storage unit, the strategy execution unit executes the first-level response strategy according to the analysis result. At this time, the user's viewing page does not change, so the user does not interact with the viewing page.

[0168] S322: If the analysis result of the user's eye-tracking viewing time value recorded by the timing sensor 5 exceeds the access interaction threshold stored in the access storage unit, the strategy execution unit executes the secondary response strategy according to the analysis result. At this time, the user's viewing page changes, enabling the user to perform eye-tracking intelligent interaction with the viewing page.

[0169] Different response strategies are intelligently executed based on different analysis results, and intelligent eye-tracking interaction is performed based on different response strategies, as shown in Table 1:

[0170] Table 1: Different response strategies are executed based on different analysis results.

[0171]

[0172]

[0173] Example 3

[0174] See Figure 5 To better demonstrate the eye-tracking interaction process of smart glasses, this embodiment proposes an eye-tracking interaction method for smart glasses, including the following steps:

[0175] S10: The user wears smart glasses. The information acquisition module collects the user's three-axis attitude angle and acceleration in real time, captures the user's eye movement in real time and senses the user's eye movement state, measures the straight-line distance from the user's head to the obstacle in front of them and records the user's eye movement viewing time. After the above eye movement data information is collected, the real-time collected eye movement data information is transmitted to the information processing module.

[0176] S20: The information processing module performs feature extraction on the real-time collected eye-tracking data, extracts useful user eye-tracking data, identifies and judges the extracted useful user eye-tracking data, finds the interrelationships between user eye-tracking data, generates corresponding three-dimensional coordinates based on three-axis attitude angles, acceleration and user eye-tracking state, and finally forms an eye-tracking coordinate system, and transmits the eye-tracking coordinate system data to the information analysis module.

[0177] S30: The information analysis module delineates the area that the user wants to interact with from the received eye-tracking coordinate system data using squares, circles, ellipses, irregular polygons, etc., finds the ROI area that the user wants to interact with, and automatically analyzes the ROI area that the user wants to interact with based on the user's eye-tracking viewing time recorded by the timing sensor 5 and with reference to the access interaction threshold stored in the access storage unit. It then determines whether the user needs to perform intelligent interaction and feeds back the analysis results to the interaction execution module.

[0178] S40: The interaction execution module reads the analysis results of the ROI area that the user wants to interact with, and intelligently executes different response strategies for different analysis results, and performs intelligent eye-tracking interaction based on different response strategies.

[0179] In summary, the eye-tracking interactive smart glasses and method of the present invention involve a user wearing smart glasses. An information acquisition module collects the user's three-axis attitude angles and acceleration in real time, captures real-time images of the user's eye movements and senses the user's eye movement state, measures the straight-line distance from the user's head to an obstacle directly in front of them, and records the user's eye-tracking viewing time. After the eye-tracking data is collected, an information processing module performs feature extraction on the real-time collected eye-tracking data, extracting useful user eye-tracking data. This useful user eye-tracking data is then identified and judged to find the interrelationships between user eye-tracking data. Based on the three-axis attitude angles, acceleration, and user eye movement state, corresponding three-dimensional coordinates are generated, ultimately forming an eye-tracking coordinate system. This facilitates understanding the user's eye movement state and enabling intelligent interaction using the user's eye movement state. This intelligent glasses system enables smart interaction without the need for external devices, making the service more convenient. The information analysis module delineates the area the user wants to interact with using rectangles, circles, ellipses, and irregular polygons from the received eye-tracking coordinate system data. It identifies the ROI (Region of Interest) that the user wants to interact with and automatically analyzes the ROI based on the user's eye-tracking viewing time recorded by the timing sensor 5 and the access interaction threshold stored in the access storage unit. It then determines whether the user needs to perform intelligent interaction. The interaction execution module reads the analysis results of the ROI that the user wants to interact with and intelligently executes different response strategies for different analysis results. Based on the different response strategies, it performs eye-tracking intelligent interaction, making intelligent interaction convenient and freeing up the user's hands.

[0180] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A smart glasses with eye-tracking interaction, comprising a smart interactive glasses body (1), wherein the smart interactive glasses body (1) is provided with a left lens (11) and a right lens (12) and the left lens (11) and the right lens (12) are symmetrically distributed, characterized in that, The smart interactive glasses body (1) is equipped with an information collection module and an interactive server. The information collection module interacts with the interactive server through a wireless network. The interactive server has a built-in information processing module; The information processing module processes the real-time eye-tracking data and performs the following operations: After the information acquisition module collects eye movement data in real time, it transmits the real-time eye movement data to the feature extraction unit. The feature extraction unit uses the real-time collected three-axis attitude angle, acceleration, user eye movement state, straight-line distance from the user's head to the obstacle in front, and user eye movement viewing time to extract useful user eye movement data, and transmits the extracted useful user eye movement data to the recognition and judgment unit. After receiving the extracted data from the feature extraction unit, the identification and judgment unit uses the identification and judgment unit to identify and judge the extracted useful user eye movement data, find the interrelationship between the user eye movement data, and generate corresponding three-dimensional coordinates based on the three-axis attitude angle, acceleration and user eye movement state, and finally form an eye movement tracking coordinate system. The recognition and judgment unit transmits the final eye-tracking coordinate system data to the region search unit, which includes: The sequence determination subunit is used to determine the eye trajectory based on the eye tracking coordinate system data, and determine the coordinate value of each eye trajectory point. Based on the point sequence of the eye trajectory points in the eye trajectory, the horizontal coordinate value of each eye trajectory point is sorted to obtain the horizontal coordinate sequence. At the same time, based on the point sequence, the vertical coordinate value of each eye trajectory point is sorted to obtain the vertical coordinate sequence. The first determining subunit is used to determine the difference between each horizontal coordinate value in the horizontal coordinate sequence and the corresponding previous horizontal coordinate value, to calculate the first total number of positive numbers and the second total number of negative numbers contained in the horizontal coordinate difference, and to use the ratio of the smaller of the first total number and the second total number to the first total number of eye movement trajectory points contained in the eye movement trajectory as the first round-trip coefficient of the eye movement trajectory. The second determining subunit is used to determine the difference between each ordinate value in the vertical coordinate sequence and the corresponding previous ordinate value, to calculate the third total number of positive numbers and the fourth total number of negative numbers contained in the ordinate difference, and to use the ratio of the smaller of the third total number and the fourth total number to the total number of the first eye movement trajectory points as the second round-trip coefficient of the eye movement trajectory. The weight determination subunit is used to take the sum of the first round-trip coefficient and the second round-trip coefficient as the comprehensive round-trip coefficient of the eye track, take the ratio of the point ordinal number of the eye track point in the eye track to the total number of first eye track points included in the eye track as the tracking weight of the eye track point, and determine the interest weight of the eye track point based on the tracking weight and the comprehensive round-trip coefficient. The region determination subunit is used to determine the tracking region of the eye track point with the eye track point as the center of the field of vision and based on the corresponding interest weight, and to summarize the tracking regions corresponding to all eye track points to obtain the corresponding comprehensive tracking region. The final determination subunit is used to determine the matching degree between the comprehensive tracking area and the preset shapes contained in the preset shape list. Based on the preset shape corresponding to the maximum matching degree, the minimum area containing the comprehensive tracking area is determined. The minimum area is taken as the ROI area that the user wants to interact with and access, and the determined ROI area that the user wants to interact with and access is transmitted to the automatic analysis unit. The preset shape list includes rectangles, circles, ellipses, and irregular polygons; Determining the matching degree between the comprehensive tracking area and the preset shapes included in the preset shape list includes: Based on the coordinates of the first contour point of the integrated tracking area and the coordinates of the second contour point of the preset shape, the matching degree between the integrated tracking area and the preset shape is calculated, including: In the formula, To comprehensively track the matching degree between the region and the preset shape, For the i-th first contour point in the comprehensive tracking region, To determine the total number of first contour points contained in the overall tracking area. For the j-th second contour point in the preset shape, This represents the total number of second contour points contained in the preset shape. Let x be the x-coordinate of the i-th first contour point. Let j be the x-coordinate value of the second contour point. Let be the ordinate value of the i-th first contour point. The ordinate value of the j-th second contour point; In the formula, the coordinates of the first contour point are (3,4), (5,12), and (6,8), and the coordinates of the second contour point are (4,4), (5,5), and (6,6). It is 0.868; Based on the difference between the coordinate difference between each first contour point in the comprehensive tracking area and each second contour point in the preset shape and the ratio of the difference between the corresponding first contour point and the origin, the matching degree between the comprehensive tracking area and the preset shape is accurately determined.

2. The smart glasses with eye-tracking interaction as described in claim 1, characterized in that, The information acquisition module includes an inertial measurement unit (2), a high-definition acquisition camera (3), a ranging sensor (4), and a timing sensor (5), wherein... The inertial measurement unit (2) is installed on the smart interactive glasses body (1), and the inertial measurement unit (2) is used to measure the three-axis attitude angle and acceleration of the user's eyeball relative to the user's head. The inertial measurement unit (2) includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometers are used to detect the acceleration signals of the user's eyeball in the independent three axes of the carrier coordinate system during the movement. The gyroscopes are used to detect the angular velocity signals of the user's eyeball relative to the navigation coordinate system during the movement. The angular velocity and acceleration of the user's eyeball in three-dimensional space during the movement are measured, and the attitude of the user's eyeball during the movement is calculated accordingly. The high-definition acquisition camera (3) is two and symmetrically installed on the smart interactive glasses body (1). The two high-definition acquisition cameras (3) are located above the left lens (11) and the right lens (12) respectively. The high-definition acquisition camera (3) is used to capture the user's eye movement in real time and sense the user's eye movement state. The distance sensor (4) is installed in the middle of the smart interactive glasses body (1). The distance sensor (4) is a laser distance sensor and is used to measure the straight distance from the user's head to the obstacle in front of him. The timing sensor (5) is installed on the smart interactive glasses body (1). The timing sensor (5) is used to time the user's eye movement and viewing time.

3. The smart glasses with eye-tracking interaction as described in claim 2, characterized in that, The interactive server has a built-in information processing module, information analysis module, and interactive execution module, wherein... The information processing module is used to process the eye-tracking data collected in real time by the information acquisition module. First, it extracts features from the real-time eye-tracking data and automatically acquires the eye-tracking data. Then, it identifies and judges the acquired eye-tracking data and generates corresponding three-dimensional coordinates, finally forming an eye-tracking coordinate system. Finally, it transmits the processed eye-tracking data to the information analysis module. The information analysis module is used to analyze the eye-tracking data processed in real time by the information processing module. First, it finds the ROI area that the user wants to interact with based on the formed eye-tracking coordinate system. Based on the user's eye-tracking viewing time recorded by the timing sensor (5) and referring to the access interaction threshold stored in the access storage unit, it automatically analyzes the ROI area that the user wants to interact with, determines whether the user needs to interact, and feeds back the analysis results to the interaction execution module. The interactive execution module is used to interactively execute the analysis results transmitted in real time by the information analysis module. First, it reads the analysis results of the ROI area that the user wants to interact with. After reading the analysis results, it intelligently executes different response strategies according to different analysis results, and performs intelligent interaction with eye tracking according to different response strategies.

4. The smart glasses with eye-tracking interaction as described in claim 3, characterized in that, The information analysis module analyzes the eye-tracking data and performs the following operations: The identification and judgment unit transmits the final eye-tracking coordinate system data to the region search unit. The region search unit uses various methods such as rectangles, circles, ellipses, and irregular polygons to delineate the region that the user wants to interact with from the received eye-tracking coordinate system data, finds the ROI region that the user wants to interact with, and transmits the found ROI region that the user wants to interact with to the automatic analysis unit. After receiving the ROI area data transmitted by the identification and judgment unit, the automatic analysis unit automatically analyzes the ROI area that the user wants to interact with and determines whether the user needs to perform intelligent interaction. When performing automated analysis of the ROI area, the automatic analysis unit sends instructions to the timing sensor (5), records the user's eye movement viewing time in real time according to the timing sensor (5), and transmits the real-time recorded user eye movement viewing time to the automatic analysis unit. After receiving the user's eye movement viewing time transmitted by the timing sensor (5), the automatic analysis unit analyzes the user's eye movement viewing time recorded by the timing sensor (5) and refers to the access interaction threshold stored in the access storage unit, and feeds back the analysis results to the interaction execution module.

5. The smart glasses with eye-tracking interaction as described in claim 4, characterized in that, The automatic analysis unit includes: The region reading subunit is used to read information from the ROI region and obtain the information of interest and the image of interest contained in the ROI region. The instruction reading subunit is used to obtain the latest interaction instruction, retrieve the interaction ROI area corresponding to the latest interaction instruction, read information from the interaction ROI area, and obtain the latest interaction information. The vocabulary sorting subunit is used to identify the first core vocabulary in the information of interest, sort the first core vocabulary according to the order of appearance of the first core vocabulary in the information of interest, and obtain a first core vocabulary sequence. At the same time, it identifies the second core vocabulary in the latest interaction information, sorts the second core vocabulary according to the order of appearance of the second core vocabulary in the latest interaction information, and obtains a second core vocabulary sequence. The third determining subunit is used to determine the degree of correlation between the information of interest and the latest interaction information based on the first core vocabulary sequence and the second core vocabulary sequence, determine the predicted degree of interest of the interest shape based on the preset interest shape list, and determine the first interaction degree coefficient based on the predicted degree of interest and the degree of correlation. The trajectory segmentation subunit is used to retrieve the corresponding eye movement trajectory based on the ROI region, determine the inflection point contained in the eye movement trajectory, and divide the eye movement trajectory into multiple eye movement sub-trajectories based on the inflection point. The coefficient determination subunit is used to determine the trajectory vibration coefficient of the eye movement trajectory based on the ratio of the total number of inflection points in the eye movement trajectory to the total number of the first eye movement trajectory points, and to determine the sub-interaction degree coefficient of the eye movement sub-trajectory based on the ratio of the total number of the second eye movement trajectory points in the eye movement sub-trajectory to the total number of the first eye movement trajectory points. The fourth determining subunit is used to determine the second interaction degree coefficient based on the maximum sub-interaction degree coefficient and the trajectory vibration coefficient; The final judgment subunit is used to determine a comprehensive interaction coefficient based on the first interaction degree coefficient and the second interaction degree coefficient. When the comprehensive interaction coefficient is greater than the interaction coefficient threshold, it is determined that the user needs to perform intelligent interaction; otherwise, it is determined that the user does not need to perform intelligent interaction.

6. The smart glasses with eye-tracking interaction as described in claim 5, characterized in that, The interactive execution module executes the analysis results and performs the following operations: The automatic analysis unit transmits the final analysis results to the information reading unit, which then reads the analysis results of the ROI area that the user wants to interact with and transmits the information reading results to the strategy execution unit. After receiving the information reading results transmitted by the information reading unit, the strategy execution unit intelligently executes different response strategies for different analysis results, and performs intelligent eye-tracking interaction based on different response strategies.

7. The smart glasses with eye-tracking interaction as described in claim 6, characterized in that, The policy execution unit performs the following operations: If the analysis result of the user's eye movement viewing time value recorded by the timing sensor (5) does not exceed the access interaction threshold stored in the access storage unit, the strategy execution unit executes the first-level response strategy according to the analysis result. At this time, the user's viewing page does not change, so the user does not interact with the viewing page. If the analysis result of the user's eye-tracking viewing time value recorded by the timing sensor (5) exceeds the access interaction threshold stored in the access storage unit, the strategy execution unit executes the secondary response strategy according to the analysis result. At this time, the user's viewing page changes, enabling the user to perform eye-tracking intelligent interaction with the viewing page.

8. An eye-tracking interaction method for smart glasses with eye-tracking interaction as described in any one of claims 1-7, characterized in that, Includes the following steps: S10: The user wears smart glasses, and the information acquisition module collects the user's three-axis attitude angle and acceleration in real time, captures the user's eye movement in real time and senses the user's eye movement state, measures the straight-line distance from the user's head to the obstacle in front of them and records the user's eye movement viewing time. After the above eye movement data information is collected, the eye movement data information collected in real time is transmitted to the information processing module. S20: The information processing module performs feature extraction on the real-time collected eye movement data information, extracts useful user eye movement data, identifies and judges the extracted useful user eye movement data, finds the interrelationship between user eye movement data, generates corresponding three-dimensional coordinates based on three-axis attitude angles, acceleration and user eye movement state, and finally forms an eye movement tracking coordinate system, and transmits the eye movement tracking coordinate system data to the information analysis module. S30: The information analysis module delineates the area that the user wants to interact with from the received eye-tracking coordinate system data in the form of squares, circles, ellipses, irregular polygons, etc., finds the ROI area that the user wants to interact with, and automatically analyzes the ROI area that the user wants to interact with based on the user's eye-tracking viewing time recorded by the timing sensor (5) and the access interaction threshold stored in the access storage unit, determines whether the user needs to perform intelligent interaction, and feeds back the analysis results to the interaction execution module; S40: The interaction execution module reads the analysis results of the ROI area that the user wants to interact with, and intelligently executes different response strategies for different analysis results, and performs intelligent eye-tracking interaction based on different response strategies.

Images