Intelligent AR glasses interaction system

Abstract

The invention relates to the technical field of AR glasses, in particular to an intelligent AR glasses interaction system which comprises the following modules: a gesture recognition module which is used for recognizing a hand action of a user in real time and judging whether the hand action is matched with a preset interaction action; the interaction interface generation module is used for generating a corresponding interaction interface according to a preset interaction action when the gesture recognition module recognizes that the hand action is matched with the preset interaction action and outputting the interaction interface in the AR glasses, and the interaction interface is distributed at the hand position of the user according to a preset arrangement mode. According to the method and the device, the eyeball movement track of the user is captured and tracked in real time, the sight focus and the gazing duration of the user are analyzed to judge the attention direction and the potential interest point of the user, multiple interaction modes are realized, more natural, smoother and personalized interaction experience is provided for the user, and the intelligence of the AR glasses interaction system is improved.

Description

Technical Field

[0001] The present invention relates to the field of AR glasses technology, and in particular to an intelligent AR glasses interaction system. Background Art

[0002] AR glasses are a product with a sci-fi feel and cutting-edge technology. Wearing them allows you to experience the future functions used by handsome male and female protagonists in sci-fi movies. AR glasses can realize many functions and can be regarded as a miniature mobile phone. It can determine the user's state by tracking the eye's line of sight and activate corresponding functions.

[0003] Traditional AR glasses interaction systems mainly rely on interaction methods such as gesture recognition and voice recognition. Although these methods have improved the user experience to a certain extent, they still have many limitations. For example, gesture recognition may cause the recognition accuracy to decrease due to the occlusion of ambient light, and it cannot recognize eye movement signals, which reduces the uniqueness of the interaction method.

[0004] Therefore, we make improvements to this and propose a smart AR glasses interaction system. Summary of the invention

[0005] The purpose of the present invention is to address the problem that the existing traditional AR glasses interaction system mainly relies on interaction methods such as gesture recognition and voice recognition, has many limitations, cannot recognize eye movement signals, and reduces the singleness of the interaction method.

[0006] In order to achieve the above-mentioned purpose of the invention, the present invention provides a smart AR glasses interaction system to improve the above-mentioned problems.

[0007] The specific application is as follows:

[0008] An intelligent AR glasses interaction system includes the following modules:

[0009] Gesture recognition module: used to recognize the user's hand movements in real time and determine whether the hand movements match the preset interactive actions;

[0010] Interaction interface generation module: When the gesture recognition module recognizes that the hand movement matches the preset interaction movement, it generates a corresponding interaction interface according to the preset interaction movement, and outputs the interaction interface in the AR glasses. The interaction interface is distributed at the user's hand position according to the preset arrangement method;

[0011] Operation instruction determination module: used to identify the user's operation on the interactive interface at the hand position and determine the control instruction corresponding to the operation;

[0012] Touchpad module: set on the AR glasses, used to detect the user's touch operation in real time, and activate and execute the preset operation when a touch operation that meets the preset operation type is detected;

[0013] Voice interaction module: In any interface, when the touchpad module detects a touch operation that meets the preset operation type, the voice interaction module is started to interactively control the current interface of the AR glasses through voice signals;

[0014] Eye tracking module: integrated into AR glasses, used to capture and track the user's eye movement trajectory in real time, analyze the user's gaze focus and gaze duration, and determine the user's attention direction and potential points of interest;

[0015] Emotion recognition module: Combining facial expression analysis and voice emotion recognition technology, it analyzes the user's emotional state in real time, including joy, surprise, confusion and anger, and provides emotional feedback for the interactive system.

[0016] As the preferred technical solution of the present application, the user is allowed to adjust the sensitivity of the emotion recognition module to specific emotions according to his or her preferences and needs.

[0017] As a preferred technical solution of the present application, it includes a wireless communication unit for wirelessly communicating with an external wireless interactive device, so that the user can interactively control the interface through the wireless interactive device, so that the AR glasses can interact with unadapted objects.

[0018] As the preferred technical solution of this application, it includes a cross-scenario behavior prediction module, which analyzes the user's interactive behavior in different scenarios, predicts the user's possible behavior in future scenarios, and optimizes the interactive experience in advance.

[0019] As the preferred technical solution of this application, including an environmental adaptive display module, AR glasses automatically adjust display parameters according to the light and color conditions of the user's environment, ensuring the best visual effects in different environments while reducing eye fatigue.

[0020] As a preferred technical solution of the present application, the eye tracking module includes the following steps:

[0021] S1. Image and signal acquisition: AR glasses have integrated cameras and photosensors to capture images of eyeballs and reflected light;

[0022] S2. Infrared light source: Integrate infrared light source on AR glasses to illuminate the eyeballs and generate reflected light spots that can be used for tracking;

[0023] S3, preprocessing: preprocessing the collected images and signals, including denoising and contrast enhancement operations;

[0024] S4, feature extraction: use image processing technology to extract the key feature points of the eyeball, including the pupil center and corneal reflection point;

[0025] S5, eye tracking algorithm: according to the extracted feature points, the eye tracking algorithm is used to calculate the eye movement trajectory and line of sight direction. The algorithm includes a two-dimensional mapping model and a neural network model;

[0026] S6. Data analysis: Conduct in-depth analysis of the tracked eye movement data, including calculating the gaze point position and gaze duration, and determining the user's attention direction and potential points of interest.

[0027] As the preferred technical solution of the present application, the analyzed eye movement data is fed back in real time, and the displayed content and interaction method are adjusted according to the user's attention direction and interest points.

[0028] As a preferred technical solution of the present application, the neural network model collects training data including eye images and corresponding gaze directions. The data is obtained by having the subjects watch the content and simultaneously recording the eye images and gaze directions. The neural network model is trained using the collected training data, and the eye images acquired in real time are input into the trained neural network model. The model will output the corresponding gaze directions and gaze point positions.

[0029] As a preferred technical solution of the present application, the AR glasses frame includes a touchpad module installed on the surface of the AR glasses frame, an AR glasses lens is installed on the inner wall of the AR glasses frame, a wearing mechanism is connected to the surface of the AR glasses frame, and the wearing mechanism includes a U-shaped block, one side of the U-shaped block is connected to the AR glasses frame, a rotating rod is connected to the inside of the U-shaped block, a gear column is sleeved on the surface of the rotating rod, a long tube is sleeved on the surface of the rotating rod, a wearing rod is sleeved on the surface of the long tube, a tooth plate is meshed on one side of the gear column, a long rod is connected to one side of the tooth plate, a partition is connected to the inner cavity of the long tube, one side of the long rod penetrates the partition and is connected to an inclined plate through a rotating shaft, one side of the inclined plate is connected to a second threaded rod through a rotating shaft, a vertical tube is threadedly sleeved on the surface of the second threaded rod, the bottom of the vertical tube is connected to the long tube, a long groove is opened on the surface of the wearing rod, the top of the vertical tube penetrates the long groove, and a locking sleeve is threadedly sleeved on the surface of the vertical tube.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] In the scheme of this application:

[0032] 1. In order to solve the problem that the traditional AR glasses interaction system in the prior art mainly relies on interaction methods such as gesture recognition and voice recognition, has many limitations, cannot recognize eye movement signals, and reduces the singleness of the interaction method, the present application realizes the interaction with the current interface of AR glasses through voice signals through the set gesture recognition module, interaction interface generation module, operation instruction determination module, touchpad module, voice interaction module and eye tracking module, captures and tracks the user's eye movement trajectory in real time, analyzes the user's line of sight and gaze duration to determine the user's attention direction and potential points of interest, realizes a variety of interaction methods, provides users with a more natural, smooth and personalized interaction experience, and improves the intelligence of the AR glasses interaction system;

[0033] 2. Through the set emotion recognition module, combined with facial expression analysis and voice emotion recognition technology, the user's emotional state, including joy, surprise, confusion and anger, is analyzed in real time to provide emotional feedback for the interactive system. This emotion-based interaction method is closer to the natural communication between people, making the interactive experience of AR glasses more humane. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The principle block diagram of the smart AR glasses interaction system provided in this application;

[0035] Figure 2 A schematic diagram of the structure of an AR glasses frame and an AR glasses lens of the smart AR glasses interaction system provided in this application;

[0036] Figure 3 A schematic diagram of the side view structure of an AR glasses frame and an AR glasses lens of the smart AR glasses interaction system provided in this application;

[0037] Figure 4 Schematic diagram of the wearing mechanism of the smart AR glasses interaction system provided in this application.

[0038] Indicated in the figure:

[0039] 1. AR glasses frame; 2. AR glasses lens; 3. wearing mechanism; 30. U-shaped block; 31. rotating rod; 32. gear column; 33. long tube; 34. wearing rod; 35. tooth plate; 36. long rod; 37. partition; 38. inclined plate; 39. second threaded rod; 310. vertical tube; 311. long groove; 312. locking sleeve. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0041] As described in the background technology, traditional AR glasses interaction systems mainly rely on interaction methods such as gesture recognition and voice recognition, which have many limitations and cannot recognize eye movement signals, reducing the uniqueness of the interaction method.

[0042] In order to solve this technical problem, the present invention provides an intelligent AR glasses interaction system, which is applied to the field of AR glasses technology.

[0043] Specifically, please refer to Figure 1 , a smart AR glasses interaction system, including the following modules:

[0044] Gesture recognition module: used to recognize the user's hand movements in real time and determine whether the hand movements match the preset interactive actions;

[0045] Interaction interface generation module: When the gesture recognition module recognizes that the hand movement matches the preset interaction movement, it generates a corresponding interaction interface according to the preset interaction movement, and outputs the interaction interface in the AR glasses. The interaction interface is distributed at the user's hand position according to the preset arrangement method;

[0046] Operation instruction determination module: used to identify the user's operation on the interactive interface at the hand position and determine the control instruction corresponding to the operation;

[0047] Touchpad module: set on the AR glasses, used to detect the user's touch operation in real time, and activate and execute the preset operation when a touch operation that meets the preset operation type is detected;

[0048] Voice interaction module: In any interface, when the touchpad module detects a touch operation that meets the preset operation type, the voice interaction module is started to interactively control the current interface of the AR glasses through voice signals;

[0049] Eye tracking module: integrated into AR glasses, used to capture and track the user's eye movement trajectory in real time, analyze the user's gaze focus and gaze duration, and determine the user's attention direction and potential points of interest;

[0050] Emotion recognition module: Combining facial expression analysis and voice emotion recognition technology, it analyzes the user's emotional state in real time, including joy, surprise, confusion and anger, and provides emotional feedback for the interactive system.

[0051] The present application realizes interaction with the current interface of AR glasses through voice signals through the set gesture recognition module, interactive interface generation module, operation instruction determination module, touchpad module, voice interaction module and eye tracking module, captures and tracks the user's eye movement trajectory in real time, analyzes the user's line of sight and gaze duration to determine the user's attention direction and potential points of interest, realizes a variety of interaction methods, provides users with a more natural, smooth and personalized interaction experience, and improves the intelligence of the AR glasses interaction system.

[0052] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0053] Example 1, please refer to Figure 1 , a smart AR glasses interaction system, including the following modules:

[0054] Gesture recognition module: used to recognize the user's hand movements in real time and determine whether the hand movements match the preset interactive actions;

[0055] Interaction interface generation module: When the gesture recognition module recognizes that the hand movement matches the preset interaction movement, it generates a corresponding interaction interface according to the preset interaction movement, and outputs the interaction interface in the AR glasses. The interaction interface is distributed at the user's hand position according to the preset arrangement method;

[0056] Operation instruction determination module: used to identify the user's operation on the interactive interface at the hand position and determine the control instruction corresponding to the operation;

[0057] Touchpad module: set on the AR glasses, used to detect the user's touch operation in real time, and activate and execute the preset operation when a touch operation that meets the preset operation type is detected;

[0058] Voice interaction module: In any interface, when the touchpad module detects a touch operation that meets the preset operation type, the voice interaction module is started to interactively control the current interface of the AR glasses through voice signals;

[0059] Eye tracking module: integrated into AR glasses, used to capture and track the user's eye movement trajectory in real time, analyze the user's gaze focus and gaze duration, and determine the user's attention direction and potential points of interest;

[0060] Emotion recognition module: Combining facial expression analysis and voice emotion recognition technology, it analyzes the user's emotional state in real time, including joy, surprise, confusion and anger, and provides emotional feedback for the interactive system.

[0061] Allow users to adjust the sensitivity of the emotion recognition module to specific emotions according to their preferences and needs.

[0062] It includes a wireless communication unit for wirelessly communicating with an external wireless interactive device, so that the user can interactively control the interface through the wireless interactive device, and the AR glasses can interact with unadapted objects.

[0063] It includes a cross-scenario behavior prediction module, which analyzes the user's interactive behavior in different scenarios, predicts the user's possible behavior in future scenarios, and optimizes the interactive experience in advance.

[0064] Including an environmental adaptive display module, AR glasses automatically adjust display parameters according to the light and color conditions of the user's environment, ensuring the best visual effects in different environments while reducing eye fatigue.

[0065] Embodiment 2 further optimizes the smart AR glasses interaction system provided in Embodiment 1. Specifically, Figure 1 As shown, the eye tracking module includes the following steps:

[0066] S1. Image and signal acquisition: AR glasses have integrated cameras and photosensors to capture images of eyeballs and reflected light;

[0067] S2. Infrared light source: Integrate infrared light source on AR glasses to illuminate the eyeballs and generate reflected light spots that can be used for tracking;

[0068] S3, preprocessing: preprocessing the collected images and signals, including denoising and contrast enhancement operations;

[0069] S4, feature extraction: use image processing technology to extract the key feature points of the eyeball, including the pupil center and corneal reflection point;

[0070] S5, eye tracking algorithm: according to the extracted feature points, the eye tracking algorithm is used to calculate the eye movement trajectory and line of sight direction. The algorithm includes a two-dimensional mapping model and a neural network model;

[0071] S6. Data analysis: Conduct in-depth analysis of the tracked eye movement data, including calculating the gaze point position and gaze duration, and determining the user's attention direction and potential points of interest.

[0072] Example 3 further optimizes the smart AR glasses interaction system provided in Example 1 or 2. Specifically, the analyzed eye movement data is fed back in real time, and the display content and interaction method are adjusted according to the user's attention direction and interest points.

[0073] The neural network model collects training data including eye images and corresponding gaze directions. The data is obtained by having the subjects watch the content and recording the eye images and gaze directions at the same time. The neural network model is trained using the collected training data. The real-time eye images are input into the trained neural network model, and the model will output the corresponding gaze direction and gaze point position.

[0074] Embodiment 4, an intelligent AR glasses interaction system, comprising an AR glasses frame 1, a touch panel module mounted on the surface of the AR glasses frame 1, an AR glasses lens 2 mounted on the inner wall of the AR glasses frame 1, a wearing mechanism 3 connected to the surface of the AR glasses frame 1, the wearing mechanism 3 comprising a U-shaped block 30, one side of the U-shaped block 30 is connected to the AR glasses frame 1, a rotating rod 31 is connected to the inside of the U-shaped block 30, a gear column 32 is sleeved on the surface of the rotating rod 31, a long tube 33 is sleeved on the surface of the rotating rod 31, a wearing rod 34 is sleeved on the surface of the long tube 33, and a gear column 32 is sleeved on the surface of the rotating rod 31. A toothed plate 35 is meshed on one side of the wheel column 32, a long rod 36 is connected to one side of the toothed plate 35, a partition 37 is connected to the inner cavity of the long tube 33, one side of the long rod 36 penetrates the partition 37 and is connected to an inclined plate 38 via a rotating shaft, one side of the inclined plate 38 is connected to a second threaded rod 39 via a rotating shaft, a vertical tube 310 is threadedly sleeved on the surface of the second threaded rod 39, the bottom of the vertical tube 310 is connected to the long tube 33, a long groove 311 is opened on the surface of the wearing rod 34, the top of the vertical tube 310 penetrates the long groove 311, and a locking sleeve 312 is threadedly sleeved on the surface of the vertical tube 310.

[0075] By providing the second threaded rod 39, the vertical tube 310, the long groove 311, the locking sleeve 312, the long tube 33 and the wearing rod 34, the wearing rod 34 can be moved laterally and the length of the wearing rod 34 can be adjusted to fit the glasses on heads of different sizes. By cooperating with the gear column 32, the tooth plate 35, the rotating rod 31, the long rod 36 and the inclined plate 38, the gear column 32 can be rotated, thereby driving the wearing rod 34 to rotate, and the distance between the two wearing rods 34 can be adjusted. The wearing rod 34 clamps the user's head, thereby improving the stability of the glasses and preventing them from falling off.

[0076] When wearing, move the wearing rod 34 to adjust the distance between it and the U-shaped block 30, rotate the locking sleeve 312 to be threadedly connected to the vertical tube 310, so that the bottom of the locking sleeve 312 is pressed against the wearing rod 34 to position the wearing rod 34, rotate the second threaded rod 39 to be threadedly connected to the vertical tube 310, so that the second threaded rod 39 drives the inclined plate 38 to move, and the inclined plate 38 drives the long rod 36 and the toothed plate 35 to move, and the toothed plate 35 drives the gear column 32 to rotate, and the gear column 32 drives the long tube 33 and the wearing rod 34 to rotate through the rotating rod 31, so that the two wearing rods 34 are clamped on the user's head, which is worn firmly and will not fall off.

[0077] Obviously, the embodiments described above are only some embodiments of the present invention, rather than all embodiments. The preferred embodiments of the present invention are given in the accompanying drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to perform equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of the present invention, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A smart AR glasses interaction system, characterized in that: Includes the following modules: Gesture recognition module: used to recognize the user's hand movements in real time and determine whether the hand movements match the preset interactive actions; Interaction interface generation module: When the gesture recognition module recognizes that the hand movement matches the preset interaction movement, it generates a corresponding interaction interface according to the preset interaction movement, and outputs the interaction interface in the AR glasses. The interaction interface is distributed at the user's hand position according to the preset arrangement method; Operation instruction determination module: used to identify the user's operation on the interactive interface at the hand position and determine the control instruction corresponding to the operation; Touchpad module: set on the AR glasses, used to detect the user's touch operation in real time, and activate and execute the preset operation when a touch operation that meets the preset operation type is detected; Voice interaction module: In any interface, when the touchpad module detects a touch operation that meets the preset operation type, the voice interaction module is started to interactively control the current interface of the AR glasses through voice signals; Eye tracking module: integrated into AR glasses, used to capture and track the user's eye movement trajectory in real time, analyze the user's gaze focus and gaze duration, and determine the user's attention direction and potential points of interest; Emotion recognition module: Combining facial expression analysis and voice emotion recognition technology, it analyzes the user's emotional state in real time, including joy, surprise, confusion and anger, and provides emotional feedback for the interactive system.

2. The intelligent AR glasses interaction system according to claim 1, characterized in that: Allow users to adjust the sensitivity of the emotion recognition module to specific emotions according to their preferences and needs.

3. The intelligent AR glasses interaction system according to claim 2, characterized in that: It includes a wireless communication unit for wirelessly communicating with an external wireless interactive device, so that the user can interactively control the interface through the wireless interactive device, and the AR glasses can interact with unadapted objects.

4. The intelligent AR glasses interaction system according to claim 3, characterized in that: It includes a cross-scenario behavior prediction module, which analyzes the user's interactive behavior in different scenarios, predicts the user's possible behavior in future scenarios, and optimizes the interactive experience in advance.

5. The intelligent AR glasses interaction system according to claim 4, characterized in that: Including an environmental adaptive display module, AR glasses automatically adjust display parameters according to the light and color conditions of the user's environment, ensuring the best visual effects in different environments while reducing eye fatigue.

6. The intelligent AR glasses interaction system according to claim 5, characterized in that: The eye tracking module includes the following steps: S1. Image and signal acquisition: AR glasses have integrated cameras and photosensors to capture images of eyeballs and reflected light; S2. Infrared light source: Integrate infrared light source on AR glasses to illuminate the eyeballs and generate reflected light spots that can be used for tracking; S3, preprocessing: preprocessing the collected images and signals, including denoising and contrast enhancement operations; S4, feature extraction: use image processing technology to extract the key feature points of the eyeball, including the pupil center and corneal reflection point; S5, eye tracking algorithm: according to the extracted feature points, the eye tracking algorithm is used to calculate the eye movement trajectory and line of sight direction. The algorithm includes a two-dimensional mapping model and a neural network model; S6. Data analysis: Conduct in-depth analysis of the tracked eye movement data, including calculating the gaze point position and gaze duration, and determining the user's attention direction and potential points of interest.

7. The intelligent AR glasses interaction system according to claim 6, characterized in that: The analyzed eye movement data is fed back in real time to adjust the displayed content and interaction method according to the user's attention direction and interest points.

8. The intelligent AR glasses interaction system according to claim 7, characterized in that: The neural network model collects training data including eye images and corresponding gaze directions. The data is obtained by having the subjects watch the content and simultaneously recording the eye images and gaze directions. The neural network model is trained using the collected training data. The eye images acquired in real time are input into the trained neural network model, and the model will output the corresponding gaze directions and gaze point positions.

9. The intelligent AR glasses interaction system according to claim 1, characterized in that: The invention comprises an AR glasses frame (1), a touch panel module is installed on the surface of the AR glasses frame (1), an AR glasses lens (2) is installed on the inner wall of the AR glasses frame (1), a wearing mechanism (3) is connected to the surface of the AR glasses frame (1), and the wearing mechanism (3) comprises a U-shaped block (30), one side of the U-shaped block (30) is connected to the AR glasses frame (1), a rotating rod (31) is connected inside the U-shaped block (30), a gear column (32) is sleeved on the surface of the rotating rod (31), a long tube (33) is sleeved on the surface of the rotating rod (31), a wearing rod (34) is sleeved on the surface of the long tube (33), and one side of the gear column (32) is meshed with a A tooth plate (35), one side of the tooth plate (35) is connected to a long rod (36), the inner cavity of the long tube (33) is connected to a partition (37), one side of the long rod (36) penetrates the partition (37) and is connected to an inclined plate (38) via a rotating shaft, one side of the inclined plate (38) is connected to a second threaded rod (39) via a rotating shaft, a vertical tube (310) is threadedly sleeved on the surface of the second threaded rod (39), the bottom of the vertical tube (310) is connected to the long tube (33), a long groove (311) is opened on the surface of the wearing rod (34), the top of the vertical tube (310) penetrates the long groove (311), and a locking sleeve (312) is threadedly sleeved on the surface of the vertical tube (310).

Images