Smart glasses device and human-computer interaction method

Abstract

The present invention discloses a smart glasses device and a human-computer interaction method. The smart glasses device comprises: smart interactive glasses, an eye-capturing camera, a server, a touch-sensitive keyboard glasses case, and a touch-sensitive key ring. The eye-capturing camera and the server are provided on the smart interactive glasses; the server is connected to the smart interactive glasses and the eye-capturing camera, respectively; the server is wirelessly connected to the touch-sensitive keyboard glasses case and the touch-sensitive key ring; keys are provided on the inner wall of the touch-sensitive keyboard glasses case; the inner walls of the touch-sensitive keyboard glasses case with keys are hingedly connected; and the touch-sensitive keyboard glasses case is used to input information into the smart interactive glasses. This invention enables users to interact with the smart glasses device naturally, efficiently, and accurately, while avoiding fatigue caused by excessive physical movement.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a smart glasses device and a human-computer interaction method. Background Art

[0002] Smart glasses, including AR and MR glasses, utilize interactive methods that can improve user interaction efficiency and reduce operational effort. Existing graphical human-computer interaction methods for personal computing devices primarily include: 1. Mouse-and-keyboard interaction. This interaction method is commonly used with console computers. The primary implementation involves manually controlling the mouse. The mouse's relative displacement reflects the relative displacement of the on-screen cursor. This is accomplished by combining clicks on the left and right mouse buttons and the scroll wheel, along with keyboard input when necessary, to issue control commands. 2. Touchpad-and-keyboard interaction. This interaction method is commonly used with laptop computers. The primary implementation involves sliding a finger across the touchpad. The relative displacement reflects the relative displacement of the on-screen cursor. This is accomplished by combining clicks on the touchpad's adapted left and right buttons, along with keyboard input when necessary, to issue control commands. 3. Touchscreen interaction. This interaction method is commonly used with smartphones. The primary implementation involves clicking the desired icon, virtual button, or soft keyboard button on the touchscreen to issue control commands. The aforementioned interaction methods are all relatively common human-computer interaction methods and generally require a keyboard or touchscreen. However, due to the portability of smart glasses, they are typically not used with keyboards or touchscreens. Therefore, none of these three interaction methods are suitable for direct use in smart glasses. The details of these three human-computer interaction methods are shown in Table 1.

[0003] Table 1

[0004]

[0005]

[0006] As a new type of personal computing device, smart glasses have great development potential. Existing smart glasses have the following main usage methods: 1. External adapter host type. This method is relatively common. Generally, the head-mounted part of the smart glasses is connected to the host adapted by the manufacturer via a data cable, and the host provides computing power and power supply. 2. External smartphone type. This method generally uses a mobile phone with a type-c interface. The head-mounted part of the smart glasses is connected to the mobile phone via a data cable, and the mobile phone realizes computing power and power supply. 3. All-in-one type. This method integrates the computing unit and battery of the glasses into the head-mounted part. 4. Wireless type. This method uses an external host or server to provide computing power for the smart glasses, similar to the external mobile phone type, but there is no data cable between the two, and they are connected wirelessly. The first two methods require a wired connection when in use, which is not convenient to wear; the integrated method lacks natural and efficient human-computer interaction methods. Gesture recognition is the most common method, but it requires hands and fingers to operate, and typing is very inconvenient; the wireless method also lacks natural and efficient human-computer interaction methods. It requires an external mouse, keyboard or other control devices, which is inconvenient to carry.

[0007] There are currently various human-computer interaction methods for using the aforementioned smart glasses, but no single method has clearly taken the lead. These various methods are all under development. Specifically: 1. Gesture interaction. Taking Microsoft's HoloLens as an example, the primary method involves a user waving their hands in front of the smart glasses' external camera. The smart glasses, using the image captured by the external camera sensor, interpret the operator's gestures as different operating commands, thereby issuing control commands to the machine. 2. Touchpad interaction. Currently, most smart glasses use a touchpad interaction method. For example, Google's Google Glass primarily involves sliding a finger across the connected smart glasses' touchpad or the temples. The relative movement of the sliding movement reflects the relative movement of the on-screen cursor. Combined with clicking the left or right button or pressing the touchpad, control commands are issued. 3. Voice interaction. Some smart glasses use voice interaction, primarily by speaking standard phrases recognized by the device's microphone to issue control commands. 4. Thought interaction. Some conceptual smart glasses utilize a mind-based interaction method. This primarily involves a person thinking specific patterns, which are then detected by the smart glasses' brainwave sensors. These patterns are then interpreted as distinct operating instructions, thereby issuing control commands to the device. However, due to their respective limitations, these human-computer interaction methods, whether used alone or in combination, do not allow for natural and accurate interaction with smart glasses, and therefore cannot be used efficiently.

[0008] In summary, current smart glasses devices are not easy to wear, not easy to carry, and cannot interact naturally, efficiently, and accurately. Summary of the Invention

[0009] Based on this, it is necessary to provide a smart glasses device and a human-computer interaction method so that users can interact with the smart glasses device more naturally, efficiently and accurately.

[0010] To achieve the above object, the present invention provides the following solutions:

[0011] A smart glasses device, comprising: smart interactive glasses, an eyeball shooting camera, a server, a touch keyboard glasses case and a touch button ring;

[0012] The eyeball-capturing camera and the server are arranged on the smart interactive glasses; the server is connected to the smart interactive glasses and the eyeball-capturing camera respectively; the server is wirelessly connected to the touch keyboard glasses box and the touch button ring; buttons are arranged on the inner wall of the touch keyboard glasses box; the inner walls of the touch keyboard glasses box on which the buttons are arranged are hinged; the touch keyboard glasses box is used to input information into the smart interactive glasses.

[0013] Optionally, the touch keyboard glasses case is further provided with a Bluetooth keyboard area; the Bluetooth keyboard area is composed of a plurality of protrusions, and the plurality of protrusions are arranged in an array;

[0014] The smart interactive glasses are also provided with a calibration camera; the calibration camera is connected to the server; the calibration camera is used to photograph the Bluetooth keyboard area.

[0015] Optionally, the touch keyboard glasses case is further provided with a first microprocessor and a first wireless communication module;

[0016] The button is connected to the first microprocessor; the first microprocessor is connected to the server via the first wireless communication module.

[0017] Optionally, a micro projector is further provided on the touch keyboard glasses box; the micro projector is connected to the first microprocessor.

[0018] Optionally, the smart interactive glasses are further provided with a posture sensor and a distance measuring sensor; both the posture sensor and the distance measuring sensor are wirelessly connected to the server.

[0019] Optionally, the touch button ring includes a second microprocessor, a touch ball, a pressure touch switch, an angle sensor and a second wireless communication module;

[0020] The second microprocessor is connected to the server via the second wireless communication module; the angle sensor is connected to the second microprocessor via the pressure touch switch; and the touch ball is arranged in contact with the pressure touch switch.

[0021] Optionally, there are multiple angle sensors; the multiple angle sensors are evenly arranged around the touch ball, and the angle between every two angle sensors is 120 degrees.

[0022] Optionally, the touch keyboard glasses case is further provided with ring charging contacts and glasses charging contacts; the ring charging contacts are used for wired or wireless charging of the touch button ring; the glasses charging contacts are used for wired or wireless charging of the smart interactive glasses.

[0023] Optionally, the smart interactive glasses are AR glasses or MR glasses.

[0024] The present invention also provides a human-computer interaction method, which is used for the above-mentioned smart glasses device; the method comprises:

[0025] Get the coordinates of the human eye's gaze point on the screen;

[0026] determining a change trend of the coordinates;

[0027] Get the information input by the touch keyboard glasses box;

[0028] Get the user's operation on the touch button ring;

[0029] an application program that controls the icon corresponding to the gaze point of the human eye according to the change trend and the operation;

[0030] Control the smart interactive glasses to display the information.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention proposes a smart glasses device and a human-computer interaction method. The smart glasses device comprises: smart interactive glasses, an eye-capturing camera, a server, a touch-sensitive keyboard glasses case, and a touch-sensitive key ring; the eye-capturing camera and the server are provided on the smart interactive glasses; the server is connected to the smart interactive glasses and the eye-capturing camera, respectively; the server is wirelessly connected to the touch-sensitive keyboard glasses case and the touch-sensitive key ring; the inner wall of the touch-sensitive keyboard glasses case is provided with keys; the inner walls of the touch-sensitive keyboard glasses case with keys are hinged; and the touch-sensitive keyboard glasses case is used to input information into the smart interactive glasses. The present invention does not require an external mouse, keyboard, or other control device, is easy to wear and carry, and enables users to interact with the smart glasses device naturally, efficiently, and accurately without excessive physical exertion. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 This is a structural diagram of a smart glasses device according to an embodiment of the present invention;

[0035] Figure 2 This is a structural diagram of the touch keyboard glasses case;

[0036] Figure 3 This is a schematic diagram of the structure of the folding part of the keyboard in the touch keyboard glasses case;

[0037] Figure 4 This is a structural diagram of smart interactive glasses;

[0038] Figure 5 This is a schematic diagram of image coordinate system calibration;

[0039] Figure 6 This is a diagram of the host location inside the touch keyboard glasses case;

[0040] Figure 7 This is a schematic diagram of the structure of the touch button ring;

[0041] Figure 8 This is a diagram of the internal structure of the touch button ring;

[0042] Figure 9 A map of the positions of the gaze points and icons on the screen in a certain state;

[0043] Figure 10 Flowchart of a human-computer interaction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Figure 1 This is a structural diagram of a smart glasses device according to an embodiment of the present invention.

[0047] See also Figure 1 The smart glasses device of the embodiment includes: smart interactive glasses 1, an eyeball shooting camera 4, a server (not shown in the figure), a touch keyboard glasses box 2 and a touch button ring 3.

[0048] The eyeball-capturing camera 4 and the server are arranged on the smart interactive glasses 1; the server is connected to the smart interactive glasses 1 and the eyeball-capturing camera 4 respectively; the server is wirelessly connected to the touch keyboard glasses box 2 and the touch button ring 3; buttons are arranged on the inner wall of the touch keyboard glasses box 2, and the cover of the touch keyboard glasses box is a foldable keyboard 5 when unfolded; the inner walls of the touch keyboard glasses box 2 provided with the buttons are hinged; the inner walls of the touch keyboard glasses box 2 provided with the buttons are hinged; the touch keyboard glasses box 2 is used to input information into the smart interactive glasses 1.

[0049] In this embodiment, the touch keyboard glasses case 2 is small in size and can be carried in a pocket; after the lid of the touch keyboard glasses case 2 is opened, the lid can be unfolded to become a foldable keyboard 5, such as Figure 2 As shown, the folding part of the keyboard is connected by a micro hinge 16, and the surface of the connection part is covered with a skin-friendly material, such as Figure 3 Part (a) of the diagram is enlarged. Figure 3 The key caps are installed in the same way as on a regular keyboard.

[0050] In this embodiment, the primary function of the eye camera 4 is to extract the position of the user's gaze point on the screen and generate a coordinate signal, similar to the position coordinates of a mouse pointer input into a computer. Its operating principle is as follows: the eye camera 4 is aimed at the user's eye and transmits the real-time eye image to a server for image processing, enabling eye tracking and providing various human-computer interaction functions.

[0051] The eye-capturing camera 4 works in conjunction with the server, allowing for quick typing even when a keyboard is inconvenient. The typing input method uses a fuzzy algorithm to determine text input based on visual trajectory. This method is currently used in mobile phone input methods and is a mature method. The only difference between this embodiment and the prior art is that mobile phones use touchscreen sliding as information input, while the smart interactive glasses 1 use eye tracking information captured by the eye-capturing camera 4 as information input.

[0052] In this embodiment, the server includes: a CPU and a memory, the CPU is connected to the memory, the CPU includes an arithmetic unit and a controller, and the memory includes internal memory and external memory. In addition, the server also includes common computing device interfaces and accessories such as a wireless communication module, a USB port, a SIM card port, an SD card port, a battery, a power port, a power cord, a switch button, a restart button, and a casing. These interfaces and accessories are mainly used to enhance the functionality of the device. In this embodiment, the server chip is a Qualcomm Snapdragon 425 (64-bit quad-core).

[0053] In this embodiment, the optical lenses of the smart interactive glasses 1 adopt a large field of view and high transmittance optical module, and support a dual-screen display function, that is, the left and right lenses display different images, which is equivalent to the effect of the human eye, thereby forming a three-dimensional stereoscopic effect.

[0054] As an optional embodiment, the touch keyboard glasses box 2 is further provided with a Bluetooth keyboard area 6, such as Figure 2 As shown, the Bluetooth keyboard area 6 is composed of a plurality of protrusions, and the plurality of protrusions are arranged in an array. The smart interactive glasses 1 are also provided with a calibration camera 7; the calibration camera 7 is connected to the server, as shown in FIG. Figure 4 As shown. The calibration camera 7 is used to photograph the Bluetooth keyboard area 6. The horizontal and vertical arrangement of the Bluetooth keyboard area 6 is equivalent to a picture with features. When the calibration camera 7 on the smart interactive glasses 1 photographs the Bluetooth keyboard area 6, the features of the keyboard in this area can be used to calibrate the image coordinate system for the calibration camera 7, as shown in FIG. Figure 5 shown.

[0055] As an optional embodiment, the touch keyboard glasses box 2 is further provided with a host 8, such as Figure 6 As shown. The host 8 includes a first microprocessor and a first wireless communication module; the button is connected to the first microprocessor; the first microprocessor is connected to the server via the first wireless communication module. The model of the first microprocessor is Intel Z8350.

[0056] As an optional embodiment, a micro projector 9 is further provided on the side of the touch keyboard glasses case 2; the micro projector 9 is connected to the first microprocessor. When the micro projector 9 faces a wall or a screen, the touch keyboard glasses case 2 can provide a projection function.

[0057] As an optional implementation, Figure 4As shown, the smart interactive glasses 1 are also equipped with a posture sensor (not shown) and a distance sensor 10; both the posture sensor and the distance sensor 10 are wirelessly connected to the server. In this embodiment, the posture sensor is a nine-axis sensor that collects the current spatial posture of the smart interactive glasses 1 so that the device can perceive the user's head position. The distance sensor 10 is a laser distance sensor that measures the straight-line distance from the user's head to an obstacle directly in front of them.

[0058] As an optional embodiment, the touch button ring 3 is in the shape of a finger ring, such as Figure 7-Figure 8 As shown, the touch button ring 3 includes a second microprocessor, a touch ball 11, a pressure switch 12, an angle sensor 13, a second wireless communication module, and a housing support. The second microprocessor is connected to the server via the second wireless communication module; the angle sensor is connected to the second microprocessor via the pressure switch. The touch ball is placed in contact with the pressure switch, providing pressure control, which can trigger single-click, double-click, and long-press operations. Multiple angle sensors are provided; these multiple angle sensors are evenly arranged around the touch ball, with the angle between each two angle sensors being 120 degrees, enabling 720-degree rotation control of the touch ball, thereby realizing two-dimensional and three-dimensional interaction. The second microprocessor is placed within the curved housing using an FPC solution. The second microprocessor is a single-chip microcontroller from the STM32 or AVR series of IFS Semiconductor.

[0059] As an optional embodiment, the touch keyboard glasses box 2 is further provided with a battery, a ring charging contact 14 and a glasses charging contact 15, such as Figure 2 As shown, the ring charging contacts 14 and the glasses charging contacts 15 are both connected to the battery; the battery is equivalent to a power bank, which performs wired or wireless charging for the touch button ring 3 through the ring charging contacts 14, and performs wired or wireless charging for the smart interactive glasses 1 through the glasses charging contacts 15.

[0060] As an optional implementation, the smart interactive glasses 1 are AR glasses or MR glasses.

[0061] As an optional embodiment, the first wireless communication module and the second wireless communication module include but are not limited to Bluetooth, WiFi, 4G module and 5G module. Preferably, the second wireless communication module is Bluetooth, and the touch button ring 3 is connected to the smart interactive glasses 1 via Bluetooth, which has low power consumption.

[0062] The smart glasses device of this embodiment does not require an external mouse, keyboard or other control devices, is easy to wear and carry, and enables users to interact with the smart glasses device naturally, efficiently and accurately.

[0063] The present invention also provides a human-computer interaction method, which is used for the above-mentioned smart glasses device. Figure 10 , the method comprising:

[0064] Step S1: Obtain the coordinates of the human eye's gaze point on the screen.

[0065] Step S2: Determine the change trend of the coordinates.

[0066] Step S3: Acquire information inputted by the touch keyboard glasses case.

[0067] Step S4: Obtain the user's operation on the touch button ring.

[0068] Step S5: controlling the application of the icon corresponding to the gaze point of the human eye according to the change trend and the operation.

[0069] Step S6: Control the smart interactive glasses to display the information. Figure 9 shown.

[0070] As an optional implementation, the method further includes: before obtaining the coordinates of the human eye gaze point on the screen, obtaining image information obtained by a calibration camera shooting the Bluetooth keyboard area, and realizing image coordinate system calibration through the captured image information.

[0071] As an optional implementation, the method further includes: obtaining the current spatial posture information of the smart interactive glasses collected by the posture sensor and the straight-line distance from the user's head to the obstacle directly in front collected by the ranging sensor, so as to perceive the user's head state and the environmental state in real time.

[0072] The interaction between the touch button ring 3, the touch keyboard glasses box 2, and the smart interactive glasses 1 is shown in Table 2. Specifically:

[0073] (1) If the human eye gaze moves to the target and clicks the touch ball on the touch button ring 3, hovering can be achieved just like moving the mouse cursor to the target.

[0074] (2) If the human eye moves to the target and quickly clicks the touch ball twice on the touch button ring 3, the application or button can be activated just like double-clicking the left button of a mouse.

[0075] (3) If a person looks at an icon and long-presses the touch ball on the touch button ring 3, an additional menu for the application or button can be opened just like a right-click of a mouse.

[0076] (4) If a person fixates on an icon, clicks the touch ball on the button ring 3, and moves the fixation point, the desktop position of the icon can be rearranged just like dragging with the left button of a mouse.

[0077] (5) If a person rolls the touch ball up and down or left and right on the touch button ring 3, the picture can be flipped up and down like a mouse wheel.

[0078] (6) If a person clicks the touch ball on the touch button ring 3 and rolls the touch ball, zooming in and out can be achieved just like hovering a mouse and turning the scroll wheel.

[0079] (7) If a person double-clicks the touch ball on the touch button ring 3 and rolls the touch ball, a three-dimensional flip can be achieved just like pressing the scroll wheel of a mouse and moving the cursor.

[0080] (8) The eyeball camera 4 cooperates with the server to judge text input based on the visual trajectory based on the fuzzy algorithm, so that information can be input like a physical keyboard.

[0081] (9) The user inputs information through the wireless touch keyboard glasses box 2 that can be carried in the pocket, realizing the input of information just like a physical keyboard.

[0082] Table 2

[0083]

[0084]

[0085] Compared with existing interaction methods, the human-computer interaction method of this embodiment can enable users to interact with smart glasses devices naturally, efficiently and accurately.

[0086] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0087] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A smart glasses device, characterized in that: include: Smart interactive glasses, eyeball capture cameras, servers, touch keyboard glasses cases, and touch button rings; The eyeball shooting camera and the server are arranged on the smart interactive glasses; The server is connected to the smart interactive glasses and the eyeball capture camera respectively; the server is wirelessly connected to the touch keyboard glasses case and the touch button ring; buttons are provided on the inner wall of the touch keyboard glasses case; the inner walls of the touch keyboard glasses case provided with the buttons are hinged; the touch keyboard glasses case is used to input information into the smart interactive glasses; The touch keyboard glasses case is also provided with a Bluetooth keyboard area; the Bluetooth keyboard area is composed of a plurality of protrusions, and the plurality of protrusions are arranged in an array; The smart interactive glasses are further provided with a calibration camera; the calibration camera is connected to the server; the calibration camera is used to photograph the Bluetooth keyboard area; When the calibration camera captures the Bluetooth keyboard area, the characteristics of the keyboard in the Bluetooth keyboard area are used to calibrate the image coordinate system for the calibration camera; The smart interactive glasses are also provided with a posture sensor and a distance sensor; both the posture sensor and the distance sensor are wirelessly connected to the server; the posture sensor is a nine-axis sensor, which uses the nine-axis sensor to collect the current spatial posture of the smart interactive glasses for sensing the user's head state; the distance sensor is a laser distance sensor, which is used to measure the straight-line distance from the user's head to the obstacle directly in front; The touch button ring includes a second microprocessor, a touch ball, a pressure touch switch, an angle sensor and a second wireless communication module; The second microprocessor is connected to the server via the second wireless communication module; the angle sensor is connected to the second microprocessor via the pressure touch switch; and the touch ball is arranged in contact with the pressure touch switch.

2. The smart glasses device according to claim 1, characterized in that: The touch keyboard glasses case is also provided with a first microprocessor and a first wireless communication module; The button is connected to the first microprocessor; the first microprocessor is connected to the server via the first wireless communication module.

3. The smart glasses device according to claim 2, characterized in that: The touch keyboard glasses box is also provided with a micro projector; the micro projector is connected to the first microprocessor.

4. The smart glasses device according to claim 1, wherein: There are multiple angle sensors; the multiple angle sensors are evenly arranged around the touch ball, and the angle between every two angle sensors is 120 degrees.

5. The smart glasses device according to claim 3, characterized in that: The touch keyboard glasses case is also provided with ring charging contacts and glasses charging contacts; the ring charging contacts are used to perform wired or wireless charging for the touch button ring; the glasses charging contacts are used to perform wired or wireless charging for the smart interactive glasses.

6. The smart glasses device according to claim 1, characterized in that: The smart interactive glasses are AR glasses or MR glasses.

7. A human-computer interaction method, characterized in that: The method is used for the smart glasses device according to any one of claims 1 to 6; the method comprises: Get the coordinates of the human eye's gaze point on the screen; determining a change trend of the coordinates; Get the information input by the touch keyboard glasses box; Get the user's operation on the touch button ring; an application program that controls the icon corresponding to the gaze point of the human eye according to the change trend and the operation; Control the smart interactive glasses to display the information.

Images