Intelligent glasses touch control method and system

Abstract

The application relates to a smart glasses touch control method and system, and the control method comprises the following steps: step S1, by using a touchpad, establishing a corresponding relationship between the coordinates of the touchpad and the coordinates of smart glasses in proportion; step S2, adding a camera beside the touchpad to identify the position of the finger midpoint on the touchpad; step S3, according to the proportion of the touchpad coordinates and the smart glasses coordinates, displaying the current finger midpoint in the smart glasses picture by using a color arrow; step S4, judging whether the finger touches the touchpad, if yes, executing step S5, and if not, executing step S6; step S5, switching the icon of the touch position from the finger midpoint position displayed by the color arrow in the smart glasses picture and triggering the event of the finger touch position; step S6, displaying the moving track in a set time period by using a curve with a color and an opposite color edge. The application realizes that the specific position of the finger on the smart glasses is known when the touchpad is touched.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart glasses, in particular to a smart glasses touch control method and system. BACKGROUND

[0002] The smart glasses have an independent operating system, and the smart glasses can install programs provided by software service providers such as software and games. The smart glasses can complete functions such as adding schedules, map navigation, interacting with friends, taking photos and videos, and video calling with friends through voice or action control, and can realize wireless network access through a mobile communication network, thereby bringing great convenience to people's life.

[0003] Since the volume of the smart glasses is usually small, many operation components cannot be arranged on the glasses body for user operation. At present, the smart glasses can only realize interaction through a customized app function through a touch panel, or need to use a Bluetooth mouse and keyboard to realize interaction with a general Android app. The smart glasses and the touch panel are similar to a mobile phone with a separated touch screen. When touching, the user does not know the specific position of the finger.

[0004] At this time, the user can only use a Bluetooth mouse to operate. For example, the user installs WeChat in the mobile phone and wants to click on a person. Since the position of the finger is unknown, the user cannot select the right person to send a message to. Generally, the smart glasses provide a special app, such as a desktop function, which is switched by left and right sliding touch. If the user wants to use the desktop of the mobile phone, the user cannot operate. If the user wants to use WeChat, the user needs to specially make a customized version of WeChat to support gesture operation, which is not possible. If the user wants to use the general version of WeChat on the smart glasses, the user can only use a Bluetooth mouse to operate, which is very inconvenient. The smart glasses already have a touch panel, and the user still needs to carry a Bluetooth mouse. SUMMARY

[0005] The present application aims to provide a smart glasses touch control method and system, which can know the specific position of the finger on the smart glasses when the touch panel is touched.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a smart glasses touch control method, the control method comprising the following steps:

[0007] Step S1: establishing a corresponding relationship between the coordinates of the touch panel and the coordinates of the smart glasses in proportion by using the touch panel;

[0008] Step S2: installing a camera beside the touch panel to identify the position of the finger midpoint on the touch panel;

[0009] Step S3, according to the proportion of the touchpad coordinates and the smart glasses coordinates, the current finger midpoint is displayed in the smart glasses picture using a color arrow;

[0010] Step S4, judging whether the finger touches the touchpad, if yes, executing step S5, if not, executing step S6;

[0011] Step S5, in the smart glasses picture, the icon of the finger midpoint position switching to the touch position is displayed by a color arrow and the event of the finger touch position is triggered;

[0012] Step S6, the moving track in the set time period is displayed using a curve with a color and an opposite color edge.

[0013] Further, the position of the finger midpoint on the touchpad in step S2 is identified by establishing a model through a convolutional neural network, wherein the input of the first convolutional layer is the rgb pixel component value of the incoming finger picture, the rgb value of each pixel is substituted into a multi-layer convolutional model for iterative calculation to obtain the estimated finger key position, and a set of optimal model parameters are finally obtained by comparing the estimated finger key position with the sample key position, so that the most possible finger midpoint position can be obtained by inputting the rgb value of each pixel; the model is trained through the convolutional neural network and pictures with labeled finger midpoints.

[0014] Further, step S3 is specifically establishing a mapping relationship by collecting the four fixed point positions of the touchpad when the finger touches the touchpad and the position of the finger center point recognized by the camera in the camera display picture, obtaining the position of the finger through the mapping relationship, and displaying the current finger midpoint in the smart glasses picture using a color arrow.

[0015] Further, the color arrow in step S3 is displayed using a plurality of color circles nested.

[0016] Further, the switching in step S5 is achieved by changing the circle color of the color arrow.

[0017] Further, the set time period in step S6 is 2-5 seconds.

[0018] The application further provides a smart glasses touch control system, which comprises a proportional correspondence module, a camera recognition module, a display module, a judgment module, a touch module and a moving track module; the proportional correspondence module is responsible for establishing a correspondence between the coordinates of a touchpad and the coordinates of smart glasses in proportion by using the touchpad; the camera recognition module is responsible for recognizing the position of a finger midpoint on the touchpad by adding a camera beside the touchpad; the display module is responsible for displaying the current finger midpoint by using a color arrow in the smart glasses picture according to the proportion of the touchpad coordinates and the smart glasses coordinates; the judgment module is responsible for judging whether the finger touches the touchpad, if yes, the touch module is executed, and if not, the moving track module is executed; the touch module is responsible for switching the position of the finger midpoint displayed by the color arrow in the smart glasses picture into an icon of a touch position and triggering an event of the touch position; and the moving track module is responsible for displaying the moving track in a set time period by using a curve with a color and an opposite color edge.

[0019] Further, the camera recognition module recognizes the position of the finger midpoint on the touchpad by establishing a model through a convolutional neural network, wherein the input of the first convolutional layer is the rgb pixel component value of the input finger picture, the rgb value of each pixel is substituted into a multi-layer convolutional model for iterative calculation to obtain an estimated finger key position, and a group of optimal model parameters are finally obtained by comparing the estimated finger key position with a sample key position, so that the most possible finger midpoint position can be obtained by inputting the rgb value of each pixel of the shot picture; the model is trained by the convolutional neural network and pictures with labeled finger midpoints.

[0020] Further, the display module specifically establishes a mapping relationship by collecting the four fixed point positions of the touchpad when the finger touches the touchpad and the position of the finger center point recognized by the camera and displayed in the picture by the camera, obtains the position of the finger through the mapping relationship, and displays the current finger midpoint by using a color arrow in the smart glasses picture.

[0021] Further, the color arrow in the display module is displayed by using a plurality of color circles.

[0022] Further, the switching in the touch module is realized by changing the circle color of the color arrow.

[0023] Further, the set time period in the moving track module is 2-5 seconds.

[0024] The application has the advantages that the application can know the specific position of the finger on the smart glasses when the touchpad is touched. Since the camera is beside the touchpad and the shooting distance is short, a low-pixel camera can be used to recognize gestures, the cost is reduced, and the application can be completely compatible with the existing Android application interaction without additional finger devices such as a ring. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flow chart of the method of the present application;

[0026] Figure 2 is a schematic diagram of the module connection in the system of the present application. DETAILED DESCRIPTION

[0027] The present application will be further described below with reference to the accompanying drawings.

[0028] Referring to Figure 1 The control method comprises the following steps:

[0029] Step S1, by using a touchpad, establishing a corresponding relationship between the coordinates of the touchpad and the coordinates of the smart glasses in proportion;

[0030] Step S2, installing a camera next to the touchpad to identify the position of the finger midpoint on the touchpad;

[0031] The camera needs to be installed next to the touchpad to collect at close range, so the pixel requirement for the camera is not so high.

[0032] Step S3, according to the proportion of the touchpad coordinates and the smart glasses coordinates, displaying the current finger midpoint in the smart glasses picture using a color arrow;

[0033] Step S4, judging whether the finger touches the touchpad, if yes, executing step S5, if not, executing step S6;

[0034] Step S5, switching the icon of the finger midpoint position displayed by the color arrow in the smart glasses picture to the touch position and triggering the event of the finger touch position;

[0035] Step S6, displaying the moving track in the set time period using a curve with a color and an opposite color edge.

[0036] The position of the finger midpoint on the touchpad in step S2 is identified by establishing a model through a convolutional neural network, wherein the input of the first convolutional layer is the rgb pixel component value of the incoming finger picture, by substituting the rgb value of each pixel into a multi-layer convolutional model, iterative calculation is performed to obtain the estimated finger key position, and a set of optimal model parameters are finally obtained by comparing the estimated finger key position with the sample key position, and the most possible finger midpoint position can be obtained by inputting the rgb value of each pixel of the captured picture; the model is trained by a convolutional neural network in cooperation with pictures with labeled finger midpoints.

[0037] The step S3 is specifically establishing a mapping relationship by collecting four fixed point positions of the touchpad when the finger touches the touchpad and the position of the finger center point in the camera display screen recognized by the camera, obtaining the position of the finger through the mapping relationship, and displaying the current finger center point in the smart glasses screen using a color arrow.

[0038] The color arrow in the step S3 is displayed using a plurality of color circles nested.

[0039] The switching in the step S5 is achieved by changing the circle color of the color arrow.

[0040] The setting time period in the step S6 is 2-5 seconds.

[0041] The finger touch event remains unchanged, so that the same interactive mode as the current mobile phone touch screen can be realized to operate the app in the AR glasses; before the present scheme, the finger operation of the app triggers its own event, for example, if you double-click to enlarge the photo, the photo will be enlarged, and the present scheme is only to let you know the position of the finger on the screen before the finger touches the screen.

[0042] The present application will be further described in combination with a specific embodiment:

[0043] A smart glasses touch control method, the control method comprising the following steps:

[0044] Step S1, by using a touchpad, establishing a corresponding relationship between the coordinates of the touchpad and the smart glasses coordinates according to a proportion;

[0045] For example: the display coordinates of the AR glasses are (0, 0) to (1920, 1080), the touch coordinates of the touchpad are (0, 0) to (960, 540), and the midpoint coordinates of the touch are (1, 1), so the corresponding display coordinates are (2, 2).

[0046] Step S2, installing a camera beside the touchpad to identify the position of the finger midpoint on the touchpad;

[0047] The camera needs to be installed beside the touchpad for close-range collection, so the pixel requirement of the camera is not so high.

[0048] Step S3, according to the proportion of the touchpad coordinates and the smart glasses coordinates, displaying the current finger midpoint in the smart glasses screen using a color arrow;

[0049] When the finger is hovering at the touchpad coordinate (1,1), a color arrow will be displayed in the center of the glasses screen, which is highlighted. The color arrow can be displayed in a single color or in multiple colors, such as red, yellow, blue, and black. The top point of the color arrow is the midpoint of the finger.

[0050] Step S4: Determine whether the finger is touching the touchpad. If yes, execute step S5. If no, execute step S6.

[0051] Step S5: In the smart glasses screen, display the icon of the touch position of the finger midpoint and trigger the event of the touch position of the finger.

[0052] When the finger is touching the touchpad (10,10), the display icon of the touch position will change color, allowing the operator to see the click position. The icon can also display a water wave-like contraction effect, making it more obvious to see the position of the operator's click. Then, the event of touching (10,10) is sent to the system without any processing. For example, if there is a button at position (10,10), pressing it will trigger the event of pressing the button, just like without using this scheme. This scheme only determines the position of the finger midpoint in the smart glasses lens when the finger is touching the touchpad, without processing the application of the smart glasses itself.

[0053] Step S6: Display the movement trajectory in the last 3 seconds using a curve with a red edge and an opposite color edge.

[0054] When the finger is close to the touchpad but not touching it, it moves from (1,1) to (100,100). Then, the touch trajectory from (1,1) to (100,100) is drawn according to the positions passed by the finger. If the finger is at (1,1) at the 0th second, at (10,10) at the 1st second, at (20,20) at the 2nd second, at (30,30) at the 3rd second, and at (40,40) at the 4th second, the touch trajectory from (1,1) to (10,10) will disappear. The touch trajectory of the last three seconds will also disappear. The time can be adjusted. If the time is too long, there will be too many trajectories on the screen, making it confusing. If the time is too short, the trajectory will disappear before it can be seen. Similar to the mouse click, the left mouse button is pressed and dragged. The drag trajectory is displayed.

[0055] Please refer to Figure 2As shown, the application also provides a smart glasses touch control system, which comprises a proportional correspondence module, a camera recognition module, a display module, a judgment module, a touch module and a moving track module; the proportional correspondence module is responsible for establishing a correspondence between the coordinates of a touchpad and the coordinates of smart glasses in proportion by using the touchpad; the camera recognition module is responsible for recognizing the position of a finger midpoint on the touchpad by adding a camera beside the touchpad; the display module is responsible for displaying the current finger midpoint in the smart glasses picture by using a color arrow according to the proportion of the touchpad coordinates and the smart glasses coordinates; the judgment module is responsible for judging whether the finger touches the touchpad, if yes, executing the touch module, if not, executing the moving track module; the touch module is responsible for switching the icon of the touch position from the color arrow display of the finger midpoint position in the smart glasses picture and triggering the event of the finger touch position; the moving track module is responsible for displaying the moving track in a set time period by using a curve with a color and an opposite color edge.

[0056] The camera recognition module recognizes the position of the finger midpoint on the touchpad by establishing a model through a convolutional neural network, wherein the input of the first convolutional layer is the rgb pixel component value of the input finger picture, the rgb value of each pixel is substituted into a multi-layer convolutional model for iterative calculation to obtain the estimated finger key position, and a set of optimal model parameters are finally obtained by comparing the estimated finger key position with the sample key position, so that the most possible finger midpoint position can be obtained by inputting the rgb value of each pixel; the model is trained by the convolutional neural network and pictures with labeled finger midpoints.

[0057] The display module specifically establishes a mapping relationship by collecting the four fixed point positions of the touchpad when the finger touches the touchpad and the position of the finger center point recognized by the camera in the camera display picture, obtains the position of the finger through the mapping relationship, and displays the current finger midpoint in the smart glasses picture by using a color arrow.

[0058] The color arrow in the display module is displayed by using a plurality of color circles.

[0059] The switching in the touch module is realized by changing the circle color of the color arrow.

[0060] The set time period in the moving track module is 2-5 seconds.

[0061] In summary, by using the touchpad, the touchpad coordinates and the smart glasses coordinates are proportionally established in correspondence; a camera is additionally installed beside the touchpad to identify the position of the finger midpoint on the touchpad; in step S3, according to the proportion of the touchpad coordinates and the smart glasses coordinates, the current finger midpoint is displayed in the smart glasses picture by using color arrows; it is judged whether the finger touches the touchpad; if yes, the icon of the touch position is switched from the finger midpoint position displayed by the color arrows in the smart glasses picture and the event of the finger touch position is triggered; if no, a curve with a color and an opposite color edge is used to display the moving track in a set time period, so that the specific position of the finger on the smart glasses is known when the touchpad is touched, and the interaction of the existing Android application can be completely compatible.

[0062] The above merely describes the preferred embodiments of the present application and should not be construed as limiting the present application. Any equivalent changes and modifications made within the scope of the present application should be included in the scope of the present application.

Claims

1. A touch control method for smart glasses, characterized in that: The control method includes the following steps: Step S1: Using the touchpad, establish a proportional correspondence between the coordinates of the touchpad and the coordinates of the smart glasses. Step S2: Install a camera next to the touchpad to identify the position of the finger's midpoint on the touchpad; Step S3: Based on the ratio of the touchpad coordinates to the smart glasses coordinates, display the current finger midpoint on the smart glasses screen using a colored arrow; Step S4: Determine whether the finger touches the touchpad. If yes, proceed to step S5; otherwise, proceed to step S6. Step S5: On the smart glasses screen, an icon indicating the switch from the midpoint of the finger to the touch position is displayed by a colored arrow, and the finger touch position event is triggered. Step S6: Display the movement trajectory within a set time period using a curve with a color and an opposite edge. In step S2, identifying the position of the finger's midpoint on the touchpad is achieved by building a model using a convolutional neural network. The input to the first convolutional layer is the RGB pixel component value of the input finger image. By substituting the RGB value of each pixel into the multi-layer convolutional model and performing iterative calculations, the estimated finger's center point position is obtained. This is compared with the sample center point position to finally obtain a set of optimal model parameters. The most probable finger midpoint position can be obtained by inputting the RGB value of each captured pixel. The model is trained using a convolutional neural network in conjunction with images labeled with the finger's midpoint. Step S3 specifically involves collecting the four fixed positions of the touchpad when the finger touches it and the position of the finger's center point as seen by the camera in the image displayed on the screen. A mapping relationship is established, and the finger's position is obtained through this mapping relationship. The current finger's midpoint is then displayed on the smart glasses screen using a colored arrow.

2. The smart glasses touch control method according to claim 1, characterized in that: The color arrows in step S3 are displayed using nested circles of multiple colors.

3. The smart glasses touch control method according to claim 1, characterized in that: The switching in step S5 is achieved by changing the color of the circle of the color arrow.

4. The smart glasses touch control method according to claim 1, characterized in that: The time period set in step S6 is 2-5 seconds.

5. A smart glasses touch control system, characterized in that: It includes a ratio mapping module, a camera recognition module, a display module, a judgment module, a touch module, and a movement trajectory module; the ratio mapping module is responsible for establishing a proportional correspondence between the coordinates of the touchpad and the coordinates of the smart glasses by using the touchpad; the camera recognition module is responsible for installing a camera next to the touchpad to identify the position of the finger's midpoint on the touchpad; The display module is responsible for displaying the current finger midpoint using a colored arrow on the smart glasses screen according to the ratio of the touchpad coordinates and the smart glasses coordinates; the judgment module is responsible for determining whether the finger has touched the touchpad. If so, the touch module is executed; otherwise, the movement trajectory module is executed; the touch module is responsible for displaying an icon indicating the finger midpoint position has switched to the touch position on the smart glasses screen using a colored arrow and triggering an event indicating the finger touch position; the movement trajectory module is responsible for displaying the movement trajectory within a set time period using a curve with a different color and a contrasting edge. The camera recognition module identifies the position of the finger's midpoint on the touchpad by building a model using a convolutional neural network. The first convolutional layer takes the RGB pixel component values ​​of the input finger image as input. By substituting the RGB value of each pixel into the multi-layer convolutional model and performing iterative calculations, an estimated finger position is obtained. This estimated position is then compared with the sample position to obtain a set of optimal model parameters. By inputting the RGB value of each captured pixel, the most probable finger midpoint position can be obtained. The model is trained using a convolutional neural network and an image with the finger midpoint labeled. Specifically, the display module establishes a mapping relationship by collecting the four fixed positions of the touchpad when a finger touches it and the position of the center point of the finger as detected by the camera in the image displayed by the camera. Through the mapping relationship, the position of the finger is obtained, and the current center point of the finger is displayed on the smart glasses screen using a colored arrow.

6. The smart glasses touch control system according to claim 5, characterized in that: The color arrows in the display module are displayed using nested circles of multiple colors.

7. The smart glasses touch control system according to claim 5, characterized in that: The switching in the touch module is achieved by changing the color of the circle of the color arrow.

8. The smart glasses touch control system according to claim 5, characterized in that: The time period set in the movement trajectory module is 2-5 seconds.

Images