Neural network systems for posture, wearing, activity or carrying detection on wearable or mobile devices

Abstract

A neural network system includes an eye-mounted device. The eye-mounted device has a motion tracker, such as an accelerometer, a gyroscope, or an inertial measurement unit for measuring acceleration and rotation. The neural network system tracks the motion of the eye-mounted device from at least one finger contact input by a user on an input surface through the motion tracker. The neural network system recognizes a finger gesture by detecting at least one detected touch event based on a change in the motion of the eye-mounted device tracked over a period of time. The neural network system adjusts the image presented on the image display of the eye-mounted device based on the recognized finger gesture. The neural network system can also detect whether the user is wearing the eye-mounted device and recognize the activity of the user wearing the eye-mounted device based on the change in the motion tracked over the period of time.

Description

[0001] Citation of Related Applications

[0002] This application claims the benefit of Provisional Application No. 62 / 738,093, filed in the United States on September 28, 2018, entitled "Neural Network System for Posture, Wear, Activity, or In-Position Detection on a Wearable or Mobile Device," the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present subject matter relates to wearable devices (e.g., eye-mounted devices), mobile devices, and techniques for detecting user gestures and wear detection.

[0004] background

[0005] Computing devices such as wearable devices including portable eyewear (such as smart glasses, headgear and headsets), collars, smart watches, and mobile devices (such as tablets, smartphones and laptops) have integrated image displays and cameras. Because the image display area available on wearable devices and mobile devices is small, it can be difficult to view and interact with the displayed content on the device.

[0006] For example, the size limitations and form factors of wearable devices such as eye-mounted devices can make it difficult to incorporate a user interface into the eye-mounted device. The available area for placing various control buttons (e.g., for operating a camera) on the eye-mounted device and graphical user interface elements on the image display of the eye-mounted device is limited. For example, because the form factor of the eye-mounted device is small, it is cumbersome to manipulate and interact with content displayed on the image display.

[0007] Therefore, users need to easily interact with wearable devices (including eye-mounted devices) and mobile devices.

[0008] Brief description of the attached figure

[0009] The accompanying drawings illustrate one or more embodiments by way of example only and not by way of limitation.In the accompanying drawings, like reference numerals represent like or similar elements.

[0010] Figure 1A is a right side view of an example hardware configuration for an eye-mounted device that includes a motion tracker, used in a neural network system for posture, wear, and activity detection.

[0011] Figure 1B and 1C is a rear view of an example hardware configuration of an eye-mounted device including two different types of image displays.

[0012] Figure 1DA schematic diagram depicting operation of an eye-mounted device motion tracker coupled to an input surface to track the eye-mounted device motion of at least one finger contact input from a user.

[0013] Figure 2 yes Figure 1B and Figure 1C A cross-sectional top view of the left block of the eyewear device, depicting the motion tracker and circuit board.

[0014] Figure 3A is a high-level functional block diagram of an exemplary neural network system, including an eye-mounted device with a motion tracker to recognize finger gestures based on a neural network model, a mobile device, and a server system connected via various networks.

[0015] Figure 3B Shown for Figure 3A An example of the hardware configuration of the neural network system server system, which establishes a neural network model for recognizing finger gestures in the form of a simplified block diagram.

[0016] Figure 3C is a high-level functional block diagram of an exemplary neural network system including an eye-mounted device with a motion tracker for detecting wearable motion based on a neural network model, a mobile device, and a server system connected via various networks.

[0017] Figure 3D Shown for Figure 3B An example of the hardware configuration of the neural network system server system, which establishes a neural network model for wearable detection in the form of a simplified block diagram.

[0018] Figure 3E is a high-level functional block diagram of an exemplary neural network system including an eye-mounted device with a motion tracker connected via various networks, a mobile device, and a server system, the motion tracker identifying activities based on a neural network model while the eye-mounted device is worn.

[0019] Figure 3F Shown for Figure 3E An example of the hardware configuration of the neural network system server system of FIG. 1 is provided in the form of a simplified block diagram to establish a neural network model for identifying activities while wearing an eye-mounted device.

[0020] Figure 4A Shown for Figure 3A -B's neural network system Hardware configuration examples for wearable devices or mobile devices, including motion trackers, to recognize finger gestures based on neural network models.

[0021] Figure 4B Shown for Figure 3C-D A neural network system hardware configuration example of a wearable device or mobile device, including a motion tracker, to detect the wearing of a wearable device or the carrying of a mobile device based on a neural network model.

[0022] Figure 4C Shown for Figure 3E -F Neural Network Systems Hardware configuration examples for wearable or mobile devices, including motion trackers, to recognize activities based on neural network models.

[0023] Figure 5 Yes, you can Figure 3B , 3D Flowchart of a method implemented in the neural network system 300 of 3F to optimize the execution speed and efficiency of the neural network model generated for posture, wearable, carrying or activity recognition.

[0024] Fig. 6A , 6B 6C show a touch event detected by pressing and holding on the input surface of the eye-mounted device.

[0025] Figure 7 Touch events detected by pinching and releasing fingers on the input surface of the eyewear device are shown.

[0026] Figure 8 Shows touch events detected by finger rotation on the input surface of the eye-mounted device.

[0027] Fig. 9 Shows a touch event detected by sliding a finger on the input surface of the eye-mounted device.

[0028] Detailed description

[0029] In the following detailed description, many specific details are described by way of example in order to provide a thorough understanding of the relevant teachings. However, it is obvious to those skilled in the art that the present teachings can be implemented without these details. In other examples, in order to avoid unnecessarily obscuring various aspects of the present teachings, well-known methods, procedures, components, and circuits will be described herein at a relatively high level without elaborating on the details.

[0030] As used herein, the terms "coupled" or "connected" refer to any logical, optical, physical, or electrical connection, etc., through which an electrical or magnetic signal generated or provided by one system element is transmitted to another coupled or connected element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or propagation media that can modify, manipulate, or carry electrical signals. The term "on" refers to being directly supported by an element or indirectly supported by an element through another element that is integrated into or supported by the element.

[0031] For purposes of illustration and discussion, orientations of eye-mounted devices, related components, and any complete device incorporating a motion tracker as shown in any of the accompanying drawings are given by way of example only. In operation for gesture, wear, or activity detection, the eye-mounted device may be oriented in any other orientation suitable for the specific application of the eye-mounted device, such as upward, downward, sideways, or any other orientation. Furthermore, within the scope used herein, any directional terms such as front, back, inward, outward, toward, left, right, lateral, longitudinal, upward, downward, top, bottom, sideways, horizontal, vertical, and diagonal are used by way of example and are not limited to the orientation or orientation of any motion tracker or motion tracker component constructed as otherwise described herein.

[0032] Other purposes, advantages and novel features of the examples will be described in part in the following description, and will become apparent to those skilled in the art after viewing the following drawings, or may be learned by the production or operation of the examples. The purposes and advantages of the subject matter may be realized and obtained by the methods, means and combinations particularly pointed out in the appended claims.

[0033] Reference is now made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0034] Figure 1A 1 is a right side view of an example hardware configuration of an eye-mounted device 100 including a motion tracker (not shown) for use in a neural network system for gesture, wear, and activity detection. The eye-mounted device 100 includes a right optical assembly 180B having an image display for presenting an image. Figure 1A As shown, the eye-mounted device 100 includes a right visible light camera 114B. The eye-mounted device 100 may include a plurality of visible light cameras 114A-B, which form a passive type of depth capture camera, such as a stereo camera, wherein the right visible light camera 114B is located on the right block 110B. The eye-mounted device 100 may also include a side left visible light camera 114A on the left block 110A. The depth capture camera may be an active type depth capture camera including a single visible light camera 114B and a depth sensor (e.g., an infrared camera and an infrared emitter).

[0035] The left and right visible light cameras 114A-B are sensitive to wavelengths in the visible light range. Each of the visible light cameras 114A-B has a different forward-facing field of view that overlap to allow the generation of a three-dimensional depth image, for example, the right visible light camera 114B has a right field of view 111B as described. In general, a "field of view" is the portion of a scene that is visible through a camera at a specific position and direction in space. When an image is captured by a visible light camera, objects or object features outside the field of view 111A-111B are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range over which the image sensors of the visible light cameras 114A-B receive electromagnetic radiation from a given scene in a captured image of the given scene. The field of view can be expressed as the angular size of the cone of view, i.e., the viewing angle. The viewing angle can be measured horizontally, vertically, or diagonally.

[0036] In an example, the field of view angle of visible light camera 114A-B is between 15° and 30°, such as 24°, and has a resolution of 480x480 pixels. The "field of view angle" describes the range of angles that the lens of visible light camera 114A-B or infrared camera can effectively image. Typically, the image circle produced by the camera lens is large enough to completely cover the film or sensor, possibly including some vignetting towards the edges. If the field of view angle of the camera lens does not fill the sensor, the image circle will be visible, typically with strong vignetting towards the edges, and the effective viewing angle will be limited to the field of view angle.

[0037] Examples of such visible light cameras 114A-B include high resolution complementary metal oxide semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640×480 pixels, for a total of .30 megapixels), 720p, or 1080p. The term "overlap" as used herein with respect to the field of view means that the pixel matrices of the raw images or infrared images generated by the scene overlap by 30% or more. The term "substantially overlap" as used herein with respect to the field of view means that the pixel matrices of the raw images or infrared images generated by the scene overlap by 50% or more.

[0038] Image sensor data from visible light cameras 114A-B is captured along with geolocation data, digitized by an image processor, and stored in memory. The left and right raw images captured by the corresponding visible light cameras 114A-B are in a two-dimensional spatial domain and include a matrix of pixels on a two-dimensional coordinate system that includes an X-axis for horizontal position and a Y-axis for vertical position. Each pixel includes a color attribute (e.g., a red pixel light value, a green pixel light value, and / or a blue pixel light value) and a position attribute (e.g., an X position coordinate and a Y position coordinate).

[0039] To provide stereoscopic vision, visible light cameras 114A-B may be coupled to an image processor ( Figure 3A Component 312) for digital processing and time stamping of captured scene images. Image processor 312 includes circuits for receiving signals from visible light cameras 114A-B and processing those signals from visible light cameras 114 into a format suitable for storage in memory. The timestamp can be added by an image processor or other processor that controls the operation of visible light cameras 114A-B. Visible light cameras 114A-B allow depth capture cameras to simulate human binocular vision. The depth capture camera provides the ability to reproduce a three-dimensional image based on two captured images from visible light cameras 114A-B with the same timestamp. Such three-dimensional images allow for an immersive, lifelike experience, for example, for virtual reality or video games.

[0040] For stereoscopic vision, a pair of raw red, green, and blue (RGB) images of the scene are captured at a given moment in time (one each for the left and right visible light cameras 114A-B). Figure 3A When the image processor 312 of the image processor 312 processes the raw image pairs captured from the front-facing left and right fields of view 111A-B of the left and right visible light cameras 114A-B, a depth image is generated, and the user can perceive the generated depth image on the optical components 180A-B (e.g., of the mobile device) or other image display. The generated depth image may include a vertex matrix on a three-dimensional position coordinate system in a three-dimensional spatial domain, and the three-dimensional position coordinate system includes an X-axis for horizontal position (e.g., length), a Y-axis for vertical position (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex includes color attributes (e.g., red pixel light value, green pixel light value, and / or blue pixel light value), position attributes (e.g., X position coordinates, Y position coordinates, and Z position coordinates), texture attributes, and / or reflectivity attributes. Texture attributes quantify the perceived texture of the depth image, such as the spatial arrangement of color or brightness in the vertex area of ​​the depth image.

[0041] Figure 1B -C is a rear view of an example hardware configuration of the eye-mounted device 100 including two different types of image displays. The eye-mounted device 100 has a form configured to be worn by a user, which in the example is glasses. The eye-mounted device 100 can take other forms and can be combined with other types of frames, such as a head-mounted device, headphones, or a helmet.

[0042] In the glasses example, the eyewear device 100 includes a frame 105 including a left edge 107A connected to a right edge 107B via a nose bridge 106 adapted to the user's nose. The left and right edges 107A-B include respective holes 175A-B that hold respective optical elements 180A-B, such as lenses and display devices. As used herein, the term lens refers to a transparent or translucent glass or plastic cover having curved and / or flat surfaces that cause light to converge / diverge or cause little or no convergence or divergence.

[0043] Although shown as having two optical elements 180A-B, the eye-mounted device 100 may include other arrangements, such as a single optical element or may not include any optical elements 180A-B, depending on the application or intended user of the eye-mounted device 100. As further shown, the eye-mounted device 100 includes a left block 110A adjacent to the left side 170A of the frame 105 and a right block 110B adjacent to the right side 170B of the frame 105. The blocks 110A-B may be integrated into the frame 105 on the respective sides 170A-B (as shown) or implemented as separate components attached to the frame 105 on the respective sides 170A-B. Alternatively, the blocks 110A-B may be integrated into an eye-mounted device leg (not shown) attached to the frame 105.

[0044] In one example, the image display of optical components 180A-B includes an integrated image display. Figure 1BAs shown. The optical assembly 180A-B includes a suitable display matrix 170 of any suitable type, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or any other such display. The optical assembly 180A-B also includes an optical layer 176, which may include lenses, optical coatings, prisms, reflectors, waveguides, light strips, and other optical components in any combination. The optical layer 176A-N may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light to the user's eyes. The prisms of the optical layer 176A-N extend over all or at least a portion of the corresponding holes 175A-B formed in the left and right edges 107A-B to allow the user to see the second surface of the prism when the user's eyes look through the corresponding left and right edges 107A-B. The first surface of the prism of the optical layer 176A-176N faces upward from the frame 105, and the display matrix is ​​placed above the prism so that the photons and light emitted by the display matrix hit the first surface. The size and shape of the prisms are determined so that light is refracted within the prisms and directed toward the user's eyes by the second surfaces of the prisms of the optical layers 176A-176N. In this regard, the second surfaces of the prisms of the optical layers 176A-N may be convex to direct light toward the center of the eyes. The size and shape of the prisms may optionally be determined to magnify the image projected by the display matrix 170, and the light passes through the prisms so that the image viewed from the second surface is larger than the image emitted from the display matrix 170 in one or more dimensions.

[0045] In another example, the image display device of the optical assembly 180A-B includes Figure 1C The projected image display shown. The optical assembly 180A-B includes a laser projector 150, which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source such as the laser projector 150 is disposed in or on one of the eye-mounted device legs 125A-B of the eye-mounted device 100. The optical assembly 180A-B includes one or more light strips 155A-N that are spaced across the width of the lens of the optical assembly 180A-B or across the depth of the lens located between the front surface and the back surface of the lens.

[0046] When light projected by the laser projector 150 passes through the lenses of the optical components 180A-B, the photons encounter the light bars 155A-N. When a particular photon encounters a particular light bar, the photon is either directed toward the user's eye or passed to the next light bar. A combination of laser projector 150 modulation and light bar modulation can control specific photons or light. In an example, the processor controls the light bars 155A-N via a mechanical, acoustic, or electromagnetic start signal. Although shown as having two optical components 180A-B, the eye-mounted device 100 may include other arrangements, such as a single or three optical components, or the optical components 180A-B may be arranged differently depending on the application or intended user of the eye-mounted device 100.

[0047] like Figure 1B As further shown in FIG. 1-C, the eye-mounted device 100 includes a left block 110A adjacent to the left side 170A of the frame 105 and a right block 110B adjacent to the right side 170B of the frame 105. The blocks 110A-B can be integrated into the frame 105 on the respective sides 170A-B (as shown), or implemented as separate components attached to the frame 105 on the respective sides 170A-B. Alternatively, the blocks 110A-B can be integrated into the eye-mounted device legs 125A-B attached to the frame 105. As used herein, the blocks 110A-B can include a housing enclosing a collection of processing units, cameras, sensors, etc. (e.g., different for the right and left sides) enclosed in the housing.

[0048] In one example, the image display includes a first (left) image display and a second (right) image display. The eye-mounted device 100 includes first and second apertures 175A-B that hold respective first and second optical components 180A-B. The first optical assembly 180A includes a first image display (e.g., Figure 1B The display matrix 170A, or Figure 1C The second optical assembly 180B includes a second image display (eg, Figure 1B The display matrix 170B, or Figure 1C light bars 155A-N″ and projector 150B).

[0049] Figure 1D A schematic diagram depicting the operation of a motion tracker of an eye-mounted device 100 coupled to an input surface 181 to track the motion of at least one finger contact 179 input by a user to the eye-mounted device 100. The input surface 181 is formed of plastic, acetate, or other insulating material that forms the base of the frame 105, the eye-mounted device legs 125A-B, or the sides 170A-B.

[0050] Although touch screens exist for mobile devices such as tablets and smartphones, using a touch screen in the lens of the eye-mounted device 100 interferes with the user's line of sight of the eye-mounted device 100 and obstructs the user's viewing. For example, a finger touch can blur the optical components 180A-B (e.g., optical layers, image display, and lens) and blur or obstruct the user's vision. To avoid blurring and poor clarity when the user's eyes view through the transparent portion of the optical components 180A-B, changes in rotational acceleration, motion, spatial orientation, and other measurement characteristics collected by the motion tracker 118 caused by finger contact on the input surface 181 can be used to detect finger touch gestures. Touch gestures are inputs to the human-machine interface of the eye-mounted device 100 to perform specific actions in the application of the eye-mounted device 100 or to navigate the displayed images in an intuitive manner, which enhances and simplifies the user experience.

[0051] When used in a neural network system, such a motion tracker 118 (which may be an inertial measurement unit (IMU), accelerometer, or gyroscope) already incorporated into the eye-mounted device 100 can save the cost of additional circuitry for a touch sensor (e.g., a capacitive or resistive touch sensor). By using the measurements made by the motion tracker 118 in the neural network model, the additional space required to incorporate the touch sensor circuitry in the eye-mounted device 100 is eliminated, which reduces the form factor of the eye-mounted device 100. In addition, the savings here also include cost. The neural network detection system can be used with a touch sensor and a motion tracker 118 (e.g., an IMU). In this case, the benefit of neural networks is that they are able to recognize complex gestures and activities that would otherwise be impossible to detect with code based on handwritten heuristics.

[0052] Detecting finger gestures via a neural network model using measurements made by the motion tracker 118 as an input layer of the model can enable several functions. For example, touching anywhere on the input surface 181 can highlight an item on the image display screen of the optical assembly 180A-B. Double-clicking on the input surface 181 can select an item. Sliding a finger from front to back can slide or scroll in one direction, for example, to move to a previous video, image, page, or slide. Sliding a finger from back to front can slide or scroll in the opposite direction, for example, to move to a previous video, image, page, or slide. Pinching with two fingers can provide a zoom-in function to enlarge the content of the displayed image. Release with two fingers provides a zoom-out function to reduce the content of the displayed image. The input surface 181 can be almost anywhere on the eye-mounted device 100. To detect sliding, pinching, and releasing of fingers, a touch sensor incorporated in the eye-mounted device 100 can also be used with the motion tracker 118.

[0053] In one example, when the recognized finger gesture is a single click on the input surface 181, this initiates the selection or pressing of a graphical user interface element in the image presented on the optical assembly 180A-B image display. Adjustment of the image presented on the optical assembly 180A-B image display based on the recognized finger gesture can be a primary action that selects or submits a graphical user interface element on the optical assembly 180A-B image display for further display or execution. This is just one example of supported finger gestures, and it should be understood that the eye-mounted device 100 supports several finger gesture types, which can include single or multiple finger contacts. Examples of touch events detected by multiple finger contacts and recognized finger gestures are provided in Figures 6-9. In addition, in some examples, gestures can control other output components, such as speakers of the eye-mounted device 100, such as controlling volume.

[0054] The eye-mounted device 100 may include a wireless network transceiver, such as a cellular or local area network transceiver (e.g., WiFi or Bluetooth TM ), and run complex applications. Some applications may include a web browser for Internet navigation, an application for making phone calls, a video or image codec for watching videos or interacting with pictures, a codec for listening to music, a turn-by-turn navigation application (e.g., entering a destination address and viewing a map), an augmented reality application, and an email application (e.g., for reading and composing emails). Gestures input on input surface 181 can be used to manipulate and interact with content displayed on the image display, controlling applications.

[0055] In an example, a neural network system includes an eye-mounted device 100. The eye-mounted device 100 includes a frame 105, a left eye-mounted device leg 125A extending from a left side 170A of the frame 105, and a right eye-mounted device leg 125B extending from a right side 170B of the frame 105. The eye-mounted device 100 also includes an input surface 181 on the frame 105, the eye-mounted device legs 110A-B, the sides 170A-B, or a combination thereof. The eye-mounted device 100 also includes an image display that presents an image to a user and an image display driver coupled to the image display to control the image presented to the user. The eye-mounted device 100 also includes a motion tracker 118 connected to the input surface 181 to track the motion of at least one finger contact 179 input by a user of the eye-mounted device 100. The motion tracker 118 includes: (i) at least one accelerometer that measures acceleration, (ii) at least one gyroscope that measures rotation, or (iii) an inertial measurement unit (IMU) having at least one accelerometer and at least one gyroscope.

[0056] An inertial measurement unit is an electronic device that uses a combination of accelerometers and gyroscopes (and sometimes magnetometers) to measure and report the body's specific force, angular velocity, and sometimes the magnetic field around the body. If a magnetometer is present, the magnetic field can be used as an input to a neural network to detect specific postures that depend on the earth's or artificial magnetic fields. In this example, the inertial measurement unit determines the rotational acceleration of the eye-mounted device 100. The inertial measurement unit 972 works by using one or more accelerometers to detect linear acceleration and one or more gyroscopes to detect rotation rate. A typical configuration of an inertial measurement unit contains an accelerometer, gyroscope, and magnetometer for each of the three axes: the horizontal axis for left-right movement (X), the vertical axis (Y) for top-bottom movement, and the depth or distance axis for up-and-down movement (Z). The gyroscope detects the rate of rotation around the 3 axes (X, Y, and Z). The magnetometer detects the magnetic field (e.g., facing south, north, etc.), similar to a compass that produces a heading reference, which is a mixture of the earth's magnetic field and other artificial magnetic fields (such as those generated by power lines). The three accelerometers detect acceleration along the horizontal (X), vertical (Y), and depth or distance (Z) axes defined above, which can be defined relative to the ground, the eye-mounted device 100, the depth capture camera, or the user wearing the eye-mounted device 100. Therefore, the accelerometers detect a 3-axis acceleration vector, which can then be used to detect the Earth's gravity vector

[0057] The neural network system also includes memory ( Figure 3A 334), for example, in the eye-mounted device 100 itself or in other components of the neural network system. The neural network system also includes a display driver ( Figure 3A The processor ( Figure 3A 3 ), motion tracker 118, and memory (element 334 of FIG. 3 ), for example, in the eye-mounted device 100 itself or in other components of the neural network system.

[0058] Neural network systems also include programming in memory ( Figure 3A Component 345), processor ( Figure 3A The processor (element 343 of FIG. A ) executes programming (element 345 of FIG. A ) to configure the eye-mounted device 100 to track the movement of at least one finger contact 179 input by the eye-mounted device 100 from the user on the input surface 181 via the motion tracker 118. Tracking via the motion tracker 118 includes: (i) measuring the acceleration of the eye-mounted device 100 via at least one accelerometer, (ii) measuring the rotation of the eye-mounted device 100 via at least one gyroscope, or (iii) measuring the acceleration and rotation of the eye-mounted device 100 via an inertial measurement unit. Figure 3A Element 332) performs programming ( Figure 3AThe element 345 of the eye-mounted device 100 further configures the eye-mounted device 100 to recognize a finger gesture on the input surface 181 of the eye-mounted device 100 by detecting at least one detected touch event based on a change in the movement of the eye-mounted device 100 tracked over a period of time. The processor ( Figure 3A Element 332) performs programming ( Figure 3A Element 345 of the eye-mounted device 100 further configures the eye-mounted device 100 to adjust the image presented on the image display of the optical components 180A-B of the eye-mounted device 100 based on the recognized finger gesture.

[0059] Other arrangements of the motion tracker 118 and input surface 181 may also be implemented. In some arrangements, the input surface is on the left edge 107A or the right edge 107B, or at a different location on the frame 105, or at a different location on one or both of the blocks 110A-B, or at a different location on the sides 170A-B. The motion tracker 118 may be attached substantially anywhere on the frame 105, the left block 110A, the right block 110B, the eye-mounted device legs 125A-B, etc. to track the motion 186 of the eye-mounted device 100, such as the motion caused by the finger contact 179.

[0060] Figure 2 yes Figure 1B and 1C FIG. 1 is a cross-sectional top view of a left block 110A of an eye-mounted device 100, depicting a motion tracker 118 and a circuit board 140A. In this example, the left block 110A includes a flexible printed circuit board 140A, which includes the motion tracker 118. The left block 110A is integrated or connected to the frame 105 on the left side 170A. In some examples, the right block 110B may include a motion tracker 118 having a similar configuration.

[0061] As below Figure 3A-F and 4A-C, the neural network system implemented in the posture, wear, carry or activity detection programming consists of artificial neurons with learnable weights and biases. These convolutional neural network (CNN) models are built via a host such as a server system 398. The host can be a personal computer, an embedded high-speed processor, a GPU, an FPGA, or any other system that performs neural network training. The server system 398 can be one or more computing devices that are part of a service or network computing system, for example, it includes a processor, a memory, and a network communication interface to communicate with the mobile device 390 and the eye-mounted device 100 through a network 395. The server system 398 does not need to communicate with the mobile device 399 or the eye-mounted device 100. The computing system for training the neural network can be independent and not specifically connected to any network. In order to have a high confidence level, that is, the tracked motion over the time period 360 is a recognized posture or activity, or the device is being worn or carried, many (e.g., hundreds or thousands) features such as the measurements 361A-N, 364A-N obtained by the motion tracker 118 are input to the neural network model as a model input layer. However, to improve speed, memory usage (RAM and disk memory / flash memory), and efficiency, the posture detection programming 345, the wear / carry detection programming 386, or the activity detection programming 302 may decide to shorten the program, for example, when 5-10 significant measurement features are matched instead of all 100 or 1000 measurement features of the recognized posture, activity, or wear / carry neural network model. In addition, the neural network forward pass system with fast and low memory usage enables the posture detection programming 345, the wear / carry detection programming 386, or the activity detection programming 302 to be executed and run on a low-power computing system 320 (such as the system included in the eye-mounted device 100). Specifically, the trained neural network (forward pass) runs on a low-power circuit. Therefore, although shown in the memory 334 of the high-speed circuit 330, the neural network such as the posture detection programming 345, the wear / carry detection programming 386, or the activity detection programming 302 can be stored and executed in the low-power processor 323. The training itself is done offline as a one-time task on a server or computer, but can actually be done on any high-speed computing platform.

[0062] The basic unit of computation in a neural network is a neuron, often referred to as a node or unit. A neuron receives input from some other nodes or from an external source and computes an output. Each input has an associated weight (w) that is assigned based on its relative importance relative to other inputs. The input layer provides information from the outside world to the neural network, including input nodes (e.g., motion tracked by motion tracker 118 over time period 360). Hidden layers 348 (e.g., touch events 349A-N) are not directly connected to the outside world, but perform calculations and transmit information from input nodes to output nodes. The output layer is responsible for calculating and transmitting information from the neural network to the outside world (e.g., recognized finger gestures 369A-N and confidence levels 371A-N). The output layer is responsible for the final, classified network output. The classified output identifies a gesture, such as a double-click. Further algorithms process this result into an action.

[0063] A neuron applies a function (f) to the weighted sum of its inputs. Each neuron receives some input, performs a dot product, and optionally follows it with a nonlinearity. The nonlinearity here is the function f. Output = f(sum(input(i) * weight(i))). Therefore, a neural network consists of three layers, called the input layer, hidden layer, and output layer. The function f is nonlinear and is called the activation function. The purpose of the activation function is to introduce nonlinearity into the output of the neuron. This is important because most real-world data is nonlinear, and this allows the neuron to learn from these nonlinear representations.

[0064] Each activation function (or nonlinearity) takes a single number and performs some fixed mathematical operation on it. There are several activation functions encountered in implementation: (i) sigmoid, which takes a real-valued input and compresses it to the range between 0 and 1; (ii) tanh, which takes a real-valued input and compresses it to the range [-1, 1]; and (iii) ReLU (which stands for rectified linear unit), which takes a real-valued input and thresholds it to zero (replaces negative values ​​with zero).

[0065] Figure 3A 3 is a high-level functional block diagram of an exemplary neural network system 300, including an eye-mounted device 100, a mobile device 390, and a server system 398 connected via various networks, the eye-mounted device 100 having a motion tracker 118 to recognize finger gestures based on a neural network model (gesture detection programming 345). The eye-mounted device 100 is connected to a host. For example, the eye-mounted device 100 is paired with the mobile device 390 via a high-speed wireless connection 337, or is connected to the server system 398 via a network 395.

[0066] The neural network system 300 includes a wearable device 399, which is Figure 3A The eye-mounted device 100 in the example of FIG. The wearable device 399 can also be Figure 3B The watch, wristband, or other portable device shown is designed to be worn by a user to communicate with a mobile device 390 or a server system 398 via one or more wireless networks or wireless links. The mobile device 390 can be a smart phone, tablet, laptop, access point, or any other such device capable of connecting to the eye-mounted device 100 using a low-power wireless connection 325 and a high-speed wireless connection 337. The mobile device 390 is connected to the server system 398 and the network 395. The network 395 can include any combination of wired and wireless connections.

[0067] The eye-mounted device 100 includes a motion tracker 118 and a depth capture camera, such as at least one of the visible light cameras 114A-B, and a depth sensor (not shown, but including an infrared emitter and an infrared camera). The eye-mounted device 100 also includes two image displays of the optical assembly 180A-B (one associated with the left side 170A and one associated with the right side 170B). The eye-mounted device 100 also includes an image display driver 342, an image processor 312, a low power circuit 320, and a high speed circuit 330. The image display of the optical assembly 180A-B is used to present images and videos, which may include a series of images. The image display driver 342 is coupled to the image display of the optical assembly 180A-B to control the image display of the optical assembly 180A-B to present images. In Figure 3A , 3C 3E are located on one or more circuit boards, such as a PCB or a flexible PCB, in the edges 107A-B or the legs 125A-B of the eye-mounted device. Alternatively or in addition, the described components can be located in the blocks 110A-B, the frame 105, the hinges 226A-B, or the nosepiece 106 of the eye-mounted device 100.

[0068] Typically, the neural network is pre-trained with a labeled data set and then executed on the eye-mounted device 100 via a forward pass mechanism, where inputs 359A-N are presented and the trained weights are used to calculate outputs 368A-N. The outputs represent the probability of each type of gesture or activity to be detected.

[0069] In the neural network system 300, the eye-mounted device 100 includes a gesture model input layer 359A-N that tracks the motion of the eye-mounted device 100 over a time period 360. The motion tracked over the time period 360 includes accelerometer measurements 361A-N, which include measured accelerations (MA) 362A-N and measured acceleration time coordinates 363A-N to indicate when the measured accelerations 362A-N were taken. The motion tracked over the time period 360 also includes gyroscope measurements 364A-N, which include measured rotations (MR) 365A-N, measured rotation time coordinates 366A-N to indicate when the measured rotations 365A-N were taken, and motion interruption time coordinates 367A-N (e.g., the time when the motion was detected).

[0070] As shown, the memory 334 also includes gesture detection programming 345 to perform a subset or all of the functions described herein for gesture detection. Although the neural network model may include an input layer, a hidden layer, and an output layer, in this example, the neural network model of the gesture detection programming 345 includes a convolutional layer (several), a fully connected layer (these layers are used to become hidden layers), and a single output layer. The gesture detection programming 345 has a trained machine code gesture model 346, a set of gesture weights 347A-N, and a hidden layer 348 (which includes touch events 349A-N). The memory 334 also includes a gesture model output layer 368A-N. The gesture model output layer 368A-N has the recognized finger gestures 369A-N, the confidence levels 371A-N of the recognized finger gestures 369A-N, and the recognized gestures 372 that determine the most likely gestures of the recognized finger gestures 369A-N.

[0071] The high-speed circuit 330 includes a high-speed processor 332, a memory 334, and a high-speed wireless circuit 936. In this example, an image display driver 342 is coupled to the high-speed circuit 330 and is operated by the high-speed processor 332 to drive the left image display and the right image display of the optical assembly 180A-B. The high-speed processor 332 can be any processor capable of managing any general-purpose computing system high-speed communications and operations required by the eye-mounted device 100. The high-speed processor 332 includes the processing resources required to manage high-speed data transmission over a high-speed wireless connection 337 to a wireless local area network (WLAN) using the high-speed wireless circuit 336. In some embodiments, the high-speed processor 332 executes firmware, which includes gesture detection programming 345 and an operating system, such as a LINUX operating system or other such operating system of the eye-mounted device 100, and the operating system is stored in the memory 334 for execution. In addition to any other responsibilities, the high-speed processor 332, which executes the software architecture of the eye-mounted device 100, is also used to manage data transmission to the high-speed wireless circuit 336. In some embodiments, high-speed wireless circuit 336 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other embodiments, other high-speed communication standards may be implemented by high-speed wireless circuit 936.

[0072] The low power wireless circuit 324 and the high speed wireless circuit 336 of the eye-mounted device 100 may include a short range transceiver (Bluetooth TM ) and a wireless wide area, local area, or wide area network transceiver (e.g., cellular or WiFi). A mobile device 390 including a transceiver for communicating via a low power wireless connection 325 and a high speed wireless connection 337 can be implemented using architectural details of the eye-mounted device 100, as can other elements of the network 395.

[0073] The memory 334 comprises any storage device capable of storing various data and applications, including, among other things, the gesture model input layers 359A-N, the gesture detection programming 345, the gesture model output layers 368A-N, the camera data generated by the left and right visible light cameras 114A-B and the image processor 312, and the images and videos generated for display on the image display of the optical assembly 180A-B by the image display driver 342. Although the memory 334 is shown as being integrated with the high-speed circuit 330, in other embodiments, the memory 334 can be a separate element of the eye-mounted device 100. In some such embodiments, electrical routing lines can provide connections from the image processor 312 or the low-power processor 322 to the memory 334 through a chip containing the high-speed processor 332. In other embodiments, the high-speed processor 332 can manage the addressing of the memory 334 so that the low-power processor 322 will direct the high-speed processor 332 at any time when a read or write operation involving the memory 334 is required.

[0074] like Figure 3A As shown, the processor 332 of the eye-mounted device 100 can be coupled to the motion tracker 118, the depth capture camera (visible light camera 114A-B; or visible light camera 114A and depth sensor, not shown), the image display driver 342 and the memory 334. The eye-mounted device 100 can perform all or a subset of any functions described below because the gesture detection program 345 in the memory 334 is executed by the processor 332 of the eye-mounted device 100. Figure 4A As shown, because processor 430 of mobile device 390 executes gesture detection programming 345 in memory 440A, mobile device 390 is able to perform all or a subset of any of the functions described below.

[0075] exist Figure 3AIn the example of FIG. 3 , the processor 332 executes the gesture detection programming 345 to configure the eye-mounted device 100 to perform the following functions, including the function of tracking the eye-mounted device 100 via the motion tracker 118 from at least one finger contact 179 input by the user on the input surface 181: (i) measuring the acceleration 362A-N of the eye-mounted device 100 via at least one accelerometer, (ii) measuring the rotation 365A-N of the eye-mounted device 100 via at least one gyroscope, or (iii) measuring the acceleration 362A-N and the rotation 365A-N of the eye-mounted device 100 via an inertial measurement unit. The eye-mounted device 100 recognizes the finger gesture 369A-N on the input surface 181 of the eye-mounted device by detecting at least one detected touch event 349A-N based on the change in the movement of the eye-mounted device 100 tracked over the time period 360. The eye-mounted device 100 adjusts the image presented on the image display 180A-B of the eye-mounted device 100 based on the recognized finger gesture 369A-N.

[0076] The motion tracker 118 is also configured to track corresponding time coordinates 363A-N, 366A-N, 367A-N over the time period for each of the following measurements taken: (i) acceleration 362A-N measured via an accelerometer, (ii) rotation 365A-N measured via at least one gyroscope, or (iii) acceleration 362A-N and rotation 365A-N measured via an inertial measurement unit. The eye-mounted device 100 tracks corresponding time coordinates 363A-N, 366A-N, 367A-N for each measurement taken 362A-N, 365A-N via the motion tracker 118. The function of detecting at least one touch event 349A-N based on the tracked change in motion over the time period 360 is also based on the corresponding time coordinates 363A-N, 366A-N, 367A-N for each measurement taken 362A-N, 365A-N.

[0077] The function of identifying a finger gesture on the input surface 181 of the eye-mounted device 100 by detecting at least one detected touch event 349A-N based on a change in the tracked motion 360 of the eye-mounted device 100 over a time period includes the following functions. First, a plurality of model inputs 359A-N including the following measurements are applied to the identified gesture model 346 to determine the similarity of the tracked motion 360 changes to the gesture identified in the identified gesture model 346. The measurements are: (i) acceleration 362A-N measured via an accelerometer, (ii) rotation 365A-N measured via at least one gyroscope, or (iii) acceleration 362A-N and rotation 365A-N measured via an inertial measurement unit, the measurements being made at a sampling frequency during the time period 360. The recognized gesture model 346 includes a set of gesture weights 347A-N based on the following training data 376A-N acquired at one or more time intervals: (i) acceleration 378A-N, (ii) rotation 381A-N, or (iii) acceleration 378A-N and rotation 381A-N. Second, based on the determined similarity of the tracked motion 360 changes to the changes of each recognized gesture 369A-N, a model output 368A-N including a confidence 371A-N for each recognized gesture 369A-N is determined. The neural network is pre-trained with a labeled data set, and then the neural network (e.g., gesture detection programming 345) is executed on the eye-mounted device 100 through a forward pass mechanism (e.g., the trained machine code gesture model 346), where the inputs 359A-N are presented and the trained weights (gesture weight sets 347A-N) are used to calculate the outputs 368A-N. The outputs 368A-N represent the probability of each type of gesture to be detected.

[0078] The functionality of applying the plurality of model inputs 359A-N to determine the similarity of the tracked changes in motion 360 to the recognized gestures in the recognized gesture model 346 and determining the model outputs 368A-N is embedded in the eye-mounted device 100 as firmware programming. The functionality of identifying a finger gesture on the input surface 181 of the eye-mounted device 100 by detecting at least one detected touch event 349A-N based on the tracked changes in motion 360 of the eye-mounted device 100 over a period of time also includes: recognizing the recognized gesture 372 as a corresponding recognized gesture 369A-N having the highest corresponding confidence level 371A-N in the model outputs 368A-N. The functionality of recognizing the recognized gesture as the corresponding recognized gesture 369A-N having the highest corresponding confidence level 369A-N in the model outputs 368A-N is application layer programming in the eye-mounted device 100.

[0079] The output components of the eye-mounted device 100 include visual components, such as Figure 1B -C shows the left and right image displays of the optical components 180A-B (e.g., liquid crystal display (LCD), plasma display panel (PDP), light emitting diode (LED) display, projector or waveguide and other displays). The left and right image displays of the optical components 180A-B can present images, such as in a video. The image displays of the optical components 180A-B are driven by the image display driver 342. The output components of the eye-mounted device 100 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the eye-mounted device 100, the mobile device 390, and the server system 398 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide touch positions and touch forces or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0080] The eye-mounted device 100 may optionally include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the eye-mounted device 100. For example, the peripheral device element may include any I / O component, including an output component, a motion component, a position component, or any other such component described herein.

[0081] For example, the biometric component includes a component for detecting expressions (such as hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biological signals (such as blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying people (such as voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (such as an accelerometer), a gravity sensor component, a rotation sensor component (such as a gyroscope), etc. The position component includes a position sensor component for generating position coordinates (such as a global positioning system (GPS) receiver component), a WiFi or Bluetooth receiver for generating positioning system coordinates, etc. TM transceiver, altitude sensor component (e.g., an altimeter or barometer, a barometer is used to detect air pressure from which altitude can be derived), direction sensor component (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from mobile device 390 using wireless connections 325 and 337 via low power wireless circuitry 324 or high speed wireless circuitry 336.

[0082] Figure 3B Shown for Figure 3AThe hardware configuration example of the neural network system 300 server system 398 is used to establish a neural network model for identifying finger gestures (gesture detection programming 345) in the form of a simplified block diagram. Figure 3B As further shown, the server system 398 can be one or more computing devices as part of a service or network computing system, for example, including a memory 350, a processor 360, and a network communication interface 361 that communicates with a mobile device 390 and a wearable device 399 (e.g., an eye-mounted device 100) via a network 395. The memory 350 includes gesture training data (TD) 376A-N, which includes known but unclassified gestures 377A-N tracking motion over time intervals. The gesture training data 376A-N includes accelerometer training data (TD) 378A-N. The accelerometer training data 378A-N has acceleration measurements 379A-N and acceleration time coordinates 380A-N to indicate when the acceleration measurements 379A-N were made. The gesture training data 376A-N includes gyroscope training data 381A-N. Gyroscope training data 381A-N has rotation measurements 382A-N and rotation time coordinates 383A-N to indicate when the rotation measurements 382A-N were made. Gesture training data 376A-N also includes motion interruption time coordinates 384A-N (eg, the time when the motion was detected).

[0083] The memory 350 also includes a gesture model generator, shown as a gesture neural network programming 375. The memory 350 also includes a gesture detection programming 345 output in response to applying the gesture neural network programming 375 to the input gesture training data 376A-N. As shown, the gesture detection programming 345 includes a hidden layer 348 such as a trained machine code gesture model 346, a set of gesture weights 347A-N, and touch events 349A-N. The constructed gesture detection programming 345 is loaded into the eye-mounted device 100 or the wearable device 399 for gesture detection.

[0084] The processor 360 executes the gesture neural network programming 375 to configure the server system 398 to perform some or all of the functions described herein before the processor 332 of the eye-mounted device 100 executes the gesture detection programming 345. First, for known but unclassified gestures 377A-N, the following training data 376A-N of the eye-mounted device 100 are obtained in one or more time intervals: (i) acceleration 378A-N, (ii) rotation 381A-N, or (iii) acceleration 376A-N and rotation 381A-N. Second, a recognized gesture model 346 of the unclassified gesture is constructed based on the obtained training data 376A-N. The function of constructing the recognized gesture model 346 includes the function of calibrating the gesture weight set 347A-N according to the obtained training data 376A-N of the unclassified gesture; and the function of storing the calibrated gesture weight set 347A-N in the recognized gesture model 346 associated with the recognized gesture.

[0085] The posture neural network programming 375 generates a training model based on training data. The training data is the user-labeled inputs and outputs used to calculate the neural network weights. The scripting language here may be irrelevant because this can be done in many languages ​​or hardware. In order to improve the runtime efficiency of the constructed posture detection programming 345, the posture neural network programming 375 can implement the following functions. Based on the acquired training data 376A-N of the unclassified postures 377A-N, a trained scripting language posture model is generated in an interpreted programming language. A calibrated set of posture weights 347A-N is extracted from the trained scripting language posture model. Based on the trained scripting language posture model, a trained inference code posture model is generated in a compiled programming language. The trained inference code posture model is compiled into a trained machine inference code posture model 346. The extracted calibrated set of posture weights 347A-N is quantized.

[0086] A runtime-valid version of the gesture detection program 345 can be stored in the memory 334 of the eye-mounted device 100, including the trained inferred machine code gesture model 346 and the quantized and calibrated set of gesture weights 347A-N. To further improve runtime, the gesture detection program 345 only occupies statically allocated memory during runtime.

[0087] Figure 3C is a high-level functional block diagram of an exemplary neural network system 300, including an eye-mounted device 100 with a motion tracker 118 that detects wear based on a neural network model (wear detection programming 386), a mobile device 390, and a server system 398 connected via various networks. Figure 3C The parts in Figure 3A This has been explained in detail and will not be repeated here.

[0088] In the neural network system 300, the memory 334 of the eye-mounted device 100 includes a wear model input layer 385A-N, which is the movement tracked over the time period 360, such as Figure 3A As shown. As shown, the memory 334 also includes wear detection programming 386. The wear detection programming 386 has a trained machine code wear model 387 and a set of wear weights 388A-N. The memory 334 also includes a wear model output layer 391. The wear model output layer 391 can be a single output value as a wear detection confidence level 392, which is a probability value of being worn ranging from 0 to 1. The not worn confidence level is 100% minus the wear detection confidence level. The code outside the neural network sets this value as a threshold, typically 0.5, to obtain a binary classification.

[0089] The processor 332 executes the wearing detection programming 386 to configure the eye-mounted device 100 to perform the following functions. Track the movement of the eye-mounted device 100 from at least one finger contact 179 input by the user on the input surface 181 via the motion tracker 118 by: (i) measuring the acceleration 362A-N of the eye-mounted device 100 via at least one accelerometer, (ii) measuring the rotation 365A-N of the eye-mounted device 100 via at least one gyroscope, or (iii) measuring the acceleration 362A-N and the rotation 365A-N of the eye-mounted device 100 via an inertial measurement unit. Detect whether the user is wearing the eye-mounted device 100 based on the tracked changes in the movement of the eye-mounted device over the time period 360.

[0090] Figure 3D Shown for Figure 3B The hardware configuration example of the neural network system server system 398 is used to establish a neural network model for wearable detection. Figure 3D The parts are already in Figure 3B In addition to detecting the wearing of a wearable device 399 (e.g., eye-mounted device 100), the methods described herein can also be used to implement carrying detection of a mobile device 390 (e.g., determining when a user is walking with the device in his / her hand, on their body, or in a backpack or purse). The wearing detection system can be extended to activities, such as distinguishing between standing, walking, and running with the device. It can be used to detect gaze or attention to a specific object (with the help of a magnetometer).

[0091] like Figure 3DAs further shown in FIG. 3 , the memory 350 of the server system 398 includes wear / carry training data (TD) 351A-N. The wear / carry training data (TD) 351A-N tracks the movement of the wearable device 399 (e.g., the eye-mounted device 100) or the mobile device 390 at time intervals during wear / carrying 352A-N. The wear / carry training data 351A-N includes accelerometer training data (TD) 378A-N, which has acceleration measurements 379A-N and acceleration time coordinates 380A-N to indicate when the acceleration measurements 379A-N are made. The wear training data 351A-N includes gyroscope training data 381A-N, which includes rotation measurements 382A-N and rotation time coordinates 383A-N to indicate when the rotation measurements 382A-N are made. The wear training data 351A-N also includes motion interruption time coordinates 384A-N (e.g., the time when the motion is detected).

[0092] The memory 350 also includes a wear / carry model generator, as shown in the wear / carry neural network programming 353. The memory 350 also includes a wear / carry detection programming 386, which is output in response to applying the wear / carry neural network programming 353 to the input wear / carry training data 351A-N. As shown, the wear / carry detection programming 386 includes a trained machine code wear / carry model 387 and a set of wear / carry weights 388A-N. As used herein, "trained machine code" is an inference code (for example, written in C language and then compiled into machine code to execute on a microcontroller), which is run in a forward pass on the "model", where the model includes a neural network weight table. In this case, the "model" is a set of wear / carry weights 388A-N. The "trained machine code model" (inference code) is a trained machine code wear / carry model 387. The established wear / carry detection programming 386 is loaded in a wearable device 399 (e.g., an eye-mounted device) for wear detection or a mobile device 390 for carrying detection.

[0093] In the wear detection example, before the processor 332 of the eye-mounted device 100 executes the wear detection programming 386, the processor 360 executes the wear neural network programming 353 to configure the server system 398 to perform some or all of the functions described herein. First, during the wearing of the eye-mounted device 100 and in one or more time intervals when the eye-mounted device 100 is not worn, the training data 351A-N of (i) acceleration 378A-N, (ii) rotation 381A-N, or (iii) acceleration 378A-N and rotation 381A-N of the eye-mounted device 100 is obtained. Second, the recognized wear model 387 is established based on the obtained training data 351A-N. The function of establishing the identified wearing model 387 includes: calibrating a set of wearing weights 388A-N according to the acquired training data 351A-N during wearing of the eye-mounted device 100 and when the eye-mounted device 100 is not worn, and associating the calibrated set of wearing weights 388A-N (for example, a trained set or an optimized set) with the classification of whether the eye-mounted device 100 is worn or not worn and storing it in the identified wearing model 387.

[0094] Figure 3E 3 is a high-level functional block diagram of an exemplary neural network system, including an eye-mounted device 100 with a motion tracker connected via various networks, a mobile device 390, and a server system 398, the motion tracker identifying activities based on a neural network model (activity detection programming 302) while wearing the eye-mounted device 100. Figure 3E The parts are already in Figure 3A The detailed description is given in the section and will not be repeated here.

[0095] In the neural network system 300, the eye-mounted device 100 includes an activity model input layer 301A-N, which is a tracked motion over a time period 360. The tracked motion over the time period 360 includes accelerometer measurements 361A-N, which have measured accelerations (MA) 362A-N and measured acceleration time coordinates 363A-N to indicate when the measured accelerations 362A-N were taken. The tracked motion over the time period 360 also includes gyroscope measurements 364A-N, which include measured rotations (MR) 365A-N, measured rotation time coordinates 366A-N to indicate when the measured rotations 365A-N were taken, and motion interruption time coordinates 367A-N (e.g., the time when the motion was detected). The memory 334 also includes activity model output layers 306A-N, which include identified activities 307A-N, confidence levels 308A-N for the identified activities 307A-N, and identified activities 309 that determine the most likely activity for the identified activities 307A-N.

[0096] As shown, the memory 334 also includes activity detection programming 302. The activity detection programming 302 includes a trained machine code activity model 303, a set of activity weights 347A-N, and a hidden layer 389 (which includes a wear event 305). The processor executes the activity detection programming 302 to configure the eye-mounted device 100 to perform the following functions. In response to detecting that the user is wearing the eye-mounted device 100, such as Figure 3C As shown, an identified activity 309 of a user wearing the eye-mounted device 100 is identified based on the tracked changes in motion over a time period 360. According to the identified activity-based adjustment of the identified activity 309, the image presented on the image display 180A-B of the eye-mounted device 100 is adjusted. The identified activity-based adjustment includes launching, hiding, or displaying (e.g., opening) an application for the user to interact with or utilize the application. The identified activity-based adjustment includes displaying an application menu related to the identified object for execution (e.g., prompt). The identified activity-based adjustment includes controlling contextual notifications to enable, disable, or limit features of the application. The identified activity-based adjustment includes enabling or disabling system-level features (e.g., powering on or off peripherals of a device such as the camera 114A-B). The identified activity-based adjustment may include a combination of the above.

[0097] Figure 3F Shown for Figure 3E An example of a hardware configuration for a neural network system 300 server system is provided in the form of a simplified block diagram to establish a neural network model for identifying activities during wearing of an eye-mounted device (activity detection programming 302). The memory 350 includes activity training data (TD) 310A-N, which includes tracked motion of known but unclassified activities 311A-N over time intervals. The activity training data 310A-N includes accelerometer training data (TD) 378A-N, which has acceleration measurements 379A-N and acceleration time coordinates 380A-N to indicate when the acceleration measurements 379A-N were made. The posture training data 376A-N includes gyroscope training data 381A-N, which includes rotation measurements 382A-N and rotation time coordinates 383A-N to indicate when the rotation measurements 382A-N were made. The posture training data 376A-N also includes motion interruption time coordinates 384A-N (e.g., the time when the motion was detected).

[0098] The memory 350 also includes an activity model generator, shown as an activity neural network programming 312. The memory 350 also includes an activity detection programming 302 output in response to applying the activity neural network programming 312 to the input activity training data 310A-N. As shown, the activity detection programming 302 includes a hidden layer 389 such as a trained machine code activity model 303, a set of activity weights 304A-N, and wear events 305. The activity detection programming 302 is loaded into a wearable device 399 (e.g., the eye-mounted device 100) or a mobile device 390 for activity detection.

[0099] The processor 360 executes the activity neural network programming 312 to configure the server system 398 to perform various functions before the processor 332 of the eye-mounted device 100 executes the activity detection programming 302. First, the server system 398 obtains training data 310A-N of (i) acceleration 378A-N, (ii) rotation 381A-N, or (iii) acceleration 378A-N and rotation 381A-N of the eye-mounted device 100 at one or more time intervals of the unclassified activity 311A-N. Second, the server system 398 builds a recognized activity model 303 of the unclassified activity based on the obtained training data 310A-N. Building the recognized activity model 303 includes calibrating the activity weight set 304A-N of the unclassified activity based on the obtained training data 310A-N of the unclassified activity 311A-N. Building the recognized activity model 303 also includes storing the calibrated set of activity weights 304A-N in association with the recognized activity in the recognized activity model 303.

[0100] like Figure 3A-C, the posture detection programming 345, the wearing / carrying detection programming 386 or the activity detection programming 302 of the eye-mounted device 100 is locally stored in the read-only memory (ROM), the erasable programmable read-only memory (EPROM) or the flash memory of the high-speed circuit 330. The firmware layer of the posture detection programming 345, the wearing / carrying detection programming 386 or the activity detection programming 302 returns the keyword corresponding to the recognized posture or activity and the confidence level that the motion tracked over the time period 360 corresponds to the posture or activity of the application layer of the posture detection programming 345, the wearing / carrying detection programming 386 or the activity detection programming 302. The neural network returns many probabilities, each probability corresponding to an activity to be recognized. Usually, the maximum probability exceeding a certain threshold (usually 0.5) is selected as a method to determine the final result (activity). The firmware resides below the operating system level and is more efficient, for example, this optimizes execution speed by directly calling the hardware. The application layer of the gesture detection programming 345, wear / carry detection programming 386, or activity detection programming 302 determines the recognized gesture or activity-based adjustments to be taken based on the recognized activity or gesture and the confidence level. Having the recognized gesture or activity-based adjustment determination reside in the application layer allows for dynamic changes, with updates distributed from the server system 398 via the network 395, 337.

[0101] In some examples where runtime is not considered important, to allow propagation updates to gesture detection programming 345, wear / carry detection programming 386, or activity detection programming 302, firmware is not used for gesture, wear / carry, or activity detection, and the entire logic of the firmware resides in the application layer in the volatile type memory 334. This can initiate an update, which is transmitted from the server system 398 via the network 395, 337. For example, the server system 398 receives crowdsourced additional training data 376A-N, 351A-N, 359A-N from the same type of wearable devices 399 or mobile devices 390 of different users via the network communication interface 361. The server system 398 updates the recognized object model 366 of the recognized object based on the crowdsourced additional training data 376A-N, 351A-N, 310A-N from different users but from the same type of wearable devices 399 (e.g., eye-mounted devices) or the same mobile device mobile device 390. Updating the gesture detection programming 345, wear / carry detection programming 386, or activity detection programming 302 includes reapplying the model generator to rebuild the gesture detection programming 345, wear / carry detection programming 386, or activity detection programming 302. Retraining is done offline on a high-speed computing device. The result is a set of calculated weights that can be transmitted to the eye-mounted device to update the neural network. Typically, this is called an over-the-air (OTA) update and can include other software. Another mechanism for updating is a mechanism that occurs in the background without user intervention. The server system 398 then sends the gesture detection programming 345, wear / carry detection programming 386, or activity detection programming 302 to the wearable device 399 (e.g., eye-mounted device) or mobile device 390 via the network 395.

[0102] Figure 4A Shows Figure 3A -B's neural network system wearable device 399 or mobile device 390 hardware configuration example, including motion tracker 118 to recognize finger gestures based on the neural network model (gesture detection programming 345). Figure 4A As shown, the gesture detection programming 345 for gesture detection implemented in the wearable device 399 or the mobile device 390 is the same as Figure 3A The programming of the eye-mounted device 100 shown in FIG. 1 is the same as that of FIG. 1 . Figure 3BThe posture neural network programming 375 trains the posture detection programming 345 of the wearable device 399 and the mobile device 390 in exactly the same way as the eye-mounted device 100. However, because the posture training data 376A-N and the posture model input layer 359A-N vary depending on the type and value of measurements made on the motion tracker 118 and the form factor of the eye-mounted device 100, the wearable device 399, and the mobile device 390, a set of posture weights 347A-N and the trained machine code posture model 346 can vary from device to device. Therefore, the posture detection programming 345 can be device-specific. In some examples, the IMU positions and transformations can be reapplied to another similar device and the same neural network can be used. Although the accuracy will not be very good, in this case, only a shorter time / smaller amount of retraining passes are required.

[0103] Figure 4B Shows Figure 3C -D neural network system 300 wearable device 399 or mobile device 390 hardware configuration example, including motion tracker 118 to detect the wearing of wearable device 399 or the carrying of mobile device 390 based on the neural network model. Figure 4B As shown, the wear detection 386 detection programming 345 for wear detection is implemented in the wearable device 399 or the carrying detection programming 386 for carrying detection is implemented in the mobile device 390. Figure 3C The programming of the eye-mounted device 100 shown in FIG. 1 is the same as that of FIG. 1 . Figure 3D The wear or carry neural network programming 353 trains the wear detection programming 386 for the wearable device 399 and the carry detection programming 386 for the mobile device 390 in exactly the same manner as the eye-mounted device 100. However, because the wear or carry training data 351A-N and the wear or carry model input layers 385A-N vary depending on the type and value of measurements made on the motion tracker 118 and the form factors of the eye-mounted device 100, the wearable device 399, and the mobile device 390, a set of wear or carry weights 388A-N and the trained machine code wear or carry model 387 will vary from device to device. Therefore, the wear or carry detection programming 386 is inherently device-specific.

[0104] Figure 4C Shows Figure 3E -F neural network system 300 wearable device 399 or mobile device 390 hardware configuration example, including motion tracker 118 to identify activities based on neural network model. Figure 4C As shown, the activity detection programming 302 for activity detection implemented in the wearable device 399 or the mobile device 390 is Figure 3E The same as shown in FIG. 1 for the eye-mounted device 100 . Figure 3FThe activity neural network programming 312 trains the activity detection programming 302 for the wearable device 399 and the mobile device 390 in exactly the same manner as the eye-mounted device 100. However, because the activity training data 310A-N and the activity model input layers 301A-N vary depending on the type and value of measurements made on the motion tracker 118 and the form factor of the eye-mounted device 100, the wearable device 399, and the mobile device 390, the set of activity weights 304A-N and the trained machine code activity model 303 will vary from device to device. Therefore, the activity detection programming 302 is inherently device-specific.

[0105] like Figure 4A As shown in FIG. 3-C, the wearable device 399 or the mobile device 390 includes an image display 480 and an image display driver 490 for controlling the image display 480. Figure 4A In the example of FIG. 4 , image display 480 and user input device 491 are integrated together into a touch screen display. Examples of touch screen type mobile devices that may be used include (but are not limited to) smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example, and the subject technology as described herein is not intended to be limited thereto. For the purposes of this discussion, Figure 4A -C provides a block diagram illustration of an exemplary mobile device 390 and wearable device 399 having a touch screen display for displaying content and receiving user input as a user interface (or as part of a user interface).

[0106] The activities discussed herein generally involve data communication related to detecting finger gestures, wearing / carrying, or activities of a wearable device (e.g., eye-mounted device 100) or a mobile device 390. Figure 4A -C, the mobile device 390 and the wearable device 399 include at least one digital transceiver (XCVR) 410, which is shown as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 390 and the wearable device 399 also include additional digital or analog transceivers, such as for communication via NFC, VLC, DECT, ZigBee, Bluetooth TM , or a short-range XCVR 420 for short-range network communications such as WiFi. For example, the short-range XCVR 420 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with implementing one or more standard communication protocols in a wireless local area network, such as one of the Wi-Fi standards under IEEE802.11 and WiMAX.

[0107] To generate location coordinates for locating the mobile device 390 and the wearable device 399, the mobile device 390 and the wearable device 399 may include a global positioning system (GPS) receiver. Alternatively or in addition, the mobile device 390 and the wearable device 399 may utilize either or both of the short-range XCVR 420 and the WWAN XCVR 410 to generate location coordinates for positioning. For example, based on a cellular network, WiFi, or Bluetooth TM The positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the eye-mounted device via XCVR410, 420 through one or more network connections.

[0108] Transceivers 410, 420 (network communication interface) conform to one or more of the various digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 410 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technologies, including, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 410, 420 provide two-way wireless communication of information including digitized audio signals, still images and video signals, web page information for display and network-related input, and various types of mobile message communications to / from mobile device 390 or wearable device 399 for neural network system 300.

[0109] As described above, several of these types of communications through transceivers 410, 420 and networks involve protocols and procedures to support communications with the eye-mounted device 100 or server system 398 for detecting finger gestures, wearing, carrying, or activities. For example, such communications can be carried out over wireless connections 325 and 337 via short-range XCVR 420 to transmit packet data to and from the eye-mounted device 100, such as Figure 3A , 3C and 3E. For example, such communication can also be achieved through Figure 3B , 3D 3F transmits data using IP packet data transmission via WWAN XCVR 410. Both WWAN XCVR 410 and short range XCVR 420 are connected to associated antennas (not shown) through radio frequency (RF) transmit and receive amplifiers (not shown).

[0110] Wearable device 399 and mobile device 390 also include a microprocessor, shown as CPU 430, which is sometimes referred to as a main controller in this article. A processor is a circuit having elements constructed and arranged to perform one or more processing functions (usually various data processing functions). Although discrete logic components can be used, an example is to use components that form a programmable CPU. For example, a microprocessor includes one or more integrated circuit (IC) chips, combined with electronic components that perform CPU functions. For example, processor 430 can be based on any known or available microprocessor architecture, such as a reduced instruction set computing (RISC) using an ARM architecture, as commonly used in mobile devices and other portable electronic devices today. Of course, other processor circuits can also be used to form CPU 430 or processor hardware for smart phones, laptops, and tablets.

[0111] The microprocessor 430 acts as a programmable master controller for the mobile device 390 and the wearable device 399 by configuring the mobile device 390 and the wearable device 399 to perform various operations, for example, according to instructions or programming executable by the processor 430. For example, such operations may include various general operations of the mobile device 390 or the wearable device 399, operations related to the gesture detection programming 345, the wearing / carrying detection programming 386, or the activity detection programming 302, and communications with the eye-mounted device 100 and the server system 398. Although the processor may be configured by using hard-wired logic, a typical processor in a mobile device is a general-purpose processing circuit that is configured by executing programming.

[0112] The mobile device 390 and the wearable device 399 include a memory or storage device system for storing data and programming. In this example, the memory system may include flash memory 440A and random access memory (RAM) 440B. RAM 440B is used as short-term storage of instructions and data processed by the processor 430, such as for use as working data processing memory. Flash memory 440A generally provides longer-term storage. The mobile device 390 and the wearable device 399 may include a visible light camera 470.

[0113] In the example of the mobile device 390 or the wearable device 399, the flash memory 440A is used to store programming or instructions executed by the processor 430. For the sake of speed and efficiency as described above, the gesture detection programming 345, the wearing / carrying detection programming 386, or the activity detection programming 302 of the eye-mounted device 100 can be implemented in the form of firmware. The gesture detection programming 345, the wearing / carrying detection programming 386, or the activity detection programming 302 can be stored locally in a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or the flash memory 440A.

[0114] Alternatively or in addition, the gesture detection programming 345, the wear / carry detection programming 386, or the activity detection programming 302 may be implemented in the application layer or have a portion residing at the application layer, as previously described. Thus, depending on the type of device, the mobile device 390 and the wearable device 399 store and run a mobile operating system through which specific applications are executed, including the gesture detection programming 345, the wear / carry detection programming 386, the activity detection programming 302, and the communication with the eye-mounted device 100 and the server system 398. The application can be a local application, a hybrid application, or a web application (e.g., a dynamic web page executed by a web browser), running on the mobile device 390 or the wearable device 399. Examples of mobile operating systems include Google Android, Apple iOS (I-Phone or iPad devices), Windows Mobile, Amazon Fire OS, RIM Blackberry OS, etc.

[0115] Figure 5 Yes, you can Figure 3B , 3D 3F and 3F are flowcharts of methods implemented in the neural network system 300 to optimize the execution speed and efficiency of the neural network model generated for gesture, wear, carry or activity recognition. It should be understood that in other examples, the training model can be generated in any language, interpreted language or native language. The training model can be generated on a CPU, GPU, FPGA or ASIC, and it does not matter how the model is generated as long as the weights of the network can be optimized / calculated based on the training data. Starting at block 500, based on the acquired training data 376A-N, 310A-N, a trained script language model is generated in an interpreted program language, including tracked movements of known but unclassified gestures 377A-N or activities 311A-N that occur at time intervals when a wearable device 399 (e.g., an eye-mounted device 100) is worn and when the wearable device 399 is not worn. For wearing detection, a trained script language model is generated based on training data 351A-N acquired from motion tracked over time intervals while a wearable device 399 (e.g., eye-mounted device 100) is worn and when the wearable device 399 is not worn. For carrying detection, a trained script language model is generated based on training data 351A-N acquired from motion tracked over time intervals while the mobile device 390 is carried and when the mobile device 390 is not carried.

[0116] In the gesture recognition example, accelerometer training data is collected during a double-tap (or other multiple taps, such as three taps) gesture on either side of the eye-mounted device 100 using custom firmware for data collection. The last 800ms or other time intervals of accelerometer samples for training data are collected in the embedded multimedia controller (EMMC), as well as indications that the tap gesture occurred on the side 170A-B, the block 110A-B, or other input surface 181. This process is called training data labeling. For each individual data element, a label is usually created by the person who specifies the gesture. For example, for a double-tap detector, the label is none, left, right. In addition, a large number of samples for non-target gesture training data are collected, which is an empty class. In some examples, the data can be divided into training and test data sets. A convolutional neural network (CNN) is trained on the acquired training data (using a 1-dimensional convolution layer on the time dimension of the previously mentioned features). Keras is a high-level neural network API written in Python and can be run on top of TensorFlow, CNTK, or Theano, which can be used to train models and generate trained script language gesture models. Keras is just one example of a framework that can be used to train neural networks. You can use TensorFlow, CNTK or Theano directly or use any other neural network training framework. This can be extended to any other framework.

[0117] In the wear or activity recognition example, inertial measurement unit (IMU) features are collected as training data during a wear / not wear session, and the IMU training data with tags is saved to the EMMC using custom firmware in a comma separated value (CSV) format indicating the IMU measurement type. IMU features (e.g., training data) include: acceleration (X, Y, Z); gyroscope (pitch, yaw, roll)-rotational acceleration; time (e.g., milliseconds) since the last motion interrupt (e.g., if motion occurs after 4 seconds of no motion, the interrupt will trigger); and time (e.g., milliseconds) since the last no motion interrupt (if the device is idle for 4 seconds in a row, the interrupt will trigger). Each of these features is sampled at a sampling rate of 100 Hz. The user of the wearable device 399 (e.g., eye-mounted device 100) collects data during various activities or settings wearing or not wearing the device, such as: cycling (e.g., wearing the eye-mounted device 100 or placing the eye-mounted device 100 in a backpack), swimming, jogging, and driving. Download the wearing training data from the eye-wearing device 100 and split the data into training or test data sets. Train a convolutional neural network (CNN) based on the data (using a 1-dimensional convolution layer on the previously mentioned training data feature time dimension). The following parameters can be used with random grid search. First, the sampling rate (100Hz is faster than required, so downsampling can be used as needed), for example, a sampling rate of 300ms can be used. The slower the sampling rate, the less memory is required for a single model evaluation (but performance will be affected at a certain point). Second, the window size can be used (i.e., how many consecutive samples are checked at a given time, for example, 6 seconds of samples can be used). Third, various neural network hyperparameters (e.g., number of layers, activation functions, loss functions, etc.) can be utilized. Kerns can be used to train the model and generate a trained script language gesture model.

[0118] Once a satisfactory trained scripting language model for gesture, wear, carry or activity detection has been generated, the next challenge is to deploy the trained scripting language model to a microcontroller (where computing resources are limited, such as extremely limited in flash and RAM memory). This means that the number of weights in the model and how much data needs to be in memory at any given time is very important.

[0119] Moving to block 510, a calibrated set of posture weights 347A-N, wear / carry weights 388A-N, or activity weights 304A-N is extracted from the trained scripting language posture model. To address the limited computational resources of the microcontroller of the wearable device 399 or mobile device 390, a general framework consisting of two main parts can be utilized. First, a Python script takes a trained scripting language model (e.g., a Keras-based scripting language model) for posture, wear, carry, or activity detection and generates C code for the weights, as well as a C application programming interface (API) for performing the inference step.

[0120] Therefore, continue to block 520, and generate a trained inference code model for posture, wear / carry or activity detection in compiled programming based on the trained script language model. For example, the trained script language model generated with Keras is written in Python and passed to the Keras exporter to generate an exported C code model. In addition, C code static library implementations of many common neural network abstractions (such as dense layers, convolutional layers, various activation functions, etc.) are written in C code and used by the output C code model. Although Keras2CPP is an open source tool that can transfer Keras Python models to C++ code, the transferred code includes a lot of modern C++ (including many standard template libraries (STL), dynamic memory allocation, etc.), and the library code is too large to easily adapt to many modern microcontrollers. The trained inference code model in the C language implementation does not require dynamic memory allocation, and the code size is more than an order of magnitude smaller (using the same compilation flags, optimized size, etc.).

[0121] Moving to block 530, the trained inference code model is compiled into a trained machine inference code model. For gesture detection, the compiled Keras2CPP library size is 14128 bytes of text, 2330 bytes of data, 1 byte of statically allocated data, for a total of 16479 bytes. The size of the compiled machine code from the C code static library implementation of the common neural network abstraction is 444 bytes of text, 0 bytes of data, for a total of 444 bytes, which reduces the memory required for gesture detection programming 345, wear / carry detection programming 386, or activity detection programming 302 loaded in a wearable device (e.g., eye-mounted device 100) or mobile device 390. This is just one example, and the size reduction varies greatly for each model, however, it is worth noting that this process enables the neural network to be configured on a microcontroller using only a few hundred bytes of flash memory and a few hundred bytes of RAM.

[0122] Now proceeding to block 540, the extracted set of calibrated posture weights 347A-N, wear / carry weights 388A-N, or activity weights 304A-N is quantized. Quantization converts the weights stored in floating point format (e.g., at least 32-bit IEEE standard floating point) to 8-bit numbers, which reduces the memory required for posture detection programming 345, wear / carry detection programming 386, and activity detection programming 302 by 4 times (4x) with negligible performance degradation.

[0123] Now ending at block 550, the trained inference machine code model and the quantized, calibrated set of posture weights 347A-N, wear / carry weights 388A-N, or activity weights 304A-N are loaded into the statically allocated memory of the wearable device 399 (e.g., eye-mounted device 100) or mobile device 390, as opposed to dynamically allocated memory. These components include posture detection programming 345, wear / carry detection programming 386, or activity detection programming 302.

[0124] Once the optimized C-code neural network model is generated and compiled into machine code, it is called from the application firmware to recognize gestures, detect the wearing of a wearable device 399 (e.g., eye-mounted device 100) or the carrying of a mobile device 390, and recognize any activities performed during the wearing of the wearable device 399. For gesture detection, on the click motion interrupt 367A-N from the IMU, the last 800ms of the accelerometer measurements 361A-N from the motion tracker 118 are run through the trained machine code gesture model 346 to detect a double-click gesture on either side of the eye-mounted device 100. Different operations are implemented in the trained machine code gesture model 346 based on the double-click side on the eye-mounted device 100. For example, the C-code API generated for double-click detection takes the accelerometer measurements 361A-N as an input layer array over time from the past 800 milliseconds (ms), and outputs an output layer array of probabilities (confidence levels 371A-N) for the following three categories: no double-click, double-click on the left eye-mounted device leg 125A, and double-click on the right eye-mounted device leg 125B. For example, the C-code API generated for wearing detection takes the accelerometer measurements 361A-N, gyroscope measurements 364A-N, and motion data (e.g., motion interruption time coordinates 367A-N) from the IMU over time as an input layer array, and outputs an output layer as a single probability (confidence level 392) that the eye-mounted device 100 is being worn. For example, the C-code API generated for activity detection takes as input layer arrays accelerometer measurements 361A-N, gyroscope measurements 364A-N, and motion data (e.g., motion interruption time coordinates 367A-N) from the IMU over time, and outputs a probabilistic output layer array containing the following four levels 307A-N and confidence levels 308A-N: biking, swimming, jogging, and driving. The input (e.g., 301A-N) and output (306A-N) layer arrays are statically allocated rather than dynamically allocated to improve runtime. Of course, the size of the input and output layer arrays will vary depending on the number of postures and activities.

[0125] FIG6-9 shows several examples of touch events detected and finger gestures recognized for multiple finger contacts on the input surface 181 of the eye-mounted device 100. In each of the examples of FIG6-9, the function of detecting at least one touch event 349A-N on the input surface 181 based on at least one finger contact 179 input from a user includes the following functions. Based on first measurements 362A, 365A taken at first time coordinates 363A, 366A, 367A corresponding to the first finger contact 179A input from the user at a first input time, a first touch event 349A on the input surface 181 is detected. Based on second measurements 362B, 365B taken at second time coordinates 363B, 366B, 367B, a second touch event 349B on the input surface 181 is detected, the second time coordinates 363B, 366B, 367B corresponding to the second finger contact 179B input from the user at a second input time within the tracking time period. The functionality of recognizing a finger gesture is based on the first and second detected touch events 349A-B, the first time coordinates 363A, 366A, 367A, and the second time coordinates 363B, 366B, 367B.

[0126] Fig. 6A -C shows a touch event detected by pressing and holding on the input surface 181. As shown, multiple finger contacts occur on the input surface 181, including pressing (first finger contact 610A), holding (second finger contact 610B), and no finger contact 610C by releasing the input surface 181. Therefore, the first and second detected touch events are pressing and holding on the input surface 181. The recognized finger gesture is a press and hold of a graphical user interface element in the image presented on the image display. Adjustment of the image presented on the image display based on the recognized finger gesture is configured to allow dragging (e.g., movement) of a graphical user interface element on the image display or provide display options (e.g., a context menu associated with a graphical user interface element).

[0127] Figure 7 A pinch and unpinch detected touch event on input surface 181 is shown. Multiple finger contacts occur on input surface 181, where two fingers (first finger contact 710A and second finger contact 710B) move away from each other (finger unpinch) or toward each other (finger pinch). In the pinch detected touch event example, the first and second detected touch events are finger pinches on input surface 181. The recognized finger gesture is a magnification of an image presented on an image display. Adjustment of the image presented on the image display based on the recognized finger gesture magnifies the image presented on the image display.

[0128] In the finger release detected touch event example, the first and second detected touch events are finger releases on input surface 181. The recognized finger gesture is a reduction in the image presented on the image display. The adjustment of the image presented on the image display based on the recognized finger gesture reduces the image presented on the image display.

[0129] Figure 8 A touch event detected by finger rotation on input surface 181 is shown. As shown, multiple finger contacts occur on input surface 181, including continuous rotation of two fingers into a circle from two initial points of first finger contact 810A and second finger contact 810B to two final contact points of the two fingers. In some examples, only one finger can be rotated into a circle. The first and second detected touch events are finger rotations on input surface 181. The recognized finger gestures are finger rotations of an image presented on an image display. Adjustment of the image presented on the display based on the recognized finger gesture causes the image presented on the image display to rotate, such as a rotating view. A rotation gesture may occur when two fingers rotate around each other.

[0130] Fig. 9 A touch event detected by a finger sliding on the input surface 181 is shown. As shown, multiple finger contacts occur on the input surface 181, including dragging a finger to the left or right from the point of initial finger contact 910A to the last point of the second finger contact 910B or 910C. The first and second detected touch events are finger slides from front to back (910A to 910C) or from back to front (910A to 910B) on the input surface 181. The recognized finger gesture is the scrolling of the image presented on the image display. The adjustment of the image presented on the image display based on the recognized finger gesture causes the image presented on the image display to scroll. As shown, this scrolling or sliding gesture may occur when the user moves one or more fingers in a specific horizontal direction of the screen without significantly deviating from the main direction of travel, however, it should be understood that the direction of travel may also be vertical.

[0131] Any functionality described herein for posture, wearing, activity, or carrying detection of a wearable device (e.g., eye-mounted device 100), a mobile device 390, and a server system 398 may be embodied in one or more methods as method steps, or in one or more applications, as described above. According to some embodiments, "programming," "application," or "firmware" is a program that performs the functions determined in a program, such as logic embodied in software or hardware instructions. One or more applications may be constructed in various programming languages ​​in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than a particular platform vendor using ANDROID TM or IOS TM Software Development Kit (SDK) applications developed on IOS TM , ANDROID TM , Mobile software running on a mobile operating system such as an Android phone or other mobile operating system. In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described herein. The application can be stored in any type of computer-readable medium or computer storage device and can be executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can be embodied in special-purpose computer hardware, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0132] The program aspects of the technology can be considered as a "product" or "article of manufacture" typically in the form of executable code and / or associated data, which is carried or contained in a type of machine-readable medium. For example, the programming code may include code for fingerprint sensors, user authorization, navigation, or other functions described herein. "Storage" type media includes any or all tangible memory or its associated modules of computers, processors, etc., such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-temporary storage for software programming at any time. All or part of the software can sometimes be communicated through the Internet or various other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another, such as from a service provider's server system 398 or host to a wearable device 399 (such as an eye-mounted device 100) and a computer platform of a mobile device 390. Therefore, another type of medium that can carry programming, media content or metadata files includes optical, electrical and electromagnetic waves, such as physical interfaces between local devices, through wired and optical landline networks, and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered media that carry software. As used herein, unless limited to "non-transitory," "tangible," or "storage" media, computer or machine "readable media" and other terms refer to any medium that participates in providing instructions or data to a processor for execution.

[0133] Therefore, machine-readable media can take many forms of tangible storage media. For example, non-volatile storage media include optical or magnetic disks, such as any storage device in any computer, etc., such as can be used to implement the client device shown in the accompanying drawings, media gateways, code converters, etc. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wires and optical fibers, including wires that constitute a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or take the form of sound waves or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, for example, common forms of computer-readable media include: floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card paper tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, flash EPROMs, any other storage chips or boxes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other media from which a computer can read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0134] The scope of protection is limited solely by the appended claims. When interpreted in light of this specification and the subsequent prosecution, the scope is intended and should be interpreted to be as broad as the ordinary meaning of the language used in the claims, and should be interpreted to include all equivalent structures and functions. However, the claims are not intended to include subject matter that fails to meet the requirements of Sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a manner. Any unintentional adoption of such subject matter is hereby waived.

[0135] Except as stated above, nothing stated or described is intended or should be construed as causing any component, step, feature, object, benefit, advantage, etc. to be dedicated to the public, regardless of whether it is stated in the claims.

[0136] It should be understood that, in addition to the specific meanings set forth in this article, the terms and expressions used herein have the common meanings consistent with these terms and expressions relative to their respective fields of study. For example, the first and second relationship terms can be used only to distinguish an entity or action from another entity or action, without requiring or implying any actual such relationship or order between these entities or actions. The terms "include", "comprise" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, product or equipment including or including a list of elements or steps not only includes those elements or steps, but can also include other elements or steps that are not explicitly listed or inherent to these processes, methods, products or equipment. The elements with "a" or "an" in front do not exclude the presence of additional identical elements in the process, method, product or equipment including the element without further restriction.

[0137] Unless otherwise stated, any and all measurements, values, ratings, positions, sizes, dimensions and other specifications set forth in this specification (including the appended claims) are approximate and imprecise. Such quantities are intended to have a reasonable range consistent with the functions to which they are related and the practices in the art to which they are related. For example, parameter values, etc. may vary from the stated quantity by ±10% unless otherwise expressly stated.

[0138] In addition, in the foregoing detailed description, it can be seen that in order to simplify the present disclosure, various features are grouped together in various examples. This disclosure method should not be interpreted as reflecting the intention that the examples claimed for protection require more features than the features explicitly recited in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, wherein each claim independently serves as a separately claimed subject matter.

[0139] Although what is considered to be the best mode and other examples have been described above, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and they may be applied to many applications, only some of which are described herein. The appended claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A neural network system, comprising: Eyewear, including: frame; eyewear legs extending from sides of the frame; an input surface on the frame, the eyewear leg, the side, or a combination thereof; an image display for presenting images to a user; an image display driver coupled to the image display to control images presented to a user; A motion tracker connected to the input surface to track the motion of at least one finger contact input by the eye-mounted device from a user, the motion tracker comprising: at least one accelerometer to measure acceleration, At least one gyroscope, or an inertial measurement unit having the at least one accelerometer and the at least one gyroscope; Memory; a processor coupled to the image display driver, the motion tracker, and the memory; and Programming in the memory, wherein execution of the programming by the processor configures the eye-mounted device to perform functions including: Tracking the motion of the eye-mounted device from the at least one finger contact input by the user on the input surface via the motion tracker by the following steps: measuring an acceleration of the eye-mounted device via the at least one accelerometer, measuring a rotation of the eye-mounted device by means of the at least one gyroscope, or Measuring the acceleration and rotation of the eye-mounted device by the inertial measurement unit; and Recognize a finger gesture on the input surface of the eye-mounted device by detecting at least one detected touch event based on a tracked change in motion of the eye-mounted device over a period of time, the recognition function comprising a function of applying a plurality of model inputs to a recognition gesture model, the model inputs comprising (i) acceleration measured via the accelerometer, (ii) rotation measured via the at least one gyroscope, or (iii) acceleration and rotation measured via an inertial measurement unit at a sampling frequency over the period of time to determine a similarity of the tracked change in motion to a recognized gesture in a recognized gesture model, wherein the recognized gesture model comprises a set of gesture weights based on training data of (i) acceleration, (ii) rotation, or (iii) acceleration and rotation acquired over one or more time intervals, and determining a model output comprising a corresponding confidence level for each corresponding recognized gesture based on the determined similarity of the tracked change in motion to the corresponding recognized gesture; and An image presented on the image display of the eye-mounted device is adjusted based on the recognized finger gesture.

2. The neural network system of claim 1, wherein: The input surface is formed of a plastic, acetate or other insulating material forming the frame, the eyewear device legs or the side base; as well as The eye-mounted device further comprises: A block coupled or connected to the frame at the side; as well as The assembly block includes a circuit board including a motion tracker.

3. The neural network system of claim 1, wherein: The motion tracker is further configured to track, over the period of time, a corresponding time coordinate for each measurement of (i) acceleration measured via the accelerometer, (ii) rotation measured via the at least one gyroscope, or (iii) acceleration and rotation measured via the inertial measurement unit; Executing the programming by the processor further configures the eye-mounted device to perform functions, including tracking the corresponding time coordinates of each measurement via the motion tracker; and The functionality of detecting the at least one touch event based on changes in the tracked motion over the period of time is also based on the respective time coordinates of each measurement made.

4. The neural network system of claim 1, wherein: embedding the functionality of applying the plurality of model inputs to determine similarity of the tracked motion changes to the identified gesture in the identified gesture model and determining the model output as firmware programming in the eye-mounted device; By detecting the at least one detected touch event based on a change in the movement of the eye-mounted device tracked over the period of time, the function of recognizing the finger gesture on the input surface of the eye-mounted device further comprises the following functions: identifying the recognized gesture as the corresponding recognized gesture having the highest corresponding confidence in the model output; as well as The function of identifying the recognized gesture as the corresponding recognized gesture with the highest corresponding confidence in the model output is application layer programming in the eye-mounted device.

5. The neural network system of claim 4, wherein: The functionality of detecting the at least one touch event on the input surface based on the at least one finger contact input from the user comprises the following functionality: detecting a first touch event on the input surface based on a first measurement taken at a first time coordinate corresponding to a first finger contact input from a user at a first input time; as well as detecting a second touch event on the input surface based on a second measurement taken at a second time coordinate corresponding to a second finger contact input from the user at a second input time within the period of time; as well as The function of recognizing the finger gesture is based on the first and second detected touch events, the first time coordinate, and the second time coordinate.

6. The neural network system of claim 5, wherein: the first and second detected touch events being a press and a hold on the input surface; The recognized finger gesture is a press and hold of a graphical user interface element in an image presented on the image display; as well as The adjustment of the image presented on the image display based on the recognized finger gesture is configured to allow the graphical user interface element to be dragged on the image display.

7. The neural network system of claim 5, wherein: The first and second detected touch events are finger sliding from front to back or from back to front on the input surface; The recognized finger gesture is a scrolling of the image presented on the image display; as well as Adjustments to the image presented on the image display based on the recognized finger gesture cause the image presented on the image display to scroll.

8. The neural network system of claim 5, wherein: the first and second detected touch events are finger pinches on the input surface; The recognized finger gesture is a magnification of the image presented on the image display; as well as Adjusting the image presented on the image display based on the recognized finger gesture magnifies the image presented on the image display.

9. The neural network system of claim 5, wherein: The first and second detected touch events are finger releases on the input surface; The recognized finger gesture is a zoom-out of the image presented on the image display; as well as Adjusting the image presented on the image display based on the recognized finger gesture zooms out the image presented on the image display.

10. The neural network system of claim 5, wherein: the first and second detected touch events are rotations of a finger on the input surface; The recognized finger gesture is a finger rotation of an image presented on the image display; as well as The adjustment of the image presented on the image display based on the recognized finger gesture causes the image presented on the image display to rotate.

11. The neural network system of claim 1, wherein: The processor comprises a first processor and a second processor; The memory includes a first memory and a second memory; The eye-mounted device comprises: the first processor; the first memory; gesture detection programming in the first memory, wherein the first processor executes the gesture detection programming to configure the eye-mounted device to perform the following functions: tracking, via the motion tracker, the motion of the eye-mounted device based on the at least one finger contact input from the user on the input surface; recognizing the finger gesture on the input surface of the eye-mounted device by detecting the at least one detected touch event based on a change in the tracked motion of the eye-mounted device over the period of time; and adjusting the image presented on the image display of the eye-mounted device based on the recognized finger gesture; The neural network system further includes a host, which includes: the second processor; the second memory including training data of (i) acceleration, (ii) rotation, or (iii) acceleration and rotation of unclassified gestures and the recognized gesture model over one or more time intervals; gestural neural network programming in the second memory, wherein the second processor executes the gestural neural network programming to configure the host to perform functions including: Before the first processor of the eye-mounted device executes the gesture detection programming: Acquire training data of (i) acceleration, (ii) rotation, or (iii) acceleration and rotation of the eyewear device for the unclassified posture over one or more time intervals; Based on the acquired training data, the recognized posture model of the unclassified posture is established, and the function of establishing the recognized posture model includes the following functions: calibrating the set of posture weights based on the acquired training data of the unclassified posture; and A calibrated set of gesture weights is stored in the recognized gesture model associated with the recognized gesture.

12. The neural network system of claim 11, wherein: The second processor executes the gesture neural network programming to configure the host to perform functions including: generating a trained scripting language gesture model in an interpreted programming language based on the acquired training data of the unclassified gesture; extracting the calibrated set of gesture weights from the trained scripting language gesture model; as well as A trained inferred code gesture model is generated in a compiled programming language based on the trained scripting language gesture model.

13. The neural network system of claim 12, wherein: The second processor executes the gesture neural network programming to configure the host to perform functions including: compiling the trained inference code gesture model into a trained machine inference code gesture model; and The extracted calibrated set of pose weights is quantized.

14. The neural network system of claim 13, wherein: The gesture detection programming in the first memory of the eye-mounted device includes the trained inferred machine code gesture model and the quantized calibrated set of gesture weights; as well as The gesture detection programming only occupies statically allocated memory at run time.

15. The neural network system of claim 1, wherein: The at least one detected touch event is a single tap on the input surface; The recognized finger gesture is a selection of a graphical user interface element in the image presented on the image display; A graphical user interface element is selected for display or execution on the image display based on the adjustment of the image presented on the image display by the recognized finger gesture.

Images