Eyewear system, device and method for providing assistance to a user
Through the design of the modular glasses system, the existing glasses devices are solved, and convenient information display and assisted reality functions are realized to ensure that users can hear far-field sounds and provide personalized user experience.
Patent Information
- Application Number
- CN202080007982.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-01-07
- Filing Date
- 2020-01-08
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-01-08
AI Technical Summary
Existing glasses devices are expensive and difficult to reconfigure, unable to meet the needs of different users, while closed earphones block the ear canal, hindering the reception of far-field and near-field sounds.
A modular glasses system was designed to integrate electronic devices through interchangeable temples and frames to provide information display and assist reality functions, including biological parameter monitoring, audio and video playback, using wireless communication systems and embedded sensors. Ensure that users can hear far-field sounds when worn.
It realizes convenient information display and assistive reality functions, reduces the cost of glasses devices, avoids the problem of headphone blockage, provides a personalized user experience, and improves the user's information acquisition efficiency.
Smart Images

Figure CN113260902B_ABST
Abstract
Description
[0001] This patent application is a continuation-in-part of U.S. non-provisional patent application serial number 16 / 711,340, filed on December 11, 2019, entitled "Modular Eyewear Systems, Apparatus, and Methods," and claims the benefit of priority to U.S. Provisional Patent Application Serial Number 62 / 778,709, filed on December 12, 2018, entitled "Modular Eyewear Systems with Interchangeable Frames and Temples with Embedded Electronics for Mobile Audiovisual Augmentation and Assisted Reality," and U.S. Provisional Patent Application Serial Number 62 / 873,889, filed on July 13, 2019, entitled "Wearable Devices, Systems, and Methods." The entire contents of U.S. non-provisional patent application serial number 16 / 711,340, entitled "Modular Eyewear Systems, Apparatus, and Methods," are incorporated herein by reference. This application claims the benefit of priority to co-pending U.S. Provisional Patent Application Serial No. 62 / 789,818, filed on January 8, 2019, entitled "MODULAR EYEWEAR SYSTEM WITH INTERCHANGEABLE FRAME AND TEMPLE WITH EMBEDDED ELECTRONICS FOR AUGMENTED REALITY AND ACTIVITY MONITORING." The entirety of U.S. Provisional Patent Application Serial No. 62 / 789,818, entitled "MODULAR EYEWEAR SYSTEM WITH INTERCHANGEABLE FRAME AND TEMPLE WITH EMBEDDED ELECTRONICS FOR AUGMENTED REALITY AND ACTIVITY MONITORING," is hereby incorporated by reference in its entirety. This application claims the benefit of priority to co-pending U.S. Provisional Patent Application Serial No. 62 / 873,889, filed on July 13, 2019, entitled "WEARABLE DEVICES, SYSTEMS, AND METHODS." The entirety of U.S. Provisional Patent Application Serial No. 62 / 873,889, entitled "WEARABLE DEVICES, SYSTEMS, AND METHODS," is hereby incorporated by reference in its entirety. Technical Field
[0002] The present invention generally relates to eyewear devices, and more particularly to an apparatus, method and system for providing information to a user through a modular eyewear device. Background Art
[0003] Modern life moves at a fast pace. An individual is often constrained by time and he or she is often in situations where both hands are occupied and information is not available. This creates problems. Currently available glasses, such as prescription glasses for reading or prescription sunglasses, are expensive and are not easily reconfigured to meet the needs of different users. This creates problems. Personalized sound transmission is usually accomplished through closed in-ear devices called headphones or earplugs. Such devices block the ear canal and prevent the user from hearing far-field and near-field sounds. This creates problems. Therefore, there are technical problems that require solutions using technical means that can produce technical effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] The best understanding of the present invention may be obtained by referring to the following description and accompanying drawings which illustrate various embodiments of the present invention. The present invention is described by way of example in various embodiments and is not limited by the accompanying drawings in which like reference numerals indicate similar elements.
[0005] Figure 1 is a schematic diagram of a modular reconfigurable eyeglass system according to an embodiment of the present invention.
[0006] Figure 2 is a schematic diagram of a reconfigurable component for an eyewear device according to an embodiment of the present invention.
[0007] Figure 3 is a schematic diagram of a plurality of reconfigurable components for an eyeglass device according to an embodiment of the present invention.
[0008] Figure 4 is a schematic diagram of another reconfigurable modular eyewear system according to an embodiment of the present invention.
[0009] Figure 5 According to an embodiment of the present invention, Figure 4 Perspective and top views of the modular eyewear system.
[0010] Fig. 6A is a schematic diagram of a system structure for a modular eyewear device according to an embodiment of the present invention.
[0011] Figure 6B According to an embodiment of the present invention Fig. 6A Schematic diagram of a wireless network corresponding to the system structure for a modular eyewear device.
[0012] Figure 7 According to an embodiment of the present invention, Figure 4 Schematic diagram of another system structure of the modular eyewear device.
[0013] Figure 8 is a block diagram of a temple insertion module according to an embodiment of the present invention.
[0014] Fig. 9 is a schematic diagram of a modular eyewear device equipped with a rear neck module assembly according to an embodiment of the present invention.
[0015] Fig.10 is a perspective view of a rear neck module assembly configured with a wearable device according to an embodiment of the present invention.
[0016] Fig.11 is a schematic diagram of interlockingly coupling temples to temples according to an embodiment of the present invention.
[0017] Fig.12is a schematic diagram of coupling the rear neck module assembly with the electronic devices contained in the temple according to an embodiment of the present invention.
[0018] Fig.13 Schematic diagram of combining a rear neck module assembly with temple electronic components according to an embodiment of the present invention.
[0019] Fig.14 is a schematic diagram of a user interface on a rear neck module assembly according to an embodiment of the present invention.
[0020] Fig.15 is a block diagram of an electronic unit configured for a rear neck electronic pod (ePOD) according to an embodiment of the present invention.
[0021] Fig.16 is a schematic diagram of a method for providing assistance to a user of an eyewear device according to an embodiment of the present invention.
[0022] Fig.17 is a schematic diagram of receiving data from a sensor associated with an eyewear device according to an embodiment of the present invention.
[0023] Fig.18 4 is a schematic diagram of scene analysis of a user of an eyewear device according to an embodiment of the present invention.
[0024] Fig.19 is a schematic diagram of another method for providing assistance to a user of an eyewear device according to an embodiment of the present invention.
[0025] Fig. 20 FIG. 4 is a schematic diagram of analyzing a user state of a user of an eyewear device according to an embodiment of the present invention.
[0026] Fig.21 is a schematic diagram of different user states according to an embodiment of the present invention.
[0027] Fig. 22 is a schematic diagram of a method for providing assistance to a user of an eyewear device in combination with a user scenario and a status according to an embodiment of the present invention.
[0028] Fig.23 is a schematic diagram of a method for providing different levels of assistance to a user of an eyewear device according to an embodiment of the present invention.
[0029] Fig.24 is a schematic diagram of a system for providing assistance to a user of an eyewear device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In the following detailed description of multiple embodiments of the present invention, reference is made to multiple drawings, in which similar reference numerals indicate similar elements, and specific embodiments in which the present invention can be practiced are shown in the drawings by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. In other examples, well-known circuits, structures, and techniques are not shown in detail to avoid blurring the understanding of this specification. Therefore, the following detailed description will not be understood as limiting, and the scope of the present invention is limited only by the appended claims.
[0031] In one or more embodiments, methods, devices, and systems for providing modularity of eyewear systems for users are described. As described in the following embodiments, various combinations and configurations of electronic devices are described as being incorporated into eyewear devices. Configurations of certain electronic devices are removably coupled to eyewear devices. In some embodiments, the configuration of electronic devices is built into eyewear devices. In other embodiments, the rear neck module assembly is detachably coupled to the eyewear device. In various embodiments, the modular reconfigurable eyewear device provides information to the user through the eyewear device. As described in the embodiments, the information includes streaming audio in the form of music, and the information also includes biological parameters of the user (e.g., physiological biometrics, biomechanics, etc.), such as but not limited to: heart rate, breathing rate, posture, number of steps, rhythm, etc. The information also includes notifications and information of interest to the user from digital applications embedded in mobile computing platforms (e.g., smartphones, Internet of Things (iOT) devices, etc.) or from natural environments or spatial scenes based on external sensory information, such as but not limited to: geolocation data, or information about the vehicle the user is using, such as: bicycle revolutions per minute (RPM), engine parameters such as RPM, oil pressure, cooling water temperature, wind speed, water depth, air speed, etc. In various embodiments, information is presented to the user through the eyewear device, for example, through an audio broadcast heard by the user and a video broadcast played on a display viewed by the user on the eyewear device, or an image viewed projected onto the user's pupil. Therefore, within the scope of the embodiments taught herein, information will have a broad meaning.
[0032] Figure 1 FIG. 1 is a diagram showing a modular reconfigurable eyewear system according to an embodiment of the present invention. Figure 1, the modular eyeglass device shown in the perspective view is represented by 100. The modular eyeglass device 100 has a frame 102. In various embodiments, the frame 102 is ophthalmologically constructed to provide a rim portion that supports lenses 120 and lenses 122. Lenses 120 and 122 can provide any function provided by the eyeglass device, such as but not limited to, safety glass lenses, prescription lenses, sunglass lenses, welding glass lenses, etc. The eyeglass device can also include a single lens instead of the double lenses shown in the figure. In some embodiments, a nose pad is provided to provide a cushion for the contact area with the user's nose. In some embodiments, the nose pad is made of a flexible material such as silicone rubber.
[0033] Temple 104 (left temple) and temple 114 (right temple) are coupled to frame 102. Temples 104 and 114 may be flexibly coupled to frame 102 by hinges as shown, or temples 104 and 114 may have a fixed orientation relative to frame 102.
[0034] In various embodiments, one or more temples (104 and 114) and the frame 102 may be equipped with electronic devices as described below. In view 100, the left temple insert module (TIM) 106 is configured with the left temple 104, and the right temple insert module (TIM) 116 is configured with the right temple 114. The temple insert modules (TIMs) are described more fully below in conjunction with the accompanying drawings.
[0035] Continue to refer Figure 1 , a modular eyewear device is represented by 130 in a perspective view. The frame 132 is ophthalmologically configured to surround the lens 140 and the lens 142 with a frame edge, thereby fixing the lens 140 and the lens 142 thereon. The brow bar 146 is fixed to the frame 132 in various ways by assembly fasteners, adhesives, etc. The temple 144 includes a left temple connector 152, which is rotatably coupled to the left frame connector 150. The left frame connector 150 and the left temple connector 152 together form a rotatable mechanical and electrical connection between the frame 132 and the left temple 144, thereby providing one or more electrical pathways to connect the frame 132 to the left temple 144. Similarly, the right temple 134 is rotatably coupled to the frame 132 by the right hinge assembly 148.
[0036] It should be noted that in some embodiments, the modular eyewear device is configured so that each temple can be removed from its hinge via an electrical / mechanical connector having one or more electrical contacts, which are not shown for clarity of illustration. These electrical contacts can be made using, for example, pins, points, pads, slots, contact devices, etc. For example, the line indicated by 154 demarcates the boundary where the right temple connector is assembled with the right temple 134. Similarly, the line indicated by 156 demarcates the boundary where the left temple connector 152 is assembled with the left temple 144.
[0037] By providing an electrical / mechanical connector between each temple, such as 134, 144, and the frame 132, the temples can be interchangeable with the eyeglass device. This function allows the user to interchange one temple with another temple. Different temples can be configured to have different electronic devices to provide different functions as described herein. Any temple can be configured to accommodate various electronic device configurations. For example, in one or more embodiments, the right interchangeable temple accommodates an electronic component, which may include a biosensor, a biomechanical sensor, an environmental sensor, a temperature sensor, an acoustic sensor, a motion sensor, a light sensor, a touch sensor, a proximity sensor, a speed sensor, an acceleration sensor, a rotation sensor, a magnetic field sensor, a global positioning system (GPS) receiver, a cable, a microphone, a micro speaker, a power source (battery), a camera, a micro display, a head-up display (HUD) module, a multi-axis inertial measurement unit, and a wireless communication system. It should be noted that the TIM may also include the above-mentioned electronic components and sensors. In various embodiments, one or more wireless communication systems are provided that use, for example, near field communication (NFC) using the Industrial-Scientific-Medical (ISM) 13.56 MHz frequency, Adaptive Network Topology (ANT) ANT+ wireless standards, wireless communication using the Bluetooth standard, Bluetooth Low Energy (BLE) standard, wireless communication using the Wi-Fi standard, and wireless communication using mobile phone standards, such as 3G, 4G, Long Term Evolution (LTE), 5G, or other wireless standards. In some embodiments, the electrical pathway from the electronics is led out of the temple via the sheath cavity, then into the temple sheath and continuing into the brow bar sheath cavity. The right interchangeable temple includes a hinge connector 148 that is secured to the brow bar 146 and the frame 132.
[0038] In one or more embodiments, the right interchangeable temple is fixed to the front of the frame by a hinge connector, which allows power and data to be transferred to the left interchangeable temple through the modular brow bar. The hinge connector is mechanically interlocked with the frame and allows power / data connection with the electrical pin conductor. In one or more embodiments, when in the open orientation for wearing, the hinge connector senses the open state of the device, allowing power or data transmission. When in the closed position (the temples are folded inward), the hinge connector, combined with the signals received from one or more proximity sensors and motion sensors, will allow the system to sense the user-device interaction state and turn off power or data transmission. In addition, in the open position, in some embodiments, sensors, such as but not limited to proximity sensors, can detect when the device is worn by the user and can therefore operate in its active (open) state. This function can reduce power consumption when folded and stowed, and can automatically power on when the user wears the device on his or her head. In addition to switching data or power transmission, the hinge connector can also provide a flexible circuit and a wired micro connector to provide stable uninterrupted power and / or data transmission.
[0039] In some embodiments, it is convenient to route electrical pathways within the volume of the brow bar 146. In some embodiments, the brow bar 146 is configured to provide a channel along its length in which the electrical pathways are routed. Therefore, the brow bar 146 provides one or more sheaths, channels, etc., along its length, in which electrical pathways and sensors can be contained. Examples of electrical pathways are, but are not limited to: wires, printed circuit boards, flexible printed circuit boards, etc. In various embodiments, it is preferred to mount one or more sensors to the brow bar 146 to form an electrical subassembly for the frame 132. In some embodiments, additional electrical pathways from the frame 132 are connected to the electrical pathways contained in the brow bar 146. In some embodiments, a flexible electronic circuit is attached to the lower top surface of the brow bar and is led out of the brow bar via the left and right sheath cavities. Alternatively, or in combination, a fully embedded flexible electronic device can be cast into the brow bar, and integrated contact points can be led out of the brow bar near each hinge. These integrated contact points located on both sides of the brow bar allow data or power to be transmitted when in contact with the integrated contact points of the left and right temples. In addition to facilitating connection to electronics, the brow bar can also conceal an optional pupil module via a fixed flange and allow the user to view the microdisplay through the brow bar pupil hole.
[0040] In a similar manner, the left temple is configured as a left interchangeable temple, connected to the front frame of the glasses via a left hinge connector. In various embodiments, the left interchangeable temple may include the same electronic device configuration / functionality as the right interchangeable temple, or the interchangeable temple may include a different electronic device configuration and different functionality.
[0041] Continue to refer Figure 1, the left temple 164 and the right temple 174 are configured as shown at 160. Each of the temples 164 and 174 contains an electronic device that is configured to provide information and functions to the user of the eyeglass system. As shown at 160, the left temple 164 has a temple door (not shown), which is removed to expose the electronic assembly represented by 186. The temple door protects the electronic device from environmental exposure and hazards. For specific applications, the temple door is fastened to the temple assembly using a suitable mechanical fastener. In some embodiments, the temple door provides a level of water intrusion through the IP code from the international protection mark IEC standard 60529. The power supply (battery) is represented by 184, and the audio speaker and port are represented by 182. The audio speaker and port 182 are generally located at the end of the temple, and in some embodiments are integrated directional projection speakers that can privately direct sound to the user's ears. The projection stereo speaker can transmit various audio signals to the user, such as but not limited to voice prompts, streaming music, smart audio assistance, data, etc. It is worth noting that the projection speaker design does not block the user's ears. Therefore, the user can hear far-field sound, and the sound quality of this far-field sound is not degraded as when using currently available earbud headphones that block the user's ears.
[0042] The right temple 174 is also provided with an electronic assembly (not shown), which is contained in the right temple 174. The right temple is provided with an audio speaker having an audio speaker port 184, which can be an integrated directional projection speaker. In one or more embodiments, the right temple 174 is configured to accommodate an external assembly 190 including a microdisplay assembly 192. Similarly, the left temple can also be configured for the external assembly 190 and the microdisplay assembly 192.
[0043] In various embodiments, the microdisplay assembly, such as 192, is a head-up display (HUD) pupil TM An optical module that houses the optical devices, electronic devices, and micro displays that make up the optical system. The pupil mechanism can also accommodate cables, flexible circuit boards, or wires that are led out of the housing and into the electronic device contact paths. In one or more embodiments, these electrical paths are connected to one side of the left temple 174 to allow the user to have a transparent head-up display external accessory to enhance the visual component of the mobile assistance and / or augmented reality experience.
[0044] In one or more embodiments, the wiring leading out from the brow bar is hidden in the left and right sheaths of the temples and enters the left and right temples through the sheath cavities, thereby protecting the wiring from environmental hazards. The vicinity of the contact point passage can also accommodate a movement mechanism for customizing the pupil distance of the head-up micro-display module.
[0045] In various embodiments, the front frame portion, such as 102 or 132 ( Figure 1 ) or any similar structure in the following figure, and right and left temple portions, such as 104, 114, 134, 144, 164, 174 ( Figure 1 ) or any similar structure in the figure below, can be part of a set of interchangeable front frame parts and temple parts, each of which has the same or different combinations of devices, accessories, capabilities and / or functions. At least one of an electronic board, a microphone, a speaker, a battery, a camera, a head-up display module, a wireless Wi-Fi radio, a GPS chipset, an LTE cellular radio, a multi-axis inertial measurement unit, a motion sensor, a touch sensor, a light and proximity sensor, etc. can be included in the desired combination. It can further include an electronic device that allows a user to perform at least one of the following: wirelessly connect to a cellular service, a smart phone, a smart watch, a smart bracelet, a mobile computer, and a sensor peripheral device. The electronic device can further include the following capabilities: viewing a perspective augmented reality image through a modular head-up display (HUD), providing stereo audio content, and providing voice and audio notifications including music through one or more integrated projection micro speakers. The front frame electrical contact means and the temple electrical contact means may include electrical contacts located in or near the respective hinge connectors for removable electrical contact with each other to electrically transmit at least one of: power, electrical signals and data between the temple portion and the front frame portion when the contacts are in an electrically closed position. In some embodiments, when assembled together, the front frame hinge connector, the front frame electrical contact means, the temple hinge connector and the temple electrical contact means may form an electromechanical hinge, hinge connector, assembly or device. In some embodiments, the system power can be turned off by folding the temple portion into a storage position thereby disconnecting the contacts.
[0046] In some embodiments, at least one of the front frame electronics and the temple electronics may include a battery, a camera, a heads-up display module, a controller, digital storage electronics, a CPU, a projected micro speaker, a microphone, a wireless Wi-Fi radio, a GPS chipset, an LTE cellular radio, a multi-axis inertial measurement system or unit, and at least one of a sense, motion, touch, light, proximity, temperature and pressure sensor, etc.
[0047] In some embodiments, at least one temple may include a temple module insert (TIM) containing selected temple electronics mounted thereon. In other embodiments, a neck smart cord is electrically connected to the back neck electronics module. The neck smart cord has right and left connectors or connector ends for mechanically and / or electrically interconnecting the back neck electronics module with the right and left temples of the eyewear device.
[0048] Figure 2According to an embodiment of the present invention, a reconfigurable component for an eyewear device is shown. Figure 2 At 200 in the figure, a joint 204 is provided on the temple 202. In the description of the present embodiment, the temple insertion module indicated by 210 is referred to as a "TIM", which is configured to be detachably coupled to the joint 204 of the temple 202. As shown by arrows 212a and 212b, the TIM 210 is installed in the temple 202. In various embodiments, the joint 204 is implemented by a mechanical connection, such as but not limited to: press fit, clips, mechanical interlocking, hooks and loops, magnetic surfaces, external clamps of the temple, flanges and mechanical connections with the temple, etc. In other embodiments, the joint 204 and the TIM 210 utilize a magnetic surface, thereby fixing the TIM 210 by magnetic force. The appearance of the joint shown at 204, as well as the appearance of any joint shown elsewhere in the drawings given herein, is for illustration only and does not constitute a limitation to the embodiments of the present invention. In the view shown in 200 , the TIM 210 is only mechanically connected to the temple 202 , and there is no electrical connection between the TIM 210 and the temple 202 .
[0049] In some embodiments, a speaker and a speaker port 214 are provided on the TIM 210, which may be a miniature projection speaker. The speaker provides information to the user through directional audio broadcasting. It should be noted that the speaker provided herein is a speaker located outside the user's ear, and therefore does not insert into the user's ear like an insert earplug. The TIM 210 is configured with an electronic component that includes a processor, a memory, a power supply, and one or more wireless communication protocols that enable the TIM 210 to communicate 222 wirelessly with one or more devices 220. The device 220 may be an external sensor, such as, but not limited to: a biosensor or a vehicle sensor, a local user device, a network node, such as a wireless router, a remote network or a remote user device such as a mobile phone accessed through a network. The following is a more comprehensive description of different sensors, networks, and remote devices in conjunction with the accompanying drawings.
[0050] follow Figure 2In the structure shown at 200, in some embodiments, a second temple and a second TIM are provided. The two TIMs in such a system can participate in wireless communication between the device 220 and between themselves as needed to provide a certain degree of design functionality to the user. For example, in one embodiment, the left TIM includes a wireless network capability that is sufficient to communicate with the remote device 220 using a first network protocol. In addition, the left TIM and the right TIM have wireless network capabilities that support communication using a second network protocol. In order to save power, the first network protocol has a greater range than the second network protocol because the distance between the left TIM and the remote device is greater than the separation distance between the left TIM and the right TIM (nominal width of the user's head). The structure shown at 200 is called true wireless because there is no wired connection between the left TIM and the right TIM. In one or more embodiments, an audio stream is provided from a user device to the first TIM using a first wireless network. Then, a second wireless audio stream is provided from one TIM to another TIM using a second wireless network so as to provide the audio stream to each of the left and right projection speakers of the eyeglass device.
[0051] As described in conjunction with the drawings herein, the temples and TIMs described at 200 provide a reconfigurable assembly for an eyeglass device. The front end 206 of the temple 202 can be engaged with a frame frame of the eyeglass device as described above, with or without a connector between the temple and the frame. Thus, depending on the intended design of the eyeglasses, the temple 202 can obtain a fixed position relative to the frame frame, or the temple can be rotatably connected to the frame frame.
[0052] refer to Figure 2250, a junction 254 is provided on the temple 252. The temple insertion module TIM at 260 is configured to be detachably coupled to the junction 254 of the temple 252. As shown by arrows 262a and 212b, the TIM 260 is installed in the temple 252. In various embodiments, the junction 204 is implemented by a combination of electrical and mechanical connections. The mechanical connection can be as described in combination 210 / 204, for example but not limited to: press fit, clip, mechanical interlock, hook and loop, etc. In other embodiments, the junction 254 and the TIM 260 utilize magnetic surfaces to fix the TIM 210 by magnetic force. For illustration, a plurality of electrical contacts 280 are provided, but this is not meant to be limiting. The electrical contacts 280 cooperate with corresponding electrical contacts in the temple 252 to provide an electrical connection with one or more electrical paths (not shown) in the temple 252. The electrical pathways within the temple 252 facilitate electrical connection between the TIM 260 and one or more sensors 272 and 274, which may also represent sources of signals provided to a display. The sensors 272 and 274 may be acoustic sensors, such as microphones or any of the sensors described herein, for use in conjunction with electronic components configured with the eyewear device. In one or more embodiments, one or more of 272 and 274 provide signals to a display, such as a HUD.
[0053] In some embodiments, a speaker and speaker port 264 are provided on the TIM 260, which may be a miniature projection speaker. The speaker provides information to the user through open-ear audio broadcasting.
[0054] follow Figure 2 The structure shown at 250 in some embodiments provides the following Figure 3 The second temple and the second TIM shown. The two TIMs in such a system participate in wireless communication between the device 220 and between themselves as needed to provide a certain degree of design functionality to the user. As described in conjunction with the figures herein, the temple and TIM described at 250 provide a reconfigurable component for the eyeglass device. For example, the front end 256 of the temple 252 can be engaged with the frame frame of the eyeglass device as described above, with or without a connector between the temple and the frame. Therefore, depending on the established design of the glasses, the temple 252 can obtain a fixed position relative to the frame frame, or the temple can be rotatably connected to the frame frame.
[0055] Figure 3 A plurality of reconfigurable components for eyewear apparatus are shown at 300 according to an embodiment of the present invention. Figure 3 300 of them, from Figure 2 The left reconfigurable assembly 250 is shown with a companion right reconfigurable assembly for an eyewear device. The right temple 352 has Figure 3360 is configured to be detachably coupled to the joint of the temple 352. As shown by arrows 362a and 362b, TIM360 is installed in the temple 352. In various embodiments, the joint of the temple 352 is realized by a combination of electrical and mechanical connection. The mechanical connection can be provided in combination with 210 / 204 as described above, such as but not limited to: press fit, clip, mechanical interlock, hook and loop, etc. In other embodiments, the joint of the temple 352 and TIM 360 utilize magnetic surfaces, thereby fixing TIM 360 by magnetic force. For illustration, multiple electrical contacts 380 are provided, but this does not mean limitation. The electrical contacts 380 cooperate with corresponding electrical contacts in the temple 352 to provide electrical connection with one or more electrical paths (not shown) in the temple 352. The electrical pathways within the temple 352 facilitate electrical connections between the TIM 360 and one or more sensors 372 and 374. The sensors 372 and 374 can be acoustic sensors, such as microphones or any of the sensors or displays described herein, for use in conjunction with an electronic component configured with the eyewear device. In various embodiments, the TIM 360 is configured with an electronic component that includes a processor, a memory, a power supply, and one or more wireless communication systems that use a protocol that enables the TIM 360 to perform wireless transmissions shown in 222 with one or more devices 220. In addition, the TIM 360 and the TIM 260 can be configured to have wireless communication capabilities that allow wireless communication between the TIMs, such as wireless transmissions shown in 382. In some embodiments, a speaker and a speaker port 364 are provided on the TIM 360, which can be a miniature projection speaker. The speaker provides information to the user through audio broadcasting.
[0056] refer to Figure 3 390, showing the electrical connection schematic of each TIM 260 and TIM 360. TIM 260 is electrically connected to sensor 272 via electrical pathway 394. Similarly, TIM 260 is electrically connected to sensor 274 via electrical pathway 392. The connectivity shown between TIM 260 and each sensor constitutes the left temple electrical schematic 384. It should be noted that the left temple electrical schematic 384 can be more or less complex than shown. Therefore, the left temple electrical schematic is only for illustration and is not meant to be limiting.
[0057] Similarly, TIM 360 is electrically connected to sensor 372 via electrical pathway 398. TIM 360 is electrically coupled to sensor 374 via electrical pathway 396. The connectivity shown between TIM 360 and the sensors constitutes a right temple electrical schematic 386. It should be noted that the right temple electrical schematic 386 can be more or less complex than shown. Therefore, the right temple electrical schematic is for illustration only and is not meant to be limiting.
[0058] The two TIMs in such a system participate in wireless communications between the devices 220 and between themselves as needed to provide a degree of designed functionality to the user. For example, in one embodiment, the left TIM includes wireless networking capabilities that are sufficient to communicate with the remote device 220 using a first network protocol. In addition, the left TIM and the right TIM have wireless networking capabilities that support wireless communications as shown at 382. The wireless communications 382 may be performed using a second network protocol that is different from the protocol used at 222. To save power, the first network protocol (222) has a greater range than the second network protocol (382) because the separation distance between the left TIM 260 and the remote device 220 is greater than the separation distance between the left TIM 260 and the right TIM 360, the latter being nominally the width of a user's head, while the former may be as far as the distance to a mobile phone cellular tower.
[0059] Figure 4 According to an embodiment of the present invention, another reconfigurable modular eyewear system is shown. Figure 4 , one or more of the sensors, power components, and computing units are distributed throughout the eyeglass device, including throughout the frame frame such as 402. The frame frame 402 is ophthalmologically constructed to surround the lens 440 with a frame edge, thereby fixing the lens 440 thereon. The left lens 404 and the right lens 414 are connected to the frame frame 402 to form the eyeglass device. The left lens 404 is configured to have a joint shown in 408. The left temple insert module (TIM) 406 is configured to engage with the joint 408 as described above, thereby providing a mechanical and electrical connection between the TIM406 and the temple 404. Similarly, as shown in the figure, the right temple 426 engages with the joint 414 of the right temple. TIM406 includes an audio speaker and an audio port as shown in 410, and TIM426 includes an audio speaker and an audio port as shown in 430. In various embodiments, the audio speakers 410 and 430 are projection speakers. The eyewear device includes a plurality of sensors and displays, 462, 464, 466, 468, and 470, which are integrated into an electrical path that passes through the left temple 404 and then extends to the right temple 414 through the frame frame 402. In various embodiments, there may be more than Figure 4More sensors or fewer sensors are shown. Figure 4 The sensors and sensor locations shown in are provided as examples only and do not constitute limitations on embodiments of the present invention. As described above in conjunction with the previous figures, at least one of the temple insert modules 406 and / or 426 is provided with a set of electronic devices necessary to provide the wireless connection 222 to the device 220.
[0060] In the eyeglass device 400, a high-level view of an electrical pathway diagram is shown at 480. Referring to 480, the left TIM 406 and the right TIM 426 are electrically coupled to sensors 462, 464, 466, 468, and 470 through electrical pathway elements 482, 484, 486, 488, 490, and 492. Electrical pathway elements, such as 484, electrically connect sensor 464. The components shown in 480 collectively provide a modular, reconfigurable set of components for the eyeglass device. In one or more embodiments, one or more acoustic sensors are located in at least one of the frame frame 402, the left temple 404, and the right temple 414. Thus, the acoustic sensor can be located anywhere on the temple (left or right) or the frame frame of the eyeglass device.
[0061] Figure 5 According to an embodiment of the present invention, generally at 500, a Figure 4 Perspective and top views of the modular eyewear system. Figure 5 , a modular eyewear device is shown at 502 in a perspective view. A modular nose pad 504 is detachably coupled to the modular eyewear device 502, as shown at 506. The modularity of the nose pad allows the user to replace the nose pad to improve the fit between the eyewear and the user's nose and facial structure. Higher comfort can be achieved through the modularity of the nose pad of the eyewear device. In addition, other sensors, such as biosensors, can be disposed in the nose pad.
[0062] According to an embodiment of the present invention, Fig. 6A A system architecture for a modular eyewear device is shown generally at 600. Fig. 6AIn various embodiments, the modular reconfigurable eyewear device may include more than one wireless communication system. In various embodiments, the eyewear device 602 has a high-level block diagram structure as shown in 604. In various embodiments, the eyewear device 602 is configured to communicate with a wireless sensor 640 and a mobile device 670. The wireless sensor 640 may include a single sensor or multiple sensors. The wireless sensor 640 may include any one or more sensors listed herein without limitation. For example, the wireless sensor 640 may include a biometric sensor or a biomechanical sensor configured for use with a user, or a sensor configured for use with a vehicle or a building. Some examples of biosensors are, but are not limited to: a heart rate monitor, a perspiration sensor, a temperature sensor, etc. Some examples of vehicle sensors are, but are not limited to: a speed sensor, an acceleration sensor, a global positioning system signal, a vehicle engine parameter, a wind speed indicator, etc. Some examples of sensors used with buildings are, but are not limited to: a temperature reading of a thermostat, a water pressure value, etc. Some non-limiting examples of vehicles are, but are not limited to: a scooter, a bicycle, a car, a boat, a yacht, a ship, an airplane, a military vehicle, a wing suit, etc. In some embodiments, data is received from a special purpose network at 640 and / or 616. For illustration and not by way of imitation, an example of a special purpose network is the National Marine Electronics Association (NMEA) NMEA 2000 network, which is designed for vessels such as yachts (powered or sailing). NMEA 2000, also referred to in the art as "NMEA2k" or "N2K", is standardized as International Electrotechnical Commission (IEC) 61162-1. NMEA 2000 is a plug-and-play communication standard for connecting marine sensors and display units in ships, boats, yachts, etc. The mobile device 670 can be any one or more of the mobile devices listed herein without limitation. For example, the mobile device can be a mobile phone, a watch, a wristband, a bracelet, a tablet computer, a laptop computer, a desktop computer, a car computer, etc.
[0063] The eyewear device 602 has a high-level structure represented by 604, which includes a speaker 606, a central processing unit 608, a power supply 610, a microphone 612, a storage device 614, and a wireless communication system 616. The wireless communication system 616 may include one or more of the following wireless communication systems: for example, a near field communication system 618, a wireless communication system using a Bluetooth communication protocol 620, a wireless communication system using a Wi-Fi communication protocol at 624, and a mobile phone communication protocol 622. The wireless communication protocol indicated by LTE at 622 is given only as an example of a wireless device, and it does not constitute a limitation of the embodiments of the present invention. Those skilled in the art will recognize that one or more antennas are included in the wireless communication system block 616, but are not shown for clarity.
[0064] The wireless sensor 640 has a high-level structure represented by 642, which includes one or more sensors 644 and a wireless communication system 646. The wireless communication system 646 can be a low data rate communication system, such as a near field communication system, BLE, ANT+, or a similar system. Alternatively, the wireless communication system 646 can be provided as a higher data rate system required by the sensor 644.
[0065] The mobile device 670 has a high-level structure represented by 672, which includes a central processing unit 674, a power supply 676, a memory 678, and one or more wireless communication systems shown in box 680. The mobile device 670 can optionally be configured to reach a remote network shown in cloud 689. The wireless communication box 680 can include one or more of the following wireless communication systems: for example: a near field communication system 682, a wireless communication system using a Bluetooth communication protocol 684, a wireless communication system using a Wi-Fi communication protocol at 686, and a mobile phone communication protocol 688. The wireless communication protocol indicated by LTE at 688 is given only as an example of a communication system for a mobile device, and it does not constitute a limitation of the embodiments of the present invention. Those skilled in the art will recognize that one or more antennas are included in the wireless communication system boxes 680 and 642, but are not shown for clarity.
[0066] In some embodiments, the wireless sensor system 642 and the eyeglass device 602 are initially configured by a user of the mobile device 670 and the mobile device user interface, as shown in paths 652a and 652b. In operation, the eyeglass device 602 receives data wirelessly from an appropriate wireless communication system, such as a near field communication system 618, as shown in 650. The wireless data obtained from the wireless sensor system 642 can be transmitted to the user device 670 / 672 via another wireless communication system as shown in 654. The wireless communication shown in 654 can be completed with a higher data rate channel using, for example, the Bluetooth protocol at 620 / 684, or the Wi-Fi network protocol at 624 / 686, or the mobile phone communication protocol shown in 622 / 688. In various embodiments, the data transmitted from the eyeglass device 602 can be stored and analyzed on the user device 670 and have different applications.
[0067] Figure 6B According to an embodiment of the present invention, generally shown at 690 is Fig. 6A The wireless network corresponding to the system architecture for the modular eyewear system in . Figure 6B, the wireless communication box 616 can be connected to multiple devices as shown. For example, one or more wireless sensors 640 can be connected to the wireless communication box 616 using a low data rate near field communication network as shown in 618. One or more user devices 670 can use the Bluetooth communication protocol as shown in 620 to communicate wirelessly with the wireless communication box. As shown in 624, one or more wireless nodes, such as the Wi-Fi node shown in 692, can communicate wirelessly with the wireless communication box 616. One or more remote networks 694 can use the cellular communication protocol as shown in 622 to communicate wirelessly with the wireless communication box 616. Therefore, the reconfigurable eyeglass device can include one or more wireless communication systems shown at 690. The eyeglass device can be reconfigured for different wireless communications by, for example, replacing one TIM module with another. Alternatively, one or more temples can be interchangeable with the above-mentioned frame frame to provide customized functionality to the eyeglass device.
[0068] Figure 7 According to an embodiment of the present invention, a method for Figure 4 Another system structure of the modular eyewear device in. Figure 7 , the wireless communication module 616 of the eyeglass device 602 can be configured to perform cellular communication directly via a mobile phone network without the need for a user device to act as an intermediary. For example, in 700, the eyeglass device 602 is configured to communicate with a remote device 702 through a wireless communication system 622, wherein the remote device 702 can be a mobile phone, thereby directly connecting to the remote device 702 through an external network shown in the cloud 704. No intermediate user mobile device is required to support such a communication line. This configuration of the eyeglass device allows a user of the eyeglass device to make a call from the eyeglass device with the help of an interface, such as a voice interface, one or more tactile interfaces similar to buttons, etc. The voice interface provides command and control of the phone call by converting the user's voice signal into a command that the device uses to cause the wireless network operation for the phone call. Examples of such commands are, but are not limited to: selecting a caller, making a call, turning up the volume, turning down the volume, ending the call, etc.
[0069] Figure 8 A block diagram of a temple insertion module (TIM) is generally shown at 800 in accordance with an embodiment of the present invention. Figure 8As used in the description of this embodiment, the TIM may be based on a device such as a computer, in which embodiments of the present invention may be used. The block diagram is a high-level conceptual representation that may be implemented in various ways and with various architectures. The bus system 802 interconnects a central processing unit (CPU) 804 (or processor as referred to herein), a read-only memory (ROM) 806, a random access memory (RAM) 808, a memory 810, an audio 822, a user interface 824, and a communication 830. The RAM 808 may also represent a dynamic random access memory (DRAM) or other form of memory. In various embodiments, the user interface 824 may be a voice interface, a touch interface, a physical button, or a combination thereof. It should be understood that a memory (not shown) may be included in the central processing unit block 804. The bus system 802 may be, for example, one or more such buses, such as a system bus, a peripheral component interconnect (PCI), an advanced graphics port (AGP), a small computer system interface (SCSI), an Institute of Electrical and Electronics Engineers (IEEE) Standard No. 994 (FireWire), a universal serial bus (USB), a universal asynchronous receiver / transmitter (UART), a serial peripheral interface (SPI), an integrated circuit (I2C), etc. The central processor 804 may be a single, multiple, or even distributed computing resource. The memory 810 may be a flash memory, etc. It should be noted that, depending on the actual implementation of the TIM, the TIM may include some, all, more, or rearranged components in the block diagram. Therefore, Figure 8 Many variations of the system are possible.
[0070] A connection to one or more wireless networks 832 is obtained through communication (COMM) 830, which enables TIM800 to wirelessly communicate with local sensors, local devices, and remote devices on remote networks. In some embodiments, 832 / 830 provides access to a remote speech-to-text conversion system, which can be located at a remote location such as a cloud-based one. 832 and 830 flexibly represent wireless communication systems in various implementations, and can represent various forms of telemetry, general packet radio service (GPRS), Ethernet, wide area network (WAN), local area network (LAN), Internet connection, Wi-Fi, WiMAX, ZigBee, infrared, Bluetooth, near field communication, mobile phone communication systems, such as 3G, 4G, LTE, 5G, etc., and combinations thereof. In various embodiments, a touch interface is optionally provided at 824. Signals from one or more sensors are input to the system via 829 and 828. At 826, global positioning system (GPS) information is received and input to the system. Audio can represent a speaker, such as a projection speaker or projection micro speaker described herein.
[0071] In various embodiments, different wireless protocols are used in the network to provide the system described in the above-mentioned figures, depending on the hardware configuration. A non-limiting embodiment of the technology for wireless signal transmission is the Bluetooth wireless technology standard, which is also commonly referred to as the IEEE 802.15.1 standard. In other embodiments, a wireless signal transmission protocol called Wi-Fi is used, which uses the IEEE 802.11 standard. In other embodiments, the ZigBee communication protocol based on the IEEE 802.15.4 standard is used. These examples are given only for illustration, and they do not constitute limitations on different embodiments. Transmission Control Protocol (TCP) and Internet Protocol (IP) are also used in different embodiments. The embodiments are not limited to the data communication protocols listed herein, and are easy to use with other data communication protocols not specifically listed herein.
[0072] In various embodiments, components in the system and the system described in the previous figures (e.g., temple insertion module (TIM)) are implemented in an integrated circuit device, which may include an integrated circuit package containing an integrated circuit. In some embodiments, the components in the system and the system are implemented in a single integrated circuit die. In other embodiments, the components in the system and the system are implemented in more than one integrated circuit die of an integrated circuit device, which may include a multi-chip package containing the integrated circuit.
[0073] Fig. 9 According to an embodiment of the present invention, a modular eyewear device equipped with a rear neck module assembly is shown. Fig. 9 , at 900, a back neck module assembly is mounted to a pair of passive glasses. Passive glasses mean that there are no electronics in the glasses. Alternatively, the glasses can be active or powered glasses, as described herein, configured with electronic components packaged into one or more temples or temple insert modules (TIMs). The glasses have a frame frame 902 containing lenses 906. A left temple 904 and a right temple 914 are connected to the frame frame 902. The back neck module assembly includes a back neck electronic pod (ePOD) 924, a left temple interlock 920, a right temple interlock 922, a left smart cord 926, and a right smart cord 928. The left smart cord 926 electrically and mechanically couples the ePOD 924 to the left temple interlock 920, and the right smart cord electrically and mechanically couples the ePOD 924 to the right temple interlock 922.
[0074] The left temple interlock 920 includes an acoustic cavity, an audio speaker, and an acoustic port. The acoustic port of the left audio speaker is indicated by 930. The left smart cord 926 includes an electrical conductor that provides an audio signal to the audio speaker contained in the left temple interlock 920. In one or more embodiments, the audio speaker contained in the left temple interlock is a miniature projection speaker. Similarly, the acoustic port of the right audio speaker is indicated by 932. The right smart cord 928 includes an electrical conductor that provides an audio signal to the audio speaker contained in the right temple interlock 922. In one or more embodiments, the audio speaker contained in the right temple interlock is a miniature projection speaker.
[0075] In various embodiments, the ePOD 924 includes an electronics unit. The electronics unit includes the electronic components and functions described herein for the temple insert module (TIM). In other words, the electronics unit is a mechanical and electrical packaged TIM for the back neck module assembly.
[0076] Electronic units having different electronic configurations and functions may be swapped in and out of the ePOD in a manner similar to how different TIMs are swapped in and out of the temples of an eyewear device.
[0077] At 950, length adjustment is provided to shorten or lengthen the right and left smart cords. A back neck electronic pod (ePOD) 954 is configured with a left smart cord 956 and a right smart cord 958 leading from the same end of the ePOD 954. This configuration of the smart cords 956 and 958 allows a slider 960 to be moved away from or toward the ePOD. Moving the slider 960 away from the ePOD 954 shortens the available free length of the smart cords 965 / 958. Moving the slider 960 toward the ePOD 954 increases the available free length of the smart cords 956 / 958.
[0078] In one or more embodiments, in operation, when in the "on" state, audio data is streamed to the electronic unit in the ePOD 924 and directed to the left speaker and the right speaker for broadcasting the audio data to the user when the rear neck module assembly is mounted on the eyewear device and the user wears the eyewear device.
[0079] Fig.10 A rear neck module assembly configured with a wearable device according to an embodiment of the present invention is generally shown in perspective view at 1000. Fig.10, a first sensor 1050 is shown on the ePOD 924. A second sensor 1052 is shown incorporated into the right temple interlock 922. A third sensor 1054 is shown incorporated into the left temple interlock 920. Sensors 1050, 1052, and 1054 can be any of the sensors previously described herein for use with a TIM or directly with electronics built into the temple.
[0080] like Fig.10 In the illustrated embodiment, each temple interlock module, such as 920 and 922, contains a through hole into which the temple of the glasses is inserted. In this embodiment, the temple interlock modules 920 and 922 are made of a flexible material, such as an elastomer or rubber, which allows sufficient elongation to allow the temple to be inserted therein. For example, the left temple interlock 920 includes a through hole 1040 into which the left temple 904 is inserted. The right temple interlock 922 includes a through hole 1042 into which the right temple 914 is inserted. Each temple interlock 920 and 922 is located on a pair of compatible glasses so that each speaker port 930 and 932 is located in front of and near the user's ear. Compatible glasses are glasses that are compatible with the mechanical attachment provided by the temple interlock.
[0081] Fig.11 An interlocking coupling of a temple to a temple according to an embodiment of the present invention is generally shown at 1100 and 1150. Fig.11 , a magnetic temple interlock is shown at 1100. The magnetic temple interlock includes a magnetic region 1108 on a temple 1102 of an eyeglass device. Temple interlock 1104 has a corresponding magnetic region 1106. In operation, magnetic regions 1106 and 1108 are brought together, thereby causing magnetic regions 1106 and 1108 to attract each other, thereby providing a clamping force between temple interlock 1104 and temple 1102. A port of an acoustic cavity containing a speaker is indicated by 1110.
[0082] Another clamping method is shown by 1150. The temple interlock 1152 includes a slot 1158 between a first side 1156a and a second side 1156b of the flexible material. The geometry of 1158, 1156a, and 1156b forms a U-shape into which the temple of the eyeglass device can be inserted. The elasticity of the material 1152 provides a removable coupling between the temple interlock 1152 and the temple of the eyeglasses (not shown). The acoustic port of the acoustic cavity that houses the speaker is shown by 1154.
[0083] Fig.12 Coupling the back neck module assembly to the electronics in the temple according to an embodiment of the present invention is generally shown at 1200. Fig.12, the back neck module assembly is coupled to the electronics contained in the temple. A portion of the back neck module assembly is shown as having a back neck electronic pod (ePOD) 1220, a left smart cord 1222, and a left temple interlock 1210. As previously described, any electronics contained in the temple can be directly contained in the temple without the temple insertion module (TIM). Alternatively, the electronics contained in the temple can be an electronics that is part of the TIM, optionally as shown in 1204. In either case, a plurality of electrical contacts are provided on the temple 1202, as shown in 1206. A corresponding number of electrical contacts 1208 are provided in the left temple interlock 1210. A mechanical interlock is provided between the temple 1202 and the left temple interlock 1210 so that the connection between 1210 and 1202 is detachably coupled. In one or more embodiments, a magnetic coupling is provided near or at a position of 1206 / 1208 to provide a detachable coupling.
[0084] Fig.13 A schematic diagram of combining a rear neck module assembly with temple electronics is generally shown at 1300 in accordance with an embodiment of the present invention. Fig.13 , the outline of the eyeglass device is represented by 1302. The eyeglass device 1302 contains electronic devices and / or electronic pathways in the left temple, the right temple and the frame of the glasses. The outline 1302 includes the frame of the glasses, the left temple and the right temple. In the system shown in the figure, the electronic pathway 1308 extends between the left temple and the right temple of the eyeglass device 1302.
[0085] The eyeglass device includes a left temple insert module (TIM) 1304 located at the left temple and a right temple insert module 1306 located at the right temple. A back neck module assembly configured with an electronic unit (ePOD) is represented by 1310. A left smart cord 1312 provides an electrical path between the ePOD 1310 and the left TIM 1304. A right smart cord 1314 provides an electrical path between the ePOD 1310 and the right TIM 1306. In various embodiments, both the left TIM 1304 and the right TIM 1306 are configured with one or more wireless communication network systems that allow wireless communication between the left TIM 1304 and the right TIM 1306, as shown in 1316. Remote device 1320 represents one or more wireless sensors or wireless user devices, as described above in conjunction with the aforementioned figures. Wireless communication 1322 is accomplished between the remote device 1320 and at least one of the left TIM 1304, the right TIM 1306, and the ePOD 1310. All of the above descriptions of electronic system functions with respect to the TIM are applicable to the ePOD, such as the ePOD 1310.
[0086] In some embodiments, the left temple is not electrically connected to the right temple, such as if electrical pathway 1308 were removed from the electrical schematic shown in 1300 .
[0087] Fig.14 A user interface on a rear neck module assembly is generally shown at 1400 according to an embodiment of the present invention. Fig.14 , represented by 1402 as a back neck electronic pod (ePOD). The ePOD 1402 has a display interface 1404. In different embodiments, the display interface 1404 can be implemented in various ways. In some embodiments, the user interface is a tactile surface button. In some embodiments, the user interface is implemented using a touch screen, such as a capacitive touch screen that presents one or more controls to the user. In some embodiments, the user interface communicates information to the user. In other embodiments, the user interface communicates information to a person viewing the user interface 1404 from behind the user wearing the ePOD 1402. An example of this information is, but is not limited to, an emoticon, an emotional state, an icon, etc., as shown in 1406.
[0088] Fig.15 A block diagram of an electronic unit for a neck-mounted electronic pod (ePOD) is generally shown at 1500 in accordance with an embodiment of the present invention. Fig.15, as used in the description of this embodiment, the back neck electronic unit can be based on a device such as a computer, and the embodiment of the present invention can be used in this device. The block diagram is a high-level concept representation, which can be implemented in various ways and various architectures. The bus system 1502 interconnects a central processing unit (CPU) 1504 (or a processor referred to herein), a read-only memory (ROM) 1506, a random access memory (RAM) 1508, a memory 1510, an audio 1522, a user interface 1524, and a communication 1530. RAM 1508 can also represent a dynamic random access memory (DRAM) or other forms of memory. In various embodiments, the user interface 1524 can be a voice interface, a touch interface, a physical button, or a combination thereof. It should be understood that a memory (not shown) can be included in the central processing unit block 1504. The bus system 1502 may be, for example, one or more such buses, such as a system bus, a peripheral component interconnect (PCI), an advanced graphics port (AGP), a small computer system interface (SCSI), an Institute of Electrical and Electronics Engineers (IEEE) Standard No. 994 (FireWire), a universal serial bus (USB), a universal asynchronous receiver / transmitter (UART), a serial peripheral interface (SPI), an inter-integrated circuit (I2C), etc. The central processor 1504 may be a single, multiple, or even distributed computing resource. The memory 1510 may be a flash memory, etc. It should be noted that, depending on the actual implementation of the back neck electronic unit, the back neck electronic unit may include some, all, more, or rearranged of the components in the block diagram. Therefore, Fig.15 Many variations of the system are possible.
[0089] A connection to one or more wireless networks 1532 is obtained through communication (COMM) 1530, which enables the back neck electronic unit 1500 to wirelessly communicate with local sensors, local devices, and remote devices on remote networks. In some embodiments, 1532 / 1530 provides access to a remote voice-to-text conversion system, which can be located at a remote location such as a cloud-based one. 1532 and 1530 flexibly represent wireless communication systems in various implementations, and can represent various forms of telemetry, general packet radio service (GPRS), Ethernet, wide area network (WAN), local area network (LAN), Internet connection, Wi-Fi, WiMAX, ZigBee, infrared, Bluetooth, near field communication, mobile phone communication systems, such as 3G, 4G, LTE, 5G, etc., and combinations thereof. In various embodiments, a touch interface is optionally provided at 1524. An optional display is provided at 1520. Signals from one or more sensors are input to the system via 1529 and 1528. At 1526, global positioning system (GPS) information is received and input to the system. The audio may represent a speaker, such as a projection speaker or projection micro speaker as described herein.
[0090] In various embodiments, different wireless protocols are used in the network to provide the system described in the above-mentioned figures, depending on the hardware configuration. A non-limiting embodiment of the technology for wireless signal transmission is the Bluetooth wireless technology standard, which is also commonly referred to as the IEEE 802.15.1 standard. In other embodiments, a wireless signal transmission protocol called Wi-Fi is used, which uses the IEEE 802.11 standard. In other embodiments, the ZigBee communication protocol based on the IEEE 802.15.4 standard is used. These examples are given only for illustration, and they do not constitute limitations on different embodiments. Transmission Control Protocol (TCP) and Internet Protocol (IP) are also used in different embodiments. The embodiments are not limited to the data communication protocols listed herein, and are easy to use with other data communication protocols not specifically listed herein.
[0091] In various embodiments, components in the system and the system described in the previous figures (e.g., the back neck electronic unit) are implemented in an integrated circuit device, which may include an integrated circuit package containing an integrated circuit. In some embodiments, the components in the system and the system are implemented in a single integrated circuit die. In other embodiments, the components in the system and the system are implemented in more than one integrated circuit die of an integrated circuit device, which may include a multi-chip package containing the integrated circuit.
[0092] In various embodiments, the description of the embodiments provided herein provides a reconfigurable assembly for a head wearable device. The reconfigurable assembly for the head wearable device includes, but is not limited to, detachable temples, detachable temple insertion modules (TIMs), a rear neck module assembly, an electronic pod ePOD for the rear neck module assembly, and a removable electronic unit for the ePODs.
[0093] In various embodiments, the operation is performed on various TIMs ( Figure 8 800) or for various rear neck electronic units ( Fig.15 1500) is one or more algorithms that provide useful functions to users on a data processing system created by a computer program product (such as ... Similarly, in other embodiments, the algorithm or a portion of the algorithm will run on a combination of a back neck electronic unit and a device such as device 220, 670, 702, 1320, etc., as shown in the above figure, or on a combination of a back neck electronic unit, a device (220, 670, 702, 1320, etc.), and cloud-based hardware and services available via one or more communication networks as described above in conjunction with the previous figures.
[0094] Fig.16 A process for providing assistance to a user of an eyewear device is generally shown at 1600 according to an embodiment of the present invention. Fig.16 , the process starts at block 1602. At block 1604, data is received from one or more sensors associated with the eyewear device. The eyewear device can be any of the eyewear devices described above, such as a modular reconfigurable eyewear device using one or more temple insertion modules (TIMs), or a modular eyewear device equipped with a back neck module assembly. In addition, the eyewear device does not need to be a modular device, and the eyewear device can be an eyewear device without a modular design.
[0095] The one or more sensors in block 1604 may be, for example but not limited to, any of the above sensors or a combination of sensors, including a camera, a microphone, a global positioning system (GPS) receiver, a multi-axis inertial measurement unit including an accelerometer, a gyroscope, a magnetometer, a proximity sensor, a motion sensor, a gyroscope, a magnetometer, a display, a temperature sensor, a pressure sensor, an infrared sensor, a biometric sensor, and a biomechanical sensor, wherein the biomechanical sensor is, for example but not limited to, an electrodermal activity (EDA) sensor, and the electrodermal activity (EDA) sensor includes but is not limited to a electrodermal response (GSR) sensor, a heart rate sensor, a breathing rate sensor, etc. Therefore, in various embodiments, the roll, pitch, orientation, position, and speed of the user wearing the eyewear device may be obtained through the data output by the sensors in block 1604.
[0096] At block 1606, the user's scene is determined. As used in the description of this embodiment, the user's scene includes one or more of body posture, activity, and the user's natural environment. The user's scene may change over time and may be a dynamic attribute or a static attribute. Fig.18 The user's scenario is described in more detail. The scenario is determined using the data received in box 1604. In box 1608, help is provided to the user in the form of feedback through the eyeglass device. The feedback can be one or more of audio feedback emitted by a speaker embedded in the eyeglass device, vibration transmitted to the user by the eyeglass device, and feedback from a device located in the user's natural environment transmitted by the eyeglass device. In box 1606, data from one or more sensors are analyzed, for example, tracking the user's head posture and / or user movement or no user movement. Based on the data analysis, in box 1608, help is provided to the user through, for example, communication with the user. The help can be in the form of an audio message broadcast from a speaker embedded in the eyeglass device to remind the user of the negative state the user is in based on the analysis of the scenario and the data from the sensor. A negative state is a state that may cause some harm to the user immediately or at some time in the future, such as, but not limited to, lack of sufficient movement frequency or a long period of time when the head remains stationary. In this example, the harm is damage to the user's biomechanical function. Help can also be help to confirm a positive state, such as confirming that there is enough head movement or enough body activity. The help may be broadcast to the user via one or more speakers associated with the eyewear device. Control of the algorithm implementing the process of 1600 may continuously loop as shown in 1610 to continuously receive data at block 1604, analyze the scene at block 1606, and provide help as needed at block 1608. Real-time help is thereby provided to the user based on the user's scene that changes over time.
[0097] The process ends at box 1612. The trigger for ending the process at box 1612 may be when the user removes the eyeglass device from the user's head. Such an event may be determined by the system through an output from a proximity sensor indicating that the user's head is no longer close to the eyeglass device. In another non-limiting example, the process ends at box 1612 when the user moves the temples of the eyeglass device to a closed position, thereby shutting down the system by a switch triggered by closing the temples. Alternatively, the system may be shut down by a voice command. These examples of initiating box 1612 are provided for illustration only and do not constitute limitations on embodiments of the present invention.
[0098] Fig.17 Receiving data from a sensor associated with an eyewear device is generally shown at 1700 in accordance with an embodiment of the present invention. Fig.17 , receiving data from sensor 1604 may include, in various embodiments, data from a sensor embedded in the eyewear device at 1702. Additionally, data may be received from external sensors of the eyewear device at 1704. An example of an external sensor is, but is not limited to, a sensor on a button of a user's shirt that measures tension on the button, or a sensor that measures pressure on a user's waistband to estimate abdominal distension due to food intake. Similarly, in some embodiments, an ingestible sensor may be used that more accurately estimates caloric intake over a period of time. Such a sensor may be used to provide data that the system uses to provide assistance to the user, for example, "I think you may be full."
[0099] Fig.18 According to an embodiment of the present invention, scene analysis of a user of an eyewear device is generally shown at 1800. Fig.18 , scene analysis is represented by 1802. Scene analysis 1802 includes one or more of body posture shown as 1804, user activities shown as 1806, and the natural environment of the user shown as 1808. Some non-limiting examples of the user's body posture 1804 are, but not limited to, sitting, standing, lying prone, handstand, etc. Some non-limiting examples of activities 1806 are, but not limited to, walking, running, eating, etc. Some non-limiting examples of the user's natural environment 1808 are, but not limited to, cars, trucks, airplanes, trains, bicycles, wheelchairs, social environments, etc. Some non-limiting examples of social environments are, but not limited to, solitary environments, public environments, etc.
[0100] Fig.19 Another process for providing assistance to a user of an eyewear device is generally shown at 1900 according to an embodiment of the present invention. Fig.19, the process starts at block 1902. At block 1904, data is received from one or more sensors associated with the eyewear device. The eyewear device can be any of the eyewear devices described above, such as a modular reconfigurable eyewear device using one or more temple insertion modules (TIMs), or a modular eyewear device equipped with a back neck module assembly. In addition, the eyewear device does not need to be a modular device, and the eyewear device can be an eyewear device without a modular design.
[0101] The one or more sensors in block 1904 may be, for example but not limited to, any of the above sensors or a combination of sensors, including a camera, a microphone, a global positioning system (GPS) receiver, a multi-axis inertial measurement unit including an accelerometer, a gyroscope, a magnetometer, a proximity sensor, a motion sensor, a gyroscope, a magnetometer, a display, a temperature sensor, a pressure sensor, an infrared sensor, a biometric sensor, and a biomechanical sensor, wherein the biomechanical sensor is, for example but not limited to, a skin electrode activity (EDA) sensor, and the skin electrode activity (EDA) sensor includes but is not limited to a skin electrode response (GSR) sensor, a heart rate sensor, a breathing rate sensor, etc. Therefore, in various embodiments, the roll, pitch, orientation, position, and speed of the user wearing the eyewear device may be obtained through the data output by the sensors in block 1904.
[0102] At block 1906, the user's state is determined. The user's state can change over time and can be a dynamic attribute or a static attribute. Fig. 20 and Fig.21The state of the user is described in more detail. The state is determined using the data received in box 1904. In box 1908, help is provided to the user in the form of feedback through the eyeglass device. The feedback can be one or more of audio feedback emitted by a speaker embedded in the eyeglass device, vibration transmitted to the user by the eyeglass device, and feedback from a device located in the user's natural environment transmitted by the eyeglass device. In box 1906, data from one or more sensors are analyzed, for example, tracking the user's head posture and / or user movement or no user movement. Based on the data analysis, in box 1908, help is provided to the user by, for example, communication with the user. The help can be in the form of an audio message broadcast from a speaker embedded in the eyeglass device to remind the user of the negative state the user is in based on the analysis of the state and the data from the sensor. A negative state is a state that may cause some harm to the user immediately or at some time in the future, such as, but not limited to, lack of sufficient movement frequency or the head remaining still for a long time. In this example, the harm is damage to the user's biomechanical function. Help can also be help to confirm a positive state, such as confirming that there is enough head movement or enough body activity. The help may be broadcast to the user via one or more speakers associated with the eyewear device. Control of the algorithm implementing the process of 1900 may continuously loop as shown in 1910 to continuously receive data at block 1904, determine status at block 1906, and provide help as needed at block 1908. Real-time help is thereby provided to the user based on the user's status that changes over time.
[0103] The process ends at box 1912. The trigger for ending the process at box 1912 may be when the user removes the eyeglass device from the user's head. Such an event may be determined by the system through an output from a proximity sensor indicating that the user's head is no longer close to the eyeglass device. In another non-limiting example, the process ends at box 1912 when the user moves the temples of the eyeglass device to a closed position, thereby shutting down the system by a switch triggered by closing the temples. Alternatively, the system may be shut down by a voice command. These examples of initiating box 1912 are provided for illustration only and do not constitute limitations on embodiments of the present invention.
[0104] The process 1900 of providing assistance to the user is ultimately completed without actually determining the user's scenario. For example, in the example given above, the user may be in one or more different social environments (different scenarios) that have not been determined by the system. However, the process is sufficient to allow the system to determine that the user is in a passive state due to lack of movement and provide the user with relevant assistance through the glasses to avoid compromising the user's biomechanical function due to inactivity.
[0105] Fig. 20According to an embodiment of the present invention, the user status analysis of the eyewear device user is generally shown at 2000. Fig. 20 , the process begins at block 2002. At block 2004, the user's state is determined by utilizing sensors embedded in the eyewear device, and in some embodiments, by utilizing external sensors of the eyewear device. At block 2006, assistance is provided to the user by one or more of a virtual intelligent agent or a real person. Examples of both are provided in conjunction with the following discussion.
[0106] Fig.21 According to an embodiment of the present invention, different user states are generally shown at 2100. Fig.21 , which is given for the sake of illustration, and no limitation is implied thereby, the user state analysis 2102 may have subcategories health 2104 , wellness 2106 , safety 2108 , cognitive 2110 , physical 2112 , and emotional 2114 . Fig.19 , Fig. 20 and Fig.21 Together, a system is formed that provides a variety of assistance to the user. For example, the user of the eyewear device may live in his or her home, on a boat, in an RV, in an assisted living facility, etc. A system that configures a medication schedule for a user like this can provide assistance to the user in the form of an audio communication to remind the user to take medication at a specified time. In an example that is intended to be illustrative and not limiting, the system generates an audio message at 11 a.m. to notify the user to take his or her midday medication before lunch. This is a non-limiting example of providing assistance related to the state of health 2104.
[0107] Continuing with the example provided here, which is for illustration only and not meant to be limiting, the system monitors the activity or inactivity of the user. After a predetermined length of inactivity is recorded by one or more sensors in the eyeglass device during a time of day when activity is usually recorded, because of the recorded and analyzed activity and / or inactivity patterns, such as sleeping during the day, the system determines that the user may have a physical or mental illness (e.g., depressive state, flu, etc.) and notifies the user's son or daughter to call the user. The glasses call the son or daughter and inform the user that there has been no activity for a period of time. The user's son or daughter calls the user and checks the user's status or active mood through personal (physical) contact. The call can be made through a phone in the user's room, or directly through an eyeglass device configured for phone calls. Alternatively, the system can generate a virtual intelligent agent to provide communication to the user in this case, thereby suggesting to the user that it is a good idea for the user to get up and walk around. This is a non-limiting example of providing help related to the state of well-being 2106.
[0108] Continuing with the example provided here, which is intended to be illustrative only and not limiting, following a phone call from a son or daughter or through an intelligent agent, the user may stand up, stumble, and then fall. Sensors in the eyewear device, such as but not limited to, a multi-axis inertial measurement unit with an embedded accelerometer, will record the signal generated by the fall. This signal can be used to alert appropriate personnel that the user has fallen and needs help. For example, if the user lives in an assisted living facility or nursing home where staff are on call, the staff will receive appropriate notification of the fall. If the user lives alone, a 911 emergency service call can be made. Depending on the complexity of the system, the user's scenario can be entered as a setup parameter, determined by sensor data, or remain unknown, requiring the system to perform a default response to the sensor data. This is a non-limiting example of providing help related to the status of security 2108.
[0109] The system can evaluate the user's cognitive state by one or more techniques, such as, but not limited to, comparing the error rate of the speech recognition system when processing the user's speech with a historical error rate metric for a given user. If the error rate is abnormally high, it may indicate that the user has encountered a bad health event such as a stroke. A stroke can have a negative impact on the user's speech, and in the worst case, the speech recognition system cannot recognize the speech. In addition, the system can test the user's cognitive state by asking the user several questions. Based on the known answers to the questions, the system can evaluate the user's cognitive state. This is a non-limiting example of providing help related to the cognitive state at 2110.
[0110] Analyzing the user's posture through sensor data recorded as a function of time allows analysis of the user's posture, biomechanics, activity level, etc. This analysis allows the system to compare the user's performance with that of healthy users in order to analyze the user's physical state. Based on such analysis, the system can generate corrective assistance and communicate it to the user. Optionally, if the user's life path is consistent with the path of a healthy state, a form of positive assistance can be generated and communicated to the user, such as but not limited to: You had a great day today, breaking up the sitting state regularly with 15,000 steps, keep working hard! "This is a non-limiting example of providing assistance related to the state of the body at 2112.
[0111] Analysis of the user's emotional state can be quantified through the output of one or more biometric sensors, such as but not limited to; heart rate, breathing rate, galvanic skin response, etc. Galvanic skin response measures changes in the user's emotional state through changes in skin conductivity. During the assessment of the user's emotional state, these data can be transmitted to and analyzed by health care workers. Such assessments can be part of sports training, medical procedures such as dental surgery, etc. In one or more embodiments, the frequency of emotional heightening may be a sign of mental instability, and medical intervention is required to prevent negative outcomes such as suicide in the worst case. Medical service providers can use data about the user's emotional state during, for example, dental surgery to make the user feel calmer and more relaxed. The galvanic skin response sensor can be located at a location where the eyewear device contacts the user's skin, such as but not limited to the bridge of the nose. These are some non-limiting examples of providing assistance related to the emotional state at 2114.
[0112] Fig. 22 According to an embodiment of the present invention, a process for providing assistance to a user of an eyewear device in combination with a user scenario and status is generally shown at 2200. Fig. 22 , the process 2200 includes determining a scenario and a user state analysis for a user before providing assistance to the user. The process starts at block 2202. At block 2204, data is received from one or more sensors associated with the eyewear device. The eyewear device can be any of the eyewear devices described above, such as a modular reconfigurable eyewear device using one or more temple insertion modules (TIMs), or a modular eyewear device equipped with a back neck module assembly. In addition, the eyewear device does not need to be a modular device, and the eyewear device can be an eyewear device without a modular design.
[0113] The one or more sensors in block 2204 may be, for example but not limited to, any of the above sensors or a combination of sensors, including a camera, a microphone, a global positioning system (GPS) receiver, a multi-axis inertial measurement unit including an accelerometer, a gyroscope, a magnetometer, a proximity sensor, a motion sensor, a gyroscope, a magnetometer, a display, a temperature sensor, a pressure sensor, an infrared sensor, a biometric sensor, and a biomechanical sensor, wherein the biomechanical sensor is, for example but not limited to, a skin electrode activity (EDA) sensor, and the skin electrode activity (EDA) sensor includes but is not limited to a skin electrode response (GSR) sensor, a heart rate sensor, a breathing rate sensor, etc. Therefore, in various embodiments, the roll, pitch, orientation, position, and speed of the user wearing the eyewear device may be obtained through the data output by the sensors in block 2204.
[0114] In block 2206, the user's scene is determined. As used in the description of this embodiment, the user's scene includes one or more of body posture, activity, and the user's natural environment. The user's scene may change over time and may be a dynamic attribute or a static attribute. Fig.18 The user's scenario is described in more detail. The scenario is determined using the data received in block 2204.
[0115] In block 2208, the user's status is determined. The user's status may change over time and may be a dynamic attribute or a static attribute. Fig. 20 and Fig.21 The state of the user is described in more detail. The state is determined using the data received in box 2204. In box 2210, help is provided to the user in the form of feedback through the eyeglass device. The feedback can be one or more of audio feedback emitted by a speaker embedded in the eyeglass device, vibration transmitted to the user by the eyeglass device, and feedback from a device located in the user's natural environment transmitted by the eyeglass device. In boxes 2206 and 2208, data from one or more sensors are analyzed, for example, tracking the user's head posture and / or user movement or no user movement. Based on the data analysis, in box 2210, help is provided to the user by, for example, communication with the user. The help can be in the form of an audio message broadcast from a speaker embedded in the eyeglass device to remind the user of the negative state the user is in based on the analysis of the state and the data from the sensor. A negative state is a state that may cause some harm to the user immediately or at some time in the future, such as, but not limited to, lack of sufficient movement frequency or a long period of time when the head remains stationary. In this example, the harm is damage to the user's biomechanical function. Help can also be help to confirm a positive state, such as confirming that there is enough head movement or enough body activity. The help may be broadcast to the user via one or more speakers associated with the eyewear device. The control of the algorithm implementing the process of 2200 may be continuously looped as shown in 2212 to continuously receive data at block 2204, determine the scene at block 2206, analyze the user's state at block 2208, and provide help as needed at block 2210. Thus, real-time help is provided to the user based on the scene and state of the user changing over time.
[0116] The process ends at box 2214. The trigger for ending the process at box 2214 may be when the user removes the eyeglass device from the user's head. Such an event may be determined by the system through an output from a proximity sensor indicating that the user's head is no longer close to the eyeglass device. In another non-limiting example, the process ends at box 2214 when the user moves the temples of the eyeglass device to a closed position, thereby shutting down the system by a switch triggered by closing the temples. Alternatively, the system may be shut down by a voice command. These examples of initiating box 2214 are provided for illustration only and do not constitute limitations on embodiments of the present invention.
[0117] The process 2200 of providing help to the user is finally completed in the case of determining the user's scenario. Determining the user's scenario can increase the relevance of the help provided by the system to the user. Fig.19 In the examples given, the user may be in one or more different social environments (scenarios) that have not been determined by the system. Fig. 22 The process can determine the context. Ultimately, the analysis may conclude that the user is indeed inactive for a long time, for example: the user is in a passive state due to lack of exercise. The system may further conclude that the user's context justifies the inactivity and that assistance is not currently needed. In one non-limiting example, given only as an illustration, the system may determine that the user is doing an examination and that this is the reason for the inactivity. In this case, the system will not provide a combination of Fig.19 The system will continue to loop at 2212, acquiring and processing further data at 2204, and determining the scene at 2206 until the scene changes, such as the end of the examination. Then, if the user continues to be inactive, the system will provide the user with relevant assistance through the glasses to avoid compromising the user's biomechanical function due to further inactivity. Therefore, in various embodiments, before providing assistance to the user, the assistance provided by the system to the user is refined through scene determination and user state analysis.
[0118] Fig.23 According to an embodiment of the present invention, a process for providing different levels of assistance to a user of an eyewear device is generally shown at 2300. Fig.23 , the process 2300 includes determining a scenario and a user state analysis for a user before providing assistance to the user. The process starts at block 2302. At block 2304, data is received from one or more sensors associated with the eyewear device. The eyewear device can be any of the eyewear devices described above, such as a modular reconfigurable eyewear device using one or more temple insertion modules (TIMs), or a modular eyewear device equipped with a back neck module assembly. In addition, the eyewear device does not need to be a modular device, and the eyewear device can be an eyewear device without a modular design.
[0119] The one or more sensors in block 2304 may be, for example but not limited to, any of the above sensors or a combination of sensors, including a camera, a microphone, a global positioning system (GPS) receiver, a multi-axis inertial measurement unit including an accelerometer, a gyroscope, a magnetometer, a proximity sensor, a motion sensor, a gyroscope, a magnetometer, a display, a temperature sensor, a pressure sensor, an infrared sensor, a biometric sensor, and a biomechanical sensor, wherein the biomechanical sensor is, for example but not limited to, an electrodermal activity (EDA) sensor, and the electrodermal activity (EDA) sensor includes but is not limited to a electrodermal response (GSR) sensor, a heart rate sensor, a breathing rate sensor, etc. Therefore, in various embodiments, the roll, pitch, orientation, position, and speed of the user wearing the eyewear device may be obtained through the data output by the sensors in block 2304.
[0120] In block 2306, the user's scene is determined. As used in the description of this embodiment, the user's scene includes body posture, activities, and the natural environment in which the user is located. The user's scene can change over time and can be a dynamic attribute or a static attribute. Fig.18 The user's scenario is described in more detail. The scenario is determined using the data received in block 2304.
[0121] In block 2308, the user's status is analyzed. The user's status may change over time and may be a dynamic attribute or a static attribute. Fig. 20 and Fig.21 The state of the user is described in more detail. The state is analyzed using the data received in block 2304. In block 2310, the algorithm determines whether the scene / state combination requires assistance. If the answer is no, control is transferred at 2312, and additional data is collected at 2304, and the scene and state blocks are executed again until the scene and state analysis at 2314, together with the current data, justifies providing assistance to the user. Control then passes to block 2316, and the algorithm determines the level of assistance to provide to the user.
[0122] The algorithm determines the risk level associated with the user's scenario, the user's state, and the current data at 2318 to determine the corresponding help provided to the user. The determination is done for a combination of a predetermined range of variables associated with the user, and in some embodiments, artificial intelligence methods including machine learning techniques are applied to the user and the data to refine and update the personalized analysis framework for a single user. Based on the analysis, the level of help provided to the user is determined at box 2320. The help can be in the form of an audio message broadcast from a speaker embedded in the eyewear device to remind the user that he is in a negative state based on the analysis of the scenario, the state, and the data from the sensor, and / or provided in digital form through a compatible application on a smart mobile device or computing platform. In addition, the help can be in the form of human help, contacting the user by phone, or by the help of a person who actually comes to the user in person according to the current form of the scenario, state, and data. For example, a scenario is a user sitting in a wheelchair, the activity is eating, the natural environment is an indoor scene, and the speed is 15 miles / per hour (MPH) The user shows a high risk and the wheelchair is out of control, which proves that the response considering all the circumstances is correct. For example, audio help for eyewear devices and instant personal access.
[0123] The control of the algorithm implementing the process of 2300 can be continuously looped as shown in 2322 to receive data at box 2304, determine the scene at box 2306, analyze the user's state at box 2308, and provide help as needed at box 2310 / 2316. Real-time help is thus provided to the user based on the user's scene and state that changes over time. Alternatively, a condition to end the process may occur at 2324.
[0124] The process ends at box 2324. The trigger for ending the process at box 2324 may be when the user removes the eyeglass device from the user's head. Such an event may be determined by the system through an output from a proximity sensor indicating that the user's head is no longer close to the eyeglass device. In another non-limiting example, when the user moves the temples of the eyeglass device to a closed position, the process ends at box 2324, thereby shutting down the system by a switch triggered by closing the temples. Alternatively, the system may be shut down by a voice command. These examples of initiating box 2324 are provided for illustration only and do not constitute limitations on embodiments of the present invention.
[0125] Fig.24 A system for providing assistance to a user of an eyewear device is generally shown at 2400 according to an embodiment of the present invention. Fig.24, the eyeglass device is represented by 2402. The eyeglass device 2402 is configured with embedded sensors 2404, 2406 to 2408. An optional local sensor 2412 is shown, which can be part of the sensor that communicates with the eyeglass device 2402. In one or more embodiments, sensor 2404 is a camera pointing to the left direction relative to the user, sensor 2406 is a camera pointing in front of the user, and sensor 2408 is a camera pointing to the right of the user. The output from one or more cameras, alone or together, is used to determine the scene for the user through image analysis. Image analysis, together with other sensor data such as microphone data, can be used by the system to determine the scene for the user, such as but not limited to a football game, a bar, a library, etc. Remote hardware and services are shown by 2414, which can employ one or more personnel 2416 to provide assistance to the user of the glasses 2402. Assistance to the user is represented as audio feedback 2410, which is broadcast to the user through one or more speakers embedded in the glasses 2402.
[0126] The one or more persons 2416 may be family members, professionals such as doctors, nurses, nursing home personnel, etc. Assistance is provided to the user in response to an output of the system from analysis of sensor data with one or more of a user context, a user state, or a combination of sensor data with a user context and a user state as described above in conjunction with the previous figures.
[0127] As described herein, in various embodiments, an eyewear device that provides various types of assistance to a user is described. By using one or more sensors embedded in the eyewear device or sensors (embedded as well as external sensors) used in a sensor system of the eyewear device, the user's scene and / or state is determined, and relevant assistance is generated and provided to the user to correct negative states or enhance positive states. In several non-limiting examples, for illustration only and not intended to be limiting, the eyewear device tracks the posture and movement of the user's head and then provides reminders (help) to the user in the following situations: (1) The user is playing an electronic game and his or her head points downward while playing the game, which can cause an incorrect posture that may injure the neck. The system provides reminders (help) of potential hazards and makes suggestions to mitigate the negative impact of the activity on health. (2) The user's head is tilted to one side and not centered. The user may not even be aware of this misalignment. The system provides reminders (help) of potential hazards and makes suggestions aimed at mitigating the negative impact of the activity on health. (3) Incorrect posture is detected over a period of time. The system provides reminders (help) to the user and recommends regular stretching to mitigate the negative impact of the activity on health. (4) Reminders (help) can be generated in a preventive manner to avoid negative outcomes in the future. For example, when the user is engaged in an activity that lasts for a long time. To avoid fatigue, in one or more embodiments, the system provides help to the user, such as calling the user. For example, in a non-limiting example for illustration only, the system can generate an audio communication broadcast in an embedded speaker that conveys: "You've been driving for 7 hours. I think you might need a cup of coffee. There is an exit in 10 miles. It might be a good idea to take a break at that exit." (5) Monitor the movement and position of the user's head during driving. Maintaining certain head angles for a long time may indicate an unsafe state, such as the user falling asleep. In various embodiments, the system provides help to "wake up the user." This help can be in the form of an audio broadcast through an embedded speaker or in the form of vibration of the glasses through a built-in transducer that provides vibration. The user's eyes are illuminated by a display combined with the glasses. In some embodiments, the glasses communicate with one or more vehicle systems (such as a horn) configured as external sensors or external devices. In this case, the algorithm uses the vehicle's horn to wake the user to avoid negative states such as traffic accidents. Thus, in various embodiments, the reminder (help) can be an audio reminder broadcast to the user through an embedded speaker and / or a reminder through a display such as the heads-up display described above, or a reminder provided by an external sensor in communication with the eyewear system.
[0128] Therefore, the process described in this article describes the user through multiple dimensions represented by the user's data, scenes and states as a function of time. Within a given set of dimensions, there are positive states or safe areas. Similarly, within the possible range of dimensions, there are negative states and potential unsafe areas. In some embodiments, safe areas and unsafe areas are known a priori. In other embodiments, safe areas and unsafe areas are learned over time and are not known a priori. And in some embodiments, safe areas and unsafe areas are a combination of information known a priori and information learned over time. Safe areas and unsafe areas are used during data analysis to formulate relevant help for users based on the collected data and one or more of the user scenarios and user states. These processes are flexible and can be used for any scenario and / or state, as well as any data related to the user.
[0129] For the purpose of discussing and understanding different embodiments, it should be understood that various terms are used by those skilled in the art to describe technology and methods. In addition, in the process of description, for the purpose of explanation, many specific details are set forth, so as to provide a thorough understanding of the embodiments. However, for those of ordinary skill in the art, it is obvious that the embodiments can be implemented without these specific details. In some instances, known structures and devices are shown in block diagram form, rather than being shown in detail, to avoid confusing various embodiments. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention, and it should be understood that other embodiments can be utilized and can be changed logically, mechanically, electrically and other without departing from the scope of the present invention.
[0130] Some parts of the description can be represented by algorithms and symbolic representations of operations on data bits in, for example, computer memories. These algorithmic descriptions and representations are the means by which those of ordinary skill in the field of data processing most effectively convey the essence of their work to other persons of ordinary skill in the art. Algorithms herein generally refer to self-consistent sequences of behaviors that result in desired results. These behaviors require physical operations on physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. It has been shown that it is often convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc., mainly for general reasons.
[0131] It should be borne in mind, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, it will be apparent from the discussion that throughout the description, discussions using terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" can refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in computer system registers and memories and converts it to other similar data represented as physical (electronic) quantities in computer system memories or registers or other such information storage, transmission or display devices.
[0132] The invention may be implemented by a device for performing the operations described herein. This device may be specifically constructed for the desired purpose, or it may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. The computer program may be stored on a computer-readable storage medium, such as, but not limited to, any type of disk, including floppy disks, hard disks, optical disks, optical disk read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), dynamic random access memories (DRAM), electrically programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), flash memory, magnetic or optical cards, RAID, etc., or any type of medium suitable for storing local or remote electronic instructions for a computer.
[0133] The algorithm proposed herein is not inherently related to any particular computer or other device with display. According to the description herein, various general-purpose systems can be used together with programs, or it is proved to be convenient to construct more specialized equipment to perform the required method. For example, any method according to the embodiment can be realized by the hard-wired circuit obtained by programming a general-purpose processor, or can be realized by any combination of hardware and software. It will be appreciated by those skilled in the art that the embodiment can be realized by other computer system configurations outside the description, and the computer system configuration includes: handheld device, multiprocessor system, microprocessor-based or programmable consumer electronic device, digital signal processing (DSP) device, set-top box, network personal computer, minicomputer, mainframe computer, etc. These embodiments can also be implemented in a distributed computing environment, where the task is performed by a remote processing device linked by a communication network.
[0134] Methods herein can be implemented using computer software. If written in a programming language that meets recognized standards, then a sequence of instructions designed to implement these methods can be compiled to execute on various hardware platforms and interface with various operating systems. In addition, the embodiments are described without reference to any specific programming language. It should be understood that various programming languages can be used to implement the embodiments described herein. In addition, software in one form or another form (e.g., program, flow, application, driver ...) is often described in the art as taking a certain action or causing a certain result. These expressions are merely a brief statement that the computer executes software so that the processor of the computer performs an action or produces a result.
[0135] It should be understood that those skilled in the art use various terms and techniques to describe communications, protocols, applications, implementations, mechanisms, etc. A similar technique is to describe the implementation of a technique using an algorithm or mathematical expression. That is, although the technique can be implemented, for example, as a code on an executing computer, the expression of the technique can be more appropriately and concisely conveyed or communicated as a formula, algorithm, or mathematical expression. Therefore, one of ordinary skill in the art will recognize that A+B=C is represented as a block of an addition function, and its implementation in hardware and / or software will take two inputs (A and B) and produce a summed output (C). Therefore, the use of a formula, algorithm, or mathematical expression as a description should be understood to have a physical representation at least in hardware and / or software (e.g., a computer system, in which the technique of the present invention can be implemented and implemented as an embodiment).
[0136] A non-transitory machine-readable medium is understood to include any mechanism for storing information (e.g., program code, etc.) in a form readable by a machine (e.g., a computer). For example, machine-readable media, synonymously referred to as computer-readable media, include read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustic or other forms of information transmission other than by propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.); and the like.
[0137] As used in this specification, "one embodiment" or "an embodiment" or similar phrases mean that the described features are included in at least one embodiment of the present invention. The "one embodiment" mentioned in this description does not necessarily refer to the same embodiment, but these embodiments are not mutually exclusive. "One embodiment" also does not mean that the present invention has only one embodiment. For example, features, structures, actions, etc. described in "one embodiment" may also be included in other embodiments. Therefore, the present invention may include various combinations and / or integrations of the embodiments described herein.
[0138] Although the present invention has been described in terms of several embodiments, those skilled in the art will recognize that the present invention is not limited to the described embodiments, but can be implemented by modification and alteration within the spirit and scope of the appended claims. Therefore, this description should be regarded as illustrative rather than restrictive.
Claims
1. A method for providing assistance to a user through an eyewear device, characterized in that: The eyeglass device comprises a modular brow bar, a frame, a left temple, and a right temple. The modular brow bar is fixed to the frame, and the modular brow bar provides a channel along its length direction, the channel is wired with an electrical path, and the modular brow bar has integrated contact points on both sides, and the integrated contact points allow data or power transmission when in contact with the integrated contact points of the left and right temples; the right temple is fixed to the frame through a hinge connector, and when the hinge connector senses that the eyeglass device is opened, the hinge connector allows power or data to be transmitted from the right temple to the left temple, and when the hinge connector senses that the eyeglass device is closed, the hinge connector combines signals received from one or more proximity sensors and motion sensors to allow the system to sense the user-device interaction state and shut down power or data transmission; The method comprises: receiving first data from a first sensor, wherein the first sensor is integrated in the eyewear device, and the first sensor measures a parameter related to the user; analyzing the user's status, wherein the analysis is performed using the first data; determining a scenario of the user, wherein the determining uses the first data; Determine whether assistance is needed based on the described scenario and status; If assistance is required, determining the risk level associated with the user's scenario, the user's state, and the current data to determine the level of assistance to provide to the user; Assistance is provided to the user, wherein the assistance is in the form of feedback, and the feedback is related to the status.
2. The method according to claim 1, characterized in that A negative state is associated with the state.
3. The method according to claim 2, characterized in that The feedback is configured to help the user correct the negative state.
4. The method according to claim 3, characterized in that The feedback is an audio signal transmitted from a speaker in a temple of the eyeglass apparatus.
5. The method according to claim 3, characterized in that The feedback is generated from outside the eyewear apparatus.
6. The method according to claim 5, characterized in that The feedback comes from humans.
7. The method according to claim 1, characterized in that A positive state is associated with the state.
8. The method according to claim 7, characterized in that The feedback is designed to assist the user by promoting the positive state.
9. The method according to claim 8, characterized in that The feedback is an audio signal transmitted from a speaker in a temple of the eyeglass apparatus.
10. The method according to claim 1, characterized in that The method further comprises: A context of the user is determined, wherein the determining uses the first data in the determining process, and the feedback is related to the state and the context.
11. The method according to claim 10, characterized in that A negative state is associated with the state.
12. The method according to claim 11, characterized in that The feedback is configured to help the user correct the negative state.
13. The method according to claim 12, characterized in that The feedback is an audio signal transmitted from a speaker in a temple of the eyeglass apparatus.
14. The method according to claim 12, characterized in that The feedback is generated from outside the eyewear apparatus.
15. The method according to claim 14, characterized in that The feedback comes from humans.
16. The method according to claim 10, characterized in that A positive state is associated with the state.
17. The method according to claim 16, characterized in that The feedback is designed to assist the user by promoting the positive state.
18. The method according to claim 10, characterized in that The method further comprises: Second data is received from a second sensor, the second sensor being an external sensor, and the analysis is performed using the first data and the second data.
19. A method for providing assistance to a user through an eyewear device, characterized in that: The eyeglass device comprises a modular brow bar, a frame, a left temple, and a right temple. The modular brow bar is fixed to the frame, and the modular brow bar provides a channel along its length direction, the channel is wired with an electrical path, and the modular brow bar has integrated contact points on both sides, and the integrated contact points allow data or power transmission when in contact with the integrated contact points of the left and right temples; the right temple is fixed to the frame through a hinge connector, and when the hinge connector senses that the eyeglass device is opened, the hinge connector allows power or data to be transmitted from the right temple to the left temple, and when the hinge connector senses that the eyeglass device is closed, the hinge connector combines signals received from one or more proximity sensors and motion sensors to allow the system to sense the user-device interaction state and shut down power or data transmission; The method comprises: receiving first data from a first sensor, wherein the first sensor is integrated in the eyewear device, and the first sensor measures a parameter related to the user; determining a scenario of the user, wherein the determining uses the first data in the determining process; If in the identified scenario the user is in a passive state due to lack of movement, but the identified scenario justifies the user's inactivity, no assistance is provided; When the user's context changes, if the user continues to be inactive, assistance is provided to the user.
20. The method of claim 19, wherein: The first element of the scene is the body posture of the user.
21. The method of claim 20, wherein: The second element of the scenario is the activity being performed by the user.
22. The method of claim 20, wherein: The third element of the scene is the user's natural environment.
23. The method of claim 19, wherein: The assistance is provided to the user in the form of an audio communication.
24. The method of claim 19, wherein: The method further comprises: Second data is received from a second sensor, the second sensor being an external sensor, and the determining uses the first data and the second data.
25. A computer-readable storage medium having program codes stored thereon, the program codes being used to cause a data processing system to perform the following steps, the steps comprising: receiving first data from a first sensor, wherein the first sensor is integrated in the eyewear device, and the first sensor measures a parameter related to the user; The eyeglass device comprises a modular brow bar, a frame, a left temple, and a right temple. The modular brow bar is fixed to the frame, and the modular brow bar provides a channel along its length direction, the channel is wired with an electrical path, and the modular brow bar has integrated contact points on both sides, and the integrated contact points allow data or power transmission when in contact with the integrated contact points of the left and right temples; the right temple is fixed to the frame through a hinge connector, and when the hinge connector senses that the eyeglass device is opened, the hinge connector allows power or data to be transmitted from the right temple to the left temple, and when the hinge connector senses that the eyeglass device is closed, the hinge connector combines signals received from one or more proximity sensors and motion sensors to allow the system to sense the user-device interaction state and shut down power or data transmission; analyzing the user's status, wherein the analysis is performed using the first data; determining a scenario of the user, wherein the determining uses the first data; Determine whether assistance is needed based on the described scenario and status; If assistance is required, determining the risk level associated with the user's scenario, the user's state, and the current data to determine the level of assistance to provide to the user; Provide help to the user, where the help is related to the scenario.
26. The computer-readable storage medium of claim 25, wherein: The first element of the scene is the body posture of the user.
27. The computer-readable storage medium of claim 26, wherein: The second element of the scenario is the activity being performed by the user.
28. The computer-readable storage medium of claim 26, wherein: The third element of the scene is the user's natural environment.
29. The computer-readable storage medium of claim 25, wherein: The assistance is provided to the user in the form of an audio communication.
30. The computer-readable storage medium of claim 25, wherein: The steps also include: The user's state is analyzed, wherein the analysis uses the first data, and the assistance is related to the state and the scenario.
31. The computer-readable storage medium of claim 30, wherein: The steps also include: Second data is received from a second sensor, the second sensor being an external sensor, and the determining uses the first data and the second data.
32. A system executed by a data processing system to provide assistance to a user through an eyewear device, characterized in that: The eyeglass device comprises a modular brow bar, a frame, a left temple, and a right temple. The modular brow bar is fixed to the frame, and the modular brow bar provides a channel along its length direction, the channel is wired with an electrical path, and the modular brow bar has integrated contact points on both sides, and the integrated contact points allow data or power transmission when in contact with the integrated contact points of the left and right temples; the right temple is fixed to the frame through a hinge connector, and when the hinge connector senses that the eyeglass device is opened, the hinge connector allows power or data to be transmitted from the right temple to the left temple, and when the hinge connector senses that the eyeglass device is closed, the hinge connector combines signals received from one or more proximity sensors and motion sensors to allow the system to sense the user-device interaction state and shut down power or data transmission; The system comprises: A voice interface, the voice interface having a microphone and a speaker, the voice interface being used to receive user commands and provide assistance to the user; a sensor data input for receiving sensor data; a processor in electrical communication with the voice interface and the sensory data; A computer program for processing the sensor data, the computer program and the processor causing the data processing system to perform the following steps, the steps comprising: receiving first data from a first sensor, wherein the first sensor is integrated in the eyewear device, and the first sensor measures a parameter related to the user; determining a scenario of the user, wherein the determining uses the first data; analyzing the user's status, wherein the analysis uses the first data; Determine whether assistance is needed based on the described scenario and status; If assistance is required, determining the risk level associated with the user's scenario, the user's state, and the current data to determine the level of assistance to provide to the user; Help is provided to the user, the help being relevant to the scenario and the status.
33. The system of claim 32, wherein: The analysis considered categories selected from the group consisting of health, well-being, safety, cognitive, physical, and emotional.
34. The system of claim 33, wherein: The assistance is designed to help the user correct a negative state.
35. The system of claim 33, wherein: The assistance is designed to assist the user by promoting a positive state.
36. The system of claim 35, wherein: The system further comprises: A device is located outside the eyeglass device, the device is in electrical communication with the eyeglass device, and the device is used to provide the assistance to the user.
37. The system of claim 32, wherein: The steps also include: Second data is received from a second sensor, the second sensor being an external sensor, the determining uses the first data and the second data, and the analyzing uses the first data and the second data.
38. The system of claim 32, wherein: The steps also include: Third data is received from a plurality of embedded sensors, the determining uses the first data and the third data, and the analyzing uses the first data and the third data.
