Systems, methods, and techniques for controlling powered devices

Abstract

Methods, systems, and techniques for controlling powered wheelchairs, beds, vehicles, and similar devices using augmented reality technology are provided. Example embodiments provide an Eye Gaze Enabled Augmented Reality Control System, which enables people living with motor neuron disabilities like ALS, Muscular Dystrophy, or Spinal Cord Injury to use the full capabilities of their personal computing devices in an integrated fashion with eye gaze technology without a need to use a separate dedicated medical device for controlling their powered wheelchair and adjusting their seating and bed positions. Example embodiments comprise a bridge device, a power device, an eye gaze enabled augmented reality device and a powered device which cooperate to provide an intermediate communications bridge between the consumer electronics side of the system and the safety critical control side of the system.

Description

SYSTEMS, METHODS, AND TECHNIQUES FOR CONTROLLINGPOWERED DEVICESCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 525,766, entitled “Ability Drive Methods and Systems for Apple Vision Pro,” filed July 10, 2023, which is incorporated herein by reference, in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates to methods, techniques, and systems for controlling powered wheelchairs and beds and, in particular, to methods, techniques, and systems for controlling powered wheelchairs, beds, vehicles, and similar devices using augmented reality technology.BACKGROUND

[0003] People living with motor neuron disabilities like ALS, Muscular Dystrophy, or Spinal Cord Injury can independently drive their wheelchairs with eye gaze. ABILITY DRIVE® (from Tolt Technologies LLC) for WINDOWS® (from Microsoft Corp.)- based eye gaze devices has proved the need, demand, and market for this capability since 2017.

[0004] However, ABILITY DRIVE for WINDOWS-based eye gaze devices serves a limited populace because it is limited to such WINDOWS-based devices only, it can be cost prohibitive for the self-pay market, requiring an arduous medical insurance pathway, and eye gaze technology available today struggles in sunlight, making driving with one’s eyes an unreliable experience outdoors.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is an example block diagram of example components of an example Eye Gaze Enabled Augmented Reality Control System.

[0006] Figure 2 is an example flow diagram of example safety watchdog logic implemented by an example Eye Gaze Enabled Augmented Reality Control System.

[0007] Figure 3 is an example block diagram of example microprocessors used to implement the safety watchdog, communications bridge, and security logic in example Eye Gaze Enabled Augmented Reality Control Systems.

[0008] Figure 4 is an example head mounted eye gaze enabled augmented reality device for use with example Eye Gaze Enabled Augmented Reality Control Systems.

[0009] Figures 5A-5C and 6 illustrates examples of a set of intrinsic components that can be integrated into a head mounted eye gaze enabled augmented reality device as part of an example Eye Gaze Enabled Augmented Reality Control System.

[0010] Figure 7 is an example diagram of an example eye gaze enabled augmented reality device that can be used with example Eye Gaze Enabled Augmented Reality Control Systems.

[0011] Figure 8 is an example screen display of an example driving application for an eye gaze enabled augmented reality device integrated into an example Eye Gaze Enabled Augmented Reality Control System.

[0012] Figure 9 is an example screen display of an example driving application for a head mounted eye gaze enabled augmented reality device integrated into an example Eye Gaze Enabled Augmented Reality Control System.

[0013] Figure 10 is an example screen display of a user interface for selection of an axis for adjustment in an example seating or bed position application that can be integrated into an example Eye Gaze Enabled Augmented Reality Control System for use with an eye gaze enabled augmented reality device.

[0014] Figure 11 is an example screen display of a user interface for adjusting an axis of movement in an example seating or bed position application that can beintegrated into an example Eye Gaze Enabled Augmented Reality Control System for use with an eye gaze enabled augmented reality device.DETAILED DESCRIPTION

[0015] Example embodiments provide an Eye Gaze Enabled Augmented Reality Control System (“EGEARCS”), which enables people living with motor neuron disabilities like ALS, Muscular Dystrophy, or Spinal Cord Injury to use the full capabilities of their personal computing devices in an integrated fashion with eye gaze technology without a need to use a separate dedicated medical device for controlling their powered wheelchair and / or adjusting their seating and bed positions.

[0016] People living with disabilities rely heavily on their personal computing device to interact with their world, whether this is a phone, tablet, laptop computer, or head mounted eye gaze enabled augmented reality device (“EARD”) like the Apple Vision Pro or Meta Quest. These personal devices allow them to engage with their world electronically, keeping in touch with friends, family members, caregivers, and doctors, provide speech assistance, control their environment and home through home automation, and access games and entertainment like TV shows and movies.

[0017] Rather than use a separate dedicated medical device for controlling their powered wheelchair and adjusting their seating and / or bed, such users would like to have a unified, integrated experience between their personal computing device and their medical equipment. However, this introduces a potential conflict, mixing consumer electronics which are not designed to be a medical device with safety critical control systems like driving a powered wheelchair, a vehicle, or adjusting a hospital bed. Consumer electronics are not required to be tested to the same standard and adhere to the same safety regulations as registered medical devices, and the manufacturers of consumer electronics shy away from the potential liability situations that occur when consumer electronics are used in medical device or safety critical contexts. For example, Apple’s “Important Safety Information for AppleVision Pro” (https: / / support.apple.com / guide / apple-vision-pro / important-safety- information-c0c84db82a44 / visionos) document states, “Apple Vision Pro is not a medical device.” This poses severe hurdles to designing a system that meets the user’s need of having one unified solution that includes their personal computing device and interfaces with both consumer and medical needs.

[0018] Example Eye Gaze Enabled Augmented Reality Control Systems support a notion of a safety watchdog coprocessor to overcome these hurdles. The safety watchdog is a dedicated, separate processor that is part of the overall EGEARCS which acts as an intermediate communications bridge between the consumer electronics side of the system and the safety critical control side of the system. Its function is to receive communications from the personal computing device, which is assumed to have some level of unreliability in its operations, and translate those communications into commands destined for the safety critical side of the system. The safety watchdog also continuously monitors for continuous “keep alive” signals emanating from the personal computing device, and, if these signals become disrupted for any reason, the safety watchdog sends the appropriate control signals to bring the safety critical components to a safe, neutral position. Examples of such interrupted signal include, for example, systems failure or low battery power condition or an unexpected restart or an application crash or a communications disruption. The safety watchdog has very specific and limited functionality so that it can meet the safety critical design guidelines and medical quality regulations that the more complex personal computing device cannot.

[0019] The safety watchdog coprocessor works in conjunction in an example EGEARCS with other dedicated, single function coprocessors. These include but are not limited to a Bluetooth communications processor that handles the inherent complexities of the Bluetooth Low Energy radio frequency communications stack; a dedicated Identity / Security chip which stores and calculates cryptographic authentication and identification data; and / or a communications sub processor thatprovides a bridge to another proprietary protocol such as the CANBus communications protocol on a powered wheelchair or vehicle.

[0020] To manage quality, safety, and liability, some manufacturers require devices that connect to their ecosystem of products to be licensed, follow their good manufacturing processes, meet applicable government regulations, and pass an internal design & quality review. For example, Apple Inc. (“Apple”) provides the “Made for iPhone (MFi)” program for other companies that wish to make devices that connect to and work with their products. As part of this MFi program, Apple provides security / identity microprocessors, referred to as iAP2 processors, that must be purchased from Apple and included in approved devices. These iAP2 processors include embedded Apple cryptographic secrets so that only Apple can source these microprocessors. Further information on the iAP2 protocol is found at “misc - Exploring Apple's MFI protocol iAP2 (wiomoc.de).” Outside of the Apple ecosystem, it is a common practice to include a cryptographic identity chip in hardware devices to ensure that the hardware being used is authentic and not a counterfeit device, which is an important step for managing quality and reliability in a system. For example, see Microchip’s line of Security integrated circuits (see “https: / / www.microchip.com / en-us / products / security / security-ics”). Using device authentication to improve system reliability is a non-binding recommendation of the FDA for “Radio Frequency Wireless Technology in Medical Devices” (see “https: / / www.fda.gov / media / 71975 / download”).

[0021] Figure 1 is an example block diagram of example components of an example Eye Gaze Enabled Augmented Reality Control System. The logical design of an example EGEARCS 1000 comprises: bridge device 1010, a power device 1020, an eye gaze enabled augmented reality device (EARD) 1030, and a powered wheelchair (bed, vehicle, or other powered device) 1040.

[0022] An eye gaze enabled augmented reality device (“EARD”) such as that illustrated in Figure 1 is a device which combines the technologies of: a display, eye gaze sensors for tracking the eve focal point on the display surface, external facingcamera(s) which are used to create a pass-through view of the external world on the display, a display compositor which combines camera views and other video signals into a unified view on the display, and a personal computing device which may be external to the display housing or integral to the display housing. This is described further below with respect to Figure 7.

[0023] One such example eye gaze enabled augmented reality device that can be used in an example Eye Gaze Enabled Augmented Reality Control System is a head mounted eye gaze enabled AR device (or “HM-EARD”) such as the APPLE VISION PRO device described below with respect to Figure 4. A head mounted eye gaze enabled augmented reality device (HM-EARD) is a device which combines the technologies of a head mount, an internal display, eye gaze sensors for tracking eye focal point in 3D space, a light shield for blocking external light sources, external facing camera(s) which are used to create a pass-through view of the external world on the internal display, a display compositor which combines camera views and other video signals into a unified view on the internal, and a personal computing device which may be external to the head mount or integral to the head mount.

[0024] Referring back to Figure 1 , the bridge device 1010 connects to the EARD 1030 over a communications protocol, in this example shown as Bluetooth Low Energy (BLE) 1050. The bridge device 1010 also connects to the powered wheelchair 1040 over a communications protocol, in this example shown as CANBUS 1060. The overarching goal of the bridge device 1010 is to be the intermediary, or bridge, between the EARD 1030 and the powered wheelchair 1040 so that the EARD’s driving application (“app”) 1080 can issue control commands to the drive system and seating actuators of the powered wheelchair (not shown). (Other powered devices such as powered beds and powered vehicles can be similarly configured and controlled.) The bridge device 1010 comprises subcomponents including: a communications bridge 1110 which translates commands between incoming and outgoing communications buses, a proprietary wheelchair control protocol based on CANBUS 1120, an authentication chip 1130to help verify identity and authorization when connecting to the EARD 1030, and a safety watchdog (coprocessor) 1140 which provides safety critical control logic in case of some form of failure occurring on the EARD 1030 or the communications protocol 1050 between the EARD 1030 and the bridge device 1010.

[0025] An authentication chip, such as authentication chip 1130, describes a single purpose custom integrated circuit that is designed to hold secrets like cryptographic private keys securely in such a way that the private key is nearly impossible to extract from the hardware. Its goal is to prevent hackers from reverse engineering or compromising the security system on a hardware device so that they can replicate or counterfeit the device. Typically, these specialty integrated circuits contain ‘one time write only’ storage for generating or storing secret keys and cryptographic algorithms like public-key cryptography and one-way functions which can use the stored secret to prove identity and authenticity without disclosing the secret key. For an overview of the concepts of public-key cryptography, see Wikipedia’s summary page on the topic (“https: / / en.wikipedia.org / wiki / Public-key_cryptography”).

[0026] The power device 1020 also connects the EARD 1030 to the powered wheelchair 1040. Its goal is to supply power from the powered wheelchair’s large batteries to the EARD 1030 to increase the number of hours of operation between recharges of the EARD 1030 and to reduce system complexity and caregiver burden by only requiring one set of batteries (the powered wheelchair 1040 batteries) to be charged each day. Typically speaking a powered wheelchair 1040 carries over 1 kWh of power in its batteries and an EARD’s power consumption is low enough that it will not significantly affect the number of hours of runtime of the powered wheelchair 1040 by increasing the power draw from the wheelchair 1040. The power device 1020 comprises subcomponents including: power monitoring 1210 to help prevent over depletion of the powered wheelchair 1040 batteries, a communications protocol 1220 to interact with the powered wheelchair battery system, an authentication chip 1230 which may or may not be physically the same as authentication chip 1130, and a power conversion system 1240 which translatesthe powered wheelchair’s battery DC power into USB power felivery (USB-PD) standards.

[0027] Example physical design 1500 gives another view of the components of an example EGEARCS 1000. In this example implementation, the components of both the bridge device 1010 and the power device 1020 are integrated together into a single system or physical device. This is one possible implementation; in other examples these logical systems could be physically separated in different configurations, embodiments, or products. The communications bridge 1110 may exist physically as a separate set of physical components connected on a chip die or via a bus on a circuit board. In this example implementation 1500, the BLE module is shown as a separate module which contains an on-chip antenna and separate logic core, while the bridge and watchdog logic share a logic core, and the CANBUS communications are handled on another logic core. This is one example implementation that uses separate cores for different sets of logic to reduce algorithmic complexity in alignment with safety critical design goals.

[0028] In operation, the bridge device 1010 contains a communications processor, communications bridge logic, and safety watchdog logic 1140 that relays and converts commands from the EARD driving app 1080 data stream to the proprietary Controller Area Network Bus (CANBUS) protocol 1060 that different powered wheelchair 1040 control systems implement and ensures safe operation of the wheelchair in the event of a disconnection, low power event, or unexpected app termination from the EARD 1030 or its driving app 1080.

[0029] The safety watchdog logic 1140 "keeps an eye on" the signal coming from the EARD 1030. It detects if / when the EARD 1030 is no longer sending data (e.g. it has powered down, the OS or application has crashed, or the communications connection from the tablet (or other consumer device has been lost) and brings the system to a safe halt. This logic is described in more detail in Figure 2.

[0030] The communications bridge 1110 takes a set of commands "e.g. drive forward" from the driving app 1080 sent over the communications bus 1050 andturns it into the appropriate CANBUS command to tell the powered wheelchair 1040 to "drive forward", etc. in its own language. It also communicates the status of the powered wheelchair back to the driving app 1080, e.g. "controlling seating, controlling driving, battery level", etc. Additional information on CANBUS can be found at “https: / / www.csselectronics.com / pages / can-bus-simple-intro-tutorial.”

[0031] The safety watchdog and the communications bridge run on a set of microcontrollers (MCUs). In one example implementation, the system consists of three dedicated function microcontrollers to be as reliable as possible. One microcontroller is dedicated to talking BLE, a complex protocol, and transforms the data into a simpler Universal Asynchronous Receiver / Transmitter (UART or serial bus) commands. Another microcontroller is dedicated to talking CANBUS and handling the wheelchair control CANBUS packets, and it also communicates in UART. In the middle is a "bridge" microcontroller that talks to both of the other microcontrollers via two UART ports. It also includes the safety watchdog logic to ensure everything is running as expected. It can independently (without any signal from the EARD 1030) send the “Stop” command to the CANBUS microcontroller to perform a safety stop.

[0032] The bridge device 1010 and power device 1020 transfer state knowledge of the powered wheelchair (driving vs. seating control, battery power levels, etc. and bridges control commands (move forward, turn right, lean seat back, honk horn, turn signals, lighting control) to the EARD. The authentication chip validates the authenticity of the bridge device and power device

[0033] As described in Figure 1 , the safety watchdog logic (typically implemented by a microcontroller) is one of the keys to a commercially viable Eye Gaze Enabled Augmented Reality Control System. Figure 2 is an example flow diagram of example safety watchdog logic implemented by an example Eye Gaze Enabled Augmented Reality Control System. Logic 2000 describes the steps by which the safety watchdog 1140 ensures that the powered wheelchair 1040 comes to a safe stop in the event of an unexpected failure or loss of communications.

[0034] When starting, in block 2010, the safety watchdog 1140 establishes communications with the EARD 1030. In block 2020, it leverages the functionality of the authentication chip 1130 to validate its authenticity and help establish a secure communications channel. Once the communications channel is established, a loop is performed in blocks 2030-2120. More specifically, in block 2030, the logic checks incoming communications for data from the EARD 1030. Note that in parallel to this incoming communications loop, there may be additional inbound and / or outbound communications between the EARD 1030 and powered wheelchair 1040 which are not detailed in this logical process 2000, such as powered wheelchair system state, battery level, etc.

[0035] In block 2030, checking incoming communications may operate using an asynchronous (e.g. callback) or synchronous methodology. In both cases, a system timer is part of the checking incoming communications logic so that if no communications are received in an established period, a decision is made in block 2040 to either process the received incoming data or to check for a keep alive timeout. If no incoming data is received in block 2040, the logic progresses to block 2090 to determine whether a wheelchair action in in progress. If so, the logic proceeds to determine whether an keep alive timeout has expired, and if so, executes a safety stop in block 2110. Otherwise (no action in progress or timeout not yet expired), the logic proceeds to block 2120 for any programmed delay before returning to the beginning of the loop in block 2030. For example, if a powered wheelchair action is in progress (block 2090) and if no data is received within one second (block 2100) then in bock 2110 a safety stop is executed. The execution of safety stop logic 2110, not detailed here, is a set of logic executed on the communications bridge 1110 and CANBus Protocol 1120 to communicate a stop command over the CANBus 1060 to the powered wheelchair 1040.

[0036] If instead communications data is received in block 2040, then the the keep alive timer is updated in block 2050 to reflect that the EARD 1030 and its associated app 1080 is working properlv and the keep alive timeout timer is reset. Thecommunications data is then processed in block 2060. The logic then determines in block 2070 whether a complete command has been received, as commands may consist of multiple bytes of data that are received into a buffer a few bytes at a time so that only part of the command is received in one loop iteration. If a complete command has not yet been received, the logic returns to the beginning of the loop in block 2030. Once it is determined in block 2070 that a full command is properly received and validated (authentication and checksum techniques may be part of the communications protocol to ensure reliable and authorized operation of the system), the associated powered wheelchair action is executed in block 2080 and the logic returns to the beginning of the loop in block 2030. The execute action logic 2080, not detailed here, is a set of logic executed on the communications bridge 1110 and CANBus Protocol 1120 to communicate a command over the CANBus 1060 to the powered wheelchair 1040.

[0037] Figure 3 is an example block diagram of example microprocessors used to implement the safety watchdog, communications bridge, and security logic in example Eye Gaze Enabled Augmented Reality Control Systems. Components 3000 shows the different microprocessors (coprocessors) present in one example implementation of an EARD powered wheelchair control system. These components include: the safety watchdog coprocessor 3100, the RF communications coprocessor 3200, the security coprocessor 3300, and the drive communications coprocessor 3400.

[0038] The safety watchdog coprocessor 3100 contains the logic for the safety watchdog 1140 and communications bridge 1110, any data associated with the above logic, processor core(s) that execute(s) the logic, and the communications buses that connect it to the other physical components of the system.

[0039] The RF communications coprocessor 3200 contains the hardware and logic necessary to implement any one of the many complex RF communications protocols commonly used on the market today, such as but not limited to Bluetooth LowEnergy (BLE), Thread / Matter, Zigbee, WiFi, etc. In today’s market, BLE and WiFi are commonly implemented on EARDs with Thread & Matter gaining acceptance.

[0040] The security coprocessor 3300 contains the secure secrets storage and one or more cryptography processor cores which enable secure authentication and authorization as described elsewhere.

[0041] The drive communications coprocessor 3400 contains communications buses to interoperate with the safety watchdog coprocessor 3100 and the powered wheelchair 1040 and the logic necessary to communicate to the proprietary CANBus protocol for the model of powered wheelchair. Many powered wheelchair manufacturers hold their communications protocols as trade secrets in order to protect their powered wheelchair’s system reliability, so this drive communications coprocessor 3400 may be a licensed, preprogramed ‘black box’ that helps to protect the systems integrity of the powered wheelchairs in their manufacturer’s eyes, hence little is documented here about this device.

[0042] Figure 7 is an example diagram of an example eye gaze enabled augmented reality device that can be used with example Eye Gaze Enabled Augmented Reality Control Systems such as those described with reference to Figures 1-3. Eye gaze enabled augmented reality device (EARD) 7000 contains the fundamental control components of an HM-EARD without being mounted to the head. In this example, the display and personal computing device are combined into a tablet computer 7200 which has an external facing camera 7100. The tablet computer 7200 is compositing the view of the camera 7100 with the driving app 1080 and displaying the composited view on the tablet display (not shown). A specialty eye gaze camera / emitter bar 7300 determines the gaze point on the tablet display and its data stream is read by a driving app (e.g. , driving app 1080 in Fig. 1 ). The resulting action commands issued by the driving app are then sent to the bridge device 7400 from the tablet computer 7200 over a communications bus (e.g., BLE 1050 in Fig. 1 ) which then issues the action command to the powered wheelchair.

[0043] Example implementations of EARDs available today that may be used in an example EGEARCS include iPad Pro devices with a HID Assistive Touch Eye Trackers such as the TD Pilot (“https: / / www.tobiidynavox.com / pages / tdpilot”) and Irisbond Hiru (“https: / / www.irisbond.com / en / hiru-made-for-ipad”), the Apple Vision Pro (“https: / / www.apple.com / apple-vision-pro”), the Tobii ISeries 13 (“https: / / us.tobiidynavox.com / pages / td-i-series”), and the SmartBox GridPad 13 (“https: / / thinksmartbox.com / grid-pad-13 / ”).

[0044] Figure 4 is an example head mounted eye gaze enabled augmented reality device for use with example Eye Gaze Enabled Augmented Reality Control Systems. A head mounted eye gaze enabled augmented reality device is a device which combines the technologies of a head mount, an internal display, eye gaze sensors for tracking eye focal point in 3D space, a light shield for blocking external light sources, external facing camera(s) which are used to create a pass-through view of the external world on the internal display, a display compositor which combines camera views and other video signals into a unified view on the internal display, and a personal computing device which may be external to the head mount or integral to the head mount. In this example, the head mounted EARD 4000 (or “HM-EARD”) is an Apple Vision Pro device. The components described above with reference to Fig. 7, including camera sensors, eye gaze cameras / emitters, and the compositing logic are attached to the head mounted system to make it operable with example Eye Gaze Enabled Augmented Reality Control Systems.

[0045] Figures 5A-5C and 6 illustrates examples of a set of intrinsic components that can be integrated into a head mounted eye gaze enabled augmented reality device as part of an example Eye Gaze Enabled Augmented Reality Control System. The components illustrated are from patent applications filed by Apple Inc. that are potentially operable with a head mounted EARD such as the Apple Vision Pro device 4000 illustrated in Fig. 4. The examples shown in Figs. 5A-5C and 6 comprise different intrinsic components that together constitute an example HM-EARD thatcan be used to build embodiments suitable for use with example Eye Gaze Enabled Augmented Reality Control Systems.

[0046] In particular, patent diagram 5100 (which is Fig. 3 of U.S. Patent Application Publication No. US20240089605A1 ) illustrates a suite of external camera sensors which are used to perceive the exterior world environment around a head mounted display such as the Apple Vision Pro (AVP). As annotated, cameras 5150, 5152, and 5154 (50, 52, and 54 in the patent figure) are example cameras that can be mounted to the AVP chassis. Similarly, patent diagram 5200 (which is Fig. 3 of U.S. Patent Application Publication No. US20180113508A1 ) as annotated shows further detail including eye gaze light emitters 5230, eye gaze cameras 5240, and an internal display system 5210 including eye pieces 5220 (respectively components 230, 240, 210, and 22 in the patent figure).

[0047] Head mounted displays have particular issues when used for eye gaze enabled technology when used with external lighting situations such as in outside environments. Patent diagram 5300 (from design registration HK 29122023 based upon U.S. Application No. 29 / 893,887 for “Light seal for head-mounted display” in Hong Kong Intellectual Property Journal, published Dec. 29, 2023, Journal No. 2023 / 144 of Designs Registered) illustrates a possible light shield component that can be integrated into a head mounted display such as the Apple Vision Pro (AVP). The light shield blocks exterior light from the inside, or ‘vision center’ of the AVP. In particular, this shield is important for the proper operation of the AVP as it enables the interior displays of the AVP to be clearly visible in all external lighting situations, including bright sunlight, it enables precise eye gaze sensing by removing external sources of light interference, and it enables the use of smaller and less expensive camera and IR emitter systems for eye gaze due to the controlled lighting environment and tightly controlled placement and distances of cameras and emitters from eyes.

[0048] Patent diagram 6000 (which is Fig. 4 of U.S. Patent Application Publication No. US20180113508A1 ) as annotated shows a display compositor 6030 whichcombines visual data from external facing camera sensors (such as camera sensors 5150, 5152, and 5154 in Fig. 5F) and an external device 6100 which is displaying the EARD driving app 1080 which is then rendered together on the display system 5210 and 5220 (Fig. 5B).

[0049] Figure 8 is an example screen display of an example driving application for an eye gaze enabled augmented reality device integrated into an example Eye Gaze Enabled Augmented Reality Control System. Screen display 8000 displays translucent controls overlayed on top of an augmented reality view of the environment in front of the EARD. This application may be displayed, for example, in one implementation, on a tablet computer 7200 (Fig. 7).

[0050] Figure 9 is an example screen display of an example driving application for a head mounted eye gaze enabled augmented reality device integrated into an example Eye Gaze Enabled Augmented Reality Control System. The HM-EARD illustrates the driving controls placed into a 3D spatial environment which is rendered on a display of the HM-EARD.

[0051] Figure 10 is an example screen display of a user interface for selection of an axis for adjustment in an example seating or bed position application that can be integrated into an example Eye Gaze Enabled Augmented Reality Control System for use with an eye gaze enabled augmented reality device. As seen in example screen display 1000, the end user may select an exit icon 1015, tilting the body (icon 1010), reclining the torso / head (icon 1005), raising the chair / bed height (icon 1012), or a lifting the feet / legs (icon 1013). Other buttons may be available such as for lifting the head / chin, turning the head, and the like. The settings icon 1020 allows the end user to configure the posture control application through a configuration interface. For example, the configuration interface may be used to adjust the user experience (such as selection times, color pallets, etc.) and to create ‘preset positions’ for a combination of posture axis position targets, such as ‘resting’ (e.g., back mostly reclined, legs lifted), ‘talking’ (e.g., sitting up, head reclined slightly), ‘driving’ (e.g., body straight, head looking directly forward), etc. The screen display1000 may include choices and selection buttons for these preset positions, not displayed here. A button can be selected from screen display 1000 by gazing at the button for a period of time. In one example, the selected axis is highlighted to the user through a change in color, background, bounding box, etc.

[0052] Figure 11 is an example screen display of a user interface for adjusting an axis of movement in an example seating or bed position application that can be integrated into an example Eye Gaze Enabled Augmented Reality Control System for use with an eye gaze enabled augmented reality device. In the example screen display 1100 illustrated, the axis previously selected in Figure 10 is displayed for reference 1105 and two arrow buttons left 1110 and right 1111 provide the user choice in the change of axis position. The current measured position (status) of the axis 1115 may be displayed in appropriate units, provided that the seating actuator controller can measure position and communicate state back to the EGEARCS. The exit button 1120 can be used to exit this screen and return to the axis selection screen 1000, for example as shown in Figure 10.

[0053] Of note, the first and secondary interfaces may be presented at the same time (e.g., without shifting to a new display screen as shown in Figures 10 and 11 ). Other configurations including sharing a display screen and presenting the adjustment user interface controls once an axis is selected also can be similarly incorporated.

[0054] The techniques and components of an Eye Gaze Enabled Augmented Reality Control System are generally applicable to any type of AR input device, including head mounted and non-head mounted devices. Also, although the examples described herein often refer to a powered wheelchair, the techniques described herein can also be used to control other powered devices (such as using eye gaze enabled AR devices. Also, although certain terms are used primarily herein, other terms could be used interchangeably to yield equivalent embodiments and examples. In addition, terms may have alternate spellings which may or may not be explicitly mentioned, and all such variations of terms are intended to be included.

[0055] Example embodiments described herein provide applications, tools, data structures and other support to implement an Eye Gaze Enabled Augmented Reality Control System to be used for controlling powered wheelchairs, vehicles, or hospital beds. Other embodiments of the described techniques may be used for other purposes. In this description, numerous specific details are set forth, such as data formats and code sequences, etc., in order to provide a thorough understanding of the described techniques. The embodiments described also can be practiced without some of the specific details described herein, or with other specific details, such as changes with respect to the ordering of the logic, different logic, etc. Thus, the scope of the techniques and / or functions described are not limited by the particular order, selection, or decomposition of aspects described with reference to any particular routine, module, component, and the like.

[0056] Furthermore, in some embodiments, some or all of the components of the Eye Gaze Enabled Augmented Reality Control System may be implemented or provided in other manners, such as at least partially in firmware and / or hardware, including, but not limited to one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers executing appropriate instructions, and including microcontrollers and / or embedded controllers, field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and the like. Some or all of the system components and / or data structures may also be stored as contents (e.g., as executable or other machine-readable software instructions or structured data) on a computer-readable medium (e.g., a hard disk; memory; network; other computer-readable medium; or other portable media article to be read by an appropriate drive or via an appropriate connection, such as a DVD or flash memory device) to enable the computer-readable medium to execute or otherwise use or provide the contents to perform at least some of the described techniques. Some or all of the components and / or data structures may be stored on tangible, non-transitory storage mediums. Some or all of the system components and data structures mav also be stored as data signals (e.q., bv being encoded as part of acarrier wave or included as part of an analog or digital propagated signal) on a variety of computer-readable transmission mediums, which are then transmitted, including across wireless-based and wired / cable-based mediums, and may take a variety of forms (e g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, embodiments of this disclosure may be practiced with other configurations.

[0057] All of the above U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification and / or listed in the Application Data Sheet, including but not limited to U.S. Provisional Patent Application No. 63 / 525,766, entitled “Ability Drive Methods and Systems for Apple Vision Pro,” filed July 10, 2023, are incorporated herein by reference, in its entirety.

[0058] From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention. For example, the methods and systems for performing eye gaze enabled control discussed herein are applicable to other architectures other than a AR head mounted display architecture. Also, the methods and systems discussed herein are applicable to differing protocols, communication media (optical, wireless, cable, etc.) and devices (such as wireless handsets, electronic organizers, personal digital assistants, portable email machines, game machines, pagers, navigation devices such as GPS receivers, etc.).

Claims

CLAIMS1 . An electronically powered device comprising: one or more actuators configured to adjust the position or motion of one or more components of the device; a control device with an display with an associated detection device and associated with microcontroller implemented logic to capture eye position, brain waves, or muscle activation of the occupant in order to determine occupant intent to select one or more selectable virtual user interface controls presented on the display; one or more external facing cameras which are used to create a pass- through view of the external world on the internal display; a display compositor which combines camera views and other video signals into a unified view on the internal display; a set of one or more microcontrollers for establishing connections, receiving commands from the control device, and issuing commands to control actuation of the one or more actuators of the device; and code logic configured, when executed by one or more of the microcontrollers, to: responsive to notification of a selected one of the one or more virtual user interface controls, determine a target actuation of the one or more actuators of the powered device; responsive to notification of an adjustment amount, determine an amount to adjust the determined target actuation; and automatically cause one or more of the one or more actuators associated with the target actuation to change position or cause motion according to the determined adjustment amount, thereby causing position or motion of the occupant to change solely responsive to selection of virtual user interface controls without the occupant typing commands or oral command input.

2. The device of claim 1 , further comprising: a light shield which blocks external light sources and sources of interference; and a head mounting system used to attach and support the control device relative to the head of the occupant.

3. The device of claims 1 or 2 wherein the head mounting systems is used to attach and support the one or more external facing cameras and the display compositor and / or the head mounting system is used to attach and support the light shield4. The device of at least one of the above claims wherein at least one of the set of microcontrollers is further configured to execute watchdog logic configured to observe the issued commands to make safety judgements based upon a content and timing of the issued commands and to independently issue one or more safety commands to control actuation of the one or more actuators of the device.

5. The device of claim 4 wherein the watchdog logic is further configured to independently issue the one or more safety commands responsive to a timeout.

6. The device of claims 4 or 5 wherein the one or more safety commands include stopping movement of the device when a failure of the control system is observed.

7. The device of at least one of the above claims wherein at least one of the set of microcontrollers is further configured to provide identity and authentication services or logic used to validate that the device is authorized for use with the control system.

8. The device of at least one of the above claims wherein the one or more actuators configured to adjust the position or motion of the device adjust an adjustable seat or bed frame to affect posture of the occupant.

9. The device of at least one of the above claims wherein the set of one or more microcontrollers further comprises a microcontroller for receiving commands from a wireless radio frequency bus and transforming the received commands from the wireless radio frequency bus to commands specific to the electronically powered device.

10. The device of at least one of the above claims wherein the set of one or more microcontrollers further comprises a microcontroller for receiving commands from a wired bus protocol and transforming the received commands from the wired bus protocol to commands specific to the electronically powered device.11 . The device of at least one of the above claims wherein the set of one or more microcontrollers further comprises an authentication microcontroller.

12. The device of at least one of the above claims wherein set of one or more microcontrollers receives commands from the control device via a first external communications bus and issues commands to a second external communications bus to control actuation of the one or more actuators of the device.

13. The device of at least one of the above claims wherein the device is a powered wheelchair or a hospital bed frame.

14. A method for controlling an electrically powered device, the device capable of adjustments to a plurality of positions or motions and having one or more actuators configured to adjust the position or motion of the powered device comprising: via an eye qaze enabled device,capturing one or more eye positions, brain waves, or muscle activation of the occupant in order to determine occupant intent to select one or more selectable virtual user interface controls presented on a display communicatively connected to the powered device; creating and presenting a pass-through view of the external world on the display; and compositing views received from one or more camera signals and other video signals into a unified view on the display; and via a bridge control system, receiving commands from the display via a first external communications bus, and issuing commands to a second external communications bus to control actuation of the one or more actuators of the powered device by issuing commands; responsive to notification of a selected one of the one or more virtual user interface controls, determining a target actuation; responsive to notification of an adjustment amount, determining an amount to adjust the determined target actuation; and automatically causing one or more of the one or more actuators associated with the target actuation to change position or cause motion according to the determined adjustment amount, thereby causing position or motion of the occupant to change solely responsive to selection of virtual user interface controls without the occupant typing commands or oral command input.

15. The method of claim 14, further comprising: via the bridge control system, executing watchdog logic to observe the issued commands and to make safety judgements based upon a content and timing of the issued commands and independently issue one or more safety commands to control actuation of the one or more actuators of the device.

16. The method of claims 14 or 15, further comprising, under control of a separate authentication chip, validating that the bridge control system is authorized for use with the electronically powered device.

17. The method of at least one of claims 14 to 16 wherein the eye gaze enabled device is a head mounted augmented reality device or a personal computing device.

Images