Augmented reality system with key moment indicator

Abstract

An augmented reality display system configured to capture key moments in a user's day in their entirety. The system captures, on a continual running basis, audio / video data of a user's surroundings captured by sensors of the system. Upon receiving an input from the user, a time stamp is generated within the audio / video data. Based on user-configurable settings, or on the input provided from the user, the system generates discrete recordings that include audio / video data captured an increment of time prior to, during, and / or after the time stamp. This allows for a user to generate discrete recordings that have already taken place but were nevertheless captured by the device prior to and / or after the user's command / input.

Description

AUGMENTED REALITY SYSTEM WITH KEY MOMENT INDICATORTECHNICAL FIELD

[0001] The present disclosure generally relates to electronic displays, and more particularly to near-eye display systems capable of recording audio and / or video data.BACKGROUND

[0002] Near-eye displays are being developed for a range of diverse uses, including military, commercial, industrial, firefighting, and entertainment applications. For many of these applications, there is value in forming a virtual image that can be visually superimposed over the real-world image that lies in the field of view of the user. In some circumstances near-eye displays can utilize a camera to record live video of the real world while the user is wearing the near-eye display.

[0003] Customers and other end-users have increasingly found it beneficial or fulfilling to record portions of their daily experiences via one or more electronic media recording devices, e.g., smart phones, cameras, etc. However, end-users rarely have these devices out and ready to capture content prior to the commencement of an event they wish to capture. Additionally, these electronic media recording devices are generally bulky and may have limited data storage capacity making it impractical or less desirable to keep these devices out and recording constantly to ensure that all interesting moments in an end-user’s life are captured or recorded. There is thus a need for a head-mountable device capable of recording, in their entirety, live audio / video data relevant to key moments or events in an end-user’s life.SUMMARY

[0004] It is an object of the present disclosure to advance the art of head-mounted recording devices having one or more sensors capable of obtaining audio and / or video data of a customer, end-user, or wearer’s or user’s surroundings.

[0005] The aspects, objects, features and advantages of the present invention will be more clearly understood and appreciated from the following detailed description of the embodiments and appended claims, and by reference to the accompanying drawings. In an exemplary embodiment, the present disclosure provides an augmented reality display system that includes a near-eye display. The near eye display includes one or more sensors arranged to capture audio data and / or video data, a processor; and a non-transitory computer-readable memory arranged to store a set of non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed, the system in configured to: receive the audio data and / or the video data from the one or more sensors; receive a user input; generate and store a timestamp associated withthe audio data and / or the video data that temporally corresponds with the receipt of the user input; and save at least one recording that includes the audio data and / or the video data received between an increment of time before the timestamp and the timestamp; or, save at least one recording that includes the audio data and / or the video data received between the timestamp and an increment of time after the timestamp.

[0006] In another exemplary embodiment a method of recording audio or video data using a neareye display is provided, the method comprising the steps of: recording, via one or more sensors of the near-eye display audio data and / or video data; receiving a user input; creating and storing a timestamp associated with the audio data and / or video data that temporally corresponds with the receipt of the user input; and storing, as a discrete file, at least one recording that includes the audio data and / or video data recorded between an increment of time before the timestamp and the timestamp; or storing, as a discrete file, at least one recording that includes the audio data and / or video data recorded between the timestamp and an increment of time after the timestamp.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings are incorporated herein as part of the specification. The drawings described herein illustrate embodiments of the presently disclosed subject matter and are illustrative of selected principles and teachings of the present disclosure. However, the drawings do not illustrate all possible implementations of the presently disclosed subject matter and are not intended to limit the scope of the present disclosure in any way.

[0008] FIG. 1 is a schematic representation of an augmented reality display system according to exemplary7embodiments of the present disclosure.

[0009] FIG. 2 is a schematic representation of the components of a near-eye display system according to exemplary' embodiments of the present disclosure.

[0010] FIG. 3 is a schematic representation of audio visualization tool showing audio data utilized to generate one or more recordings according to exemplary embodiments of the present disclosure.

[0011] FIG. 4 is a schematic representation of audio visualization tool showing audio data utilized to generate one or more recordings according to exemplary embodiments of the present disclosure.

[0012] FIG. 5 is a schematic representation of audio visualization tool showing audio data utilized to generate one or more recordings according to exemplary embodiments of the present disclosure.

[0013] FIG. 6 is a schematic representation from the perspective of a user or wearer of a near-eye display system 102 according to exemplary embodiments of the present disclosure.

[0014] FIG. 7 is a schematic representation from the perspective of a user or wearer of a near-eye display system 102 according to exemplary embodiments of the present disclosure.

[0015] FIG. 8 is a top-plan schematic representation of two users, one with a near-eye display system 102 and the other with a privacy proximity device according to exemplary embodiments of the present disclosure.

[0016] FIG. 9 is a schematic representation from the perspective of a user or wearer of a near-eye display system 102 in a privacy mode according to exemplary embodiments of the present disclosure.

[0017] FIG. 10 is a flow chart illustrating the steps of an exemplary method according to exemplary embodiments of the present disclosure.DETAILED DESCRIPTION

[0018] It is to be understood that the invention may assume various alternative orientations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific assemblies and systems illustrated in the attached drawings and described in the following specification are simply exemplary embodiments of the inventive concepts defined herein. Hence, specific dimensions, directions, or other physical characteristics relating to the embodiments disclosed are not to be considered as limiting, unless expressly stated otherwise. Also, although they may not be, like elements in various embodiments described herein may be commonly referred to with like reference numerals within this section of the application.

[0019] One skilled in the relevant art will recognize that the elements and techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In some instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects of the present disclosure. Reference throughout the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearance of the phrase “in one embodiment” or “in an embodiment” throughout the specification is not necessarily referring to the same embodiment. However, the particular features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments.

[0020] Where used herein, the terms “first”, “second”, and so on, do not necessarily denote any ordinal, sequential, or priority relation, but are simply used to more clearly distinguish one element or set of elements from another, unless specified otherwise.

[0021] Where used herein, the terms ‘"viewer”, "‘operator”, "‘observer”, “wearer”, and "‘user” are considered equivalents and refer to the person, or machine, that wears and / or views images using a device having an imaging light guide.

[0022] Where used herein, the term “set” refers to a non-empty set, as the concept of a collection of elements or members of a set is widely understood in elementary mathematics. Where used herein, the term “subset”, unless otherwise explicitly stated, refers to a non-empty proper subset, that is, to a subset of the larger set, having one or more members. For a set S, a subset may comprise the complete set S. A “proper subset” of set S, however, is strictly contained in set S and excludes at least one member of set S.

[0023] Turning now to the figures, FIG. 1 shows an exemplar} embodiment of an augmented reality display system 100 according to the present disclosure. In some example embodiments, augmented reality display system 100 includes a near-eye display 102. In other example embodiments, as illustrated in FIG. 1, augmented reality display system 100 comprises a near-eye display 102 and at least one: peripheral device 104, server 106, and / or privacy proximity device 108. Each of these devices and / or components are discussed below in further detail.

[0024] As shown in FIG. 1, near-eye display 102 is intended to be a monocular or binocular pair of smart glasses or a monocular or binocular head-mountable display configured to generate one or more virtual images within the field of view of a user or wearer to provide the user with an augmented reality experience. For example, as illustrated in FIG. 1, the near-eye display 102 is illustrated as a binocular pair of smart glasses having one or more waveguide modules positioned in front of each of the user’s eyes (when worn) that are configured to relay and project one or more virtual images to at least one of the user’s eyes. For more information about the operating principles associated with waveguides and waveguide modules see United States Patent No. 11,194,158, filed July 5, 2018, entitled “Light Guide with Beam Separator for Dual Images, and United States Patent Application Publication No. US 2021 / 0103146, filed June 18, 2020, and entitled “Augmented Reality Display System”, which applications are incorporated herein by reference in their entireties.

[0025] With continued reference to FIG. 1, peripheral device 104 is intended to be a device or collection of devices that are capable of communicating, via a wired or wireless data connection, directly or indirectly with near-eye display 102 and / or server 106. As illustrated, peripheral device 104 is intended to be a device with some form of computational processing capability, e.g., a cell phone or a smart phone; however, it should be appreciated that peripheral device 104 can be selected from at least one of: a laptop or other personal computer, a tablet, a smart watch, or adevice capable of obtaining and transmitting physiological data of its user / wearer. As will be described below, peripheral device 104 may send or receive one or more signals (described below) via the direct or indirect connection between near-eye display 102 and peripheral device 104. Additionally, peripheral device 104 may include an internal memory, and near-eye display 102 may send one or more pieces of digital data, e.g., audio data 124 and / or video data 126 (both described below and shown in at least FIG. 2), to peripheral device 104 for storage within that internal memory.

[0026] Server 106 is intended to be a remote storage device capable of communicating digital information with one or more devices of augmented reality system 100 via the internet I. For example, server 106 may be a remote storage drive or network of storage drives capable of receiving and subsequently storing one or more pieces of digital data, e.g., audio data 124 and / or video data 126 (both described below). Additionally, and as will be discussed below, server 106, via one or more data connections (wired or wireless), may also transmit one or more pieces of stored digital data to one or more devices of augmented reality display system 100.

[0027] Privacy proximity device 108 is intended to be an object or collection of objects that is / are capable of transmitting one or more wireless signals, e.g., privacy signal 144 (discussed below) to indicate proximity and / or location of a person or place that does not wish to be or should not be the subject of a data recording, e.g., recording 128 (described below). As shown, privacy proximity device 108 may take the form of a key fob, key chain, puck, or other device capable of at least sending the privacy signal 144 described herein.

[0028] As will be described in further detail, near-eye display 102 can establish one or more wired or wireless data connections (illustrated in FIG. 1 as broken lines) with the other devices of augmented reality display system 100, e.g.. near-eye display 102 can connect and communicate digital information directly with peripheral device 104, remote server 106, and / or privacy proximity device 108. Additionally, and in exemplary embodiments, near-eye display 102 can connect indirectly, via connection over the internet I, with peripheral device 104, server 106, and / or privacy proximity device 108. These data connections may utilize one or more data protocols to encode and / or decode digital information being sent or received, e.g., data protocols selected from at least one of: a Wi-Fi protocol (e.g., 802. 1 In / g / a / b / ax), a Bluetooth protocol (e.g., Bluetooth Low-Energy (BLE), Bluetooth Asynchronous Connection-Less (ACL), Bluetooth Synchronous Connection-Oriented link (SCO)), a Zigbee protocol, a Z-Wave protocol, a Radio Frequency Identification (RFID) protocol. aNear-Field Communication (NFC) protocol, etc.

[0029] FIG. 2 illustrates a schematic representation of one exemplary configuration of the component parts of near-eye display 102. In some examples near-eye display 102 comprises a processor 110 and non- transitory computer-readable memory 112 configured to execute and store, respectively, a plurality of non- transitory- computer-readable instructions 114 to perform the various functions of the near-eye display 102 as will be described herein.

[0030] Additionally, and as shown, in some example embodiments near-eye display 102 further comprises one or more sensors 116, a communications module 118, and a power storage device 120. One or more sensors 116 are selected from: a camera, a microphone, a touch-capacitive sensor, a touch-sensitive screen, a gyroscope, an accelerometer, a magnetometer, a button, a toggle or slide switch, or a scroll wheel. As will be described below, the one or more sensors 116 are utilized by the augmented reality system 100 to provide one or more user input(s) 122 that may act to initiate, modify, or terminate a recording 128 or act to establish a timestamp 132 for a discrete recording 128 (described below).

[0031] Communications module 118 can include one or more antennae, radio, or some form of automated gain control (AGC), a modulator, demodulator and / or potentially a discrete processor for bit-processing. As described herein, communications module 118 is utilized by near-eye display 102 to send and / or receive wired or wireless communications with the one or more devices within augmented reality display system 100, e g., peripheral device 104, server 106, and / or privacy proximity device 108. Power storage device 120 can be selected from at least one of: a battery, capacitor, or supercapacitor and is arranged to store and provide electronic power to the other components of the near-eye display 102 as described herein and illustrated in at least FIG. 2.

[0032] In example embodiments, one or more devices of the augmented reality display system 100 is / are configured to obtain audio data 124 and / or video data 126. For example, the one or more sensors 116 of near-eye display 102 may include a microphone and / or a camera arranged to obtain audio data 124 and / or video data 126, respectively. In this example, audio data 124 is intended to mean the digital or analog signals used to embody or represent a sound generated externally from the near-eye display 102. Additionally, video data 126 is intended to mean the digital or analog signals used to embody or represent an image or series of images from a source that is external to the near-eye display 102. As will be described further below, a memory- device of the augmented reality display system 100 is configured to store audio data 124 and / or video data 126 as discrete recordings 128 or discrete recording files. It should be appreciated that a recording 128 can include only audio data 124, only video data 126, or can include audio data 124 and video data 126.

[0033] With continued reference to FIG. 2, and in example embodiments, near-eye display 102 further includes a first-in-first-out (FIFO) memory device 130 capable of storing data (e.g., audio data 124 and / or video data 126) obtained by the one or more sensors 116. As illustrated, the FIFO memory device 130 can be a discrete device within near-eye display 102 or it may be formed as a portion of non-transitory computer-readable memory 112. The FIFO memory device 130 is intended to be a ring buffer, circular buffer, circular queue, a cyclic buffer or any other data structure that uses a fixed-buffer size as if it were connected end-to-end. As a result, the FIFO memory' device 130 can store a fixed amount of continuous data before the first data element in the ring or circular data structure is overwritten with new data or a new data element. In example embodiments, FIFO memory device 130 has a fixed storage capacity capable of storing sixty - minutes of continuous audio data 124 and / or video data 126. In other words, FIFO memory' 130 is capable of storing a sixty-minute-long discrete recording 128. In other example embodiments, the FIFO memory device 130 is capable of storing a recording 128 of approximately thirty’ minutes, twenty minutes, fifteen minutes, ten minutes, five minutes, four minutes, three minutes, two minutes, one minute, thirty seconds, fifteen seconds, ten seconds, five seconds, one second, or any integer quantity of seconds or minutes therebetween. As such, during operation of near-eye display 102, and as will be described below in further detail, near-eye display 102 is capable of obtaining and storing a recording 128 (e.g.. audio data 124 and / or video data 126) on a running basis. For example, should the FIFO memory device 130 have enough storage capacity' to record sixty minutes of continuous audio data 124 and / or video data 126, then the near-eye display 102 will be capable of generating or creating a discrete recording 128 based on the audio data 124 and / or the video data 126 obtained by the one or more sensors 116 of any fixed window of time within that sixty minute data window. Although FIG. 2 illustrates the FIFO memory device 130 as a component of near-eye display' 102, it should be appreciated that in other exemplary' embodiments FIFO memory' device 130 is a component of a remote storage device, e.g., peripheral device 104 or server 106. and the audio data 124 and / or video data 126 can be sent and / or received by said remote storage device via one or more wired or wireless connections described above.

[0034] FIGS. 3-5 illustrate schematic visualizations of an example recording 128 that includes only audio data 124. As shown, the audio data 124 is represented as though executed by an audio visualization tool where different volume or frequency data is used to represent rhythm, loudness, and / or frequency ranges of the audio data 124 as a function of time (where time flows from left to right). As shown, the total length of the audio file from left to right in FIGS. 3-5 represents the total memory capacity of FIFO memory device 130 as described above. It should be appreciated that only portions of that total memory capacity may include audio data 124 of sufficient intensity'or frequency to be received by the one or more sensor 116 (e.g., a microphone) and therefore, there may be portions of the visualized data stream that are represented by a flat line rather than contiguous or adjacent vertical segments.

[0035] With reference to FIGS. 1-5, during operation of the augmented reality system 100 described herein, the user may provide one or more user inputs 122 (also used in the singular "user input 122”) that is / are operable to toggle on or off a recording mode RM for the near-eye display 102. User input 122 can include user engagement with one or more sensors 116, e.g., user engagement with: a touch-capacitive sensor, a touch-sensitive screen, a button, a toggle or slide switch, and / or a scroll wheel. User input 122 can also be defined by: a user's hand gesture as detected by the camera; user movement data based on the gyroscope, accelerometer, and / or magnetometer data; or a verbal command or wake word detected by the microphone after application of known audio processing techniques. It should be appreciated that user input 122 can be provided through one or more peripheral devices 104 as will be described below. When the recording mode RM is “on” or “active”, the one or more sensors 116 are configured to obtain and relay audio data 124 and / or video data 126 to FIFO memory device 130. Furthermore, when the recording mode RM is on or active the FIFO memory device 130 records the aforesaid data (e.g., audio data 124 and / or video data 126) until the maximum storage capacity of the FIFO memory device 130 is reached, at which point new data will overw rite the oldest memory element in a continuous loop. As will be described below with respect to privacy proximity device 108, the recording mode RM may be toggled to an “off” or “inactive” state as long as the near-eye display 102 continues to receive the privacy signal 144. In other examples augmented reality display system 100 and / or near-eye display 102 are configured to be in an “on” or “active” mode continuously as long as the devices are on and / or have sufficient electrical power, e.g., the system and near-eye display do not need to receive a user input 122 to initiate recording of audio data 124 and / or video data 126.

[0036] When the near-eye display 102 and / or the augmented reality display system 100 is in an on or active recording mode RM, the user may provide a user input 122. Receipt of a user input 122, while in a recording mode RM, operates to generate or create a timestamp 132 within the data stored by the FIFO memory device 130 that is contemporaneous with the receipt of the user input 122 and is associated with a single discrete recording 128. As will be described below, the timestamp 132 operates as an anchor point within the continuous stream of data (e.g., audio data 124 show n in FIG. 3) from which the temporal boundaries or parameters of the discrete recording 128 are determined. For example, the augmented reality7display system 100 described herein is operable in at least two recording modes RM, a default recording mode DRM, and a numericallydriven recording mode NDM. In the default recording mode DRM, the temporal parameters of the discrete recording are determined by a user-configurable setting established prior to the user input 122 that generates the timestamp 132. In the numerically driven recording mode NDM, one or more sensors 116 are utilized to determine, contemporaneously with the generation of the timestamp 132, an increment of time 134 from a numerical value 140 provided as a part of a user input 122. In some examples, the increment of time 134 can be used to generate an increment of time before the timestamp 136 and / or generate an increment of time after the timestamp 138. It should be appreciated that FIGS. 3-5 will provide visual context for both a default recording mode DRM and a numerically driven recording mode NDM as will be described below.

[0037] For example, in a default recording mode DRM, the user may (prior to providing a user input 122 that is operable to create a timestamp 132 and prior to recording any data) save, as a user-configurable setting, a default increment of time 134 and an indication as to whether the default increment of time 134 should be used as an increment of time before the timestamp 136, an increment of time after the timestamp 138, or both an increment of time before the timestamp 136 and an increment of time after the timestamp 138. The increment of time 134 can be selected from one of thirty minutes, twenty minutes, fifteen minutes, ten minutes, five minutes, four minutes, three minutes, two minutes, one minute, thirty seconds, fifteen seconds, ten seconds, five seconds, one second, or any integer quantity of seconds or minutes therebetween. While in the default recording mode DRM and while the device is obtaining audio data 124 and / or video data 126 and storing that data in the FIFO memory device 130. should the user provide a user input 122 operable to create or generate a timestamp 132 within the audio data 124 and / or video data 126, the temporal parameters of a discrete recording 128 will be determined based on the timestamp 132, the default increment of time 134, and the indication by the user as to whether the default increment of time 134 should be applied before, after, or both before and after the timestamp 132.

[0038] For example, as shown in FIG. 3, in examples where the FIFO memory device 130 can store up to sixty minutes of data, a user could set the default increment of time 134 to be equal to five minutes and indicate that the default increment of time 134 should be used as an increment of time before the timestamp 136. Thereafter, during a default recording mode DRM, the user could provide a user input 122, e.g.. via user interaction with a button or switch on the near-eye display system 102 that creates a timestamp 132 within the audio data 124 and / or video data 126 being stored in the FIFO memory device 130. Since the FIFO memory device 130 has stored the audio data 124 and / or video data 126 of the previous sixty minutes, a discrete recording 128 is generated that includes the audio data 124 and / or video data 126 obtained between the incrementof time before the timestamp 136 (e.g., five minutes before the timestamp 132) and the timestamp 132. This would generate a discrete recording 128 that is five minutes long which represents the previous five minutes of time prior to the creation of the timestamp 132.

[0039] In another example, as shown in FIG. 4, where the FIFO memory device 130 can store up to sixty minutes of data, a user could set the default increment of time 134 to be equal to five minutes and indicate that the increment of time 134 should be used as an increment of time after the timestamp 138. Thereafter, and during a default recording mode DRM, the user could provide a user input 122, e.g., via user interaction with a button or switch on the near-eye display system 102 that creates a timestamp 132 within the audio data 124 and / or video data 126 being stored in the FIFO memory device 130. Since the FIFO memory device 130 can store the audio data 124 and / or video data 126 for a total period of sixty minutes, once the increment of time after the timestamp 138 has elapsed, a discrete recording 128 or recording file is generated that includes the audio data 124 and / or video data 126 obtained between the timestamp 132 and the increment of time after the timestamp 138 (e.g., five minutes after the timestamp 132). This would generate a discrete recording 128 that is five minutes long that represents the next five minutes of time after the creation of the timestamp 132.

[0040] Within the default recording mode DRM, a combination of those exemplary embodiments is possible. For example, and as shown in FIG. 5, where the FIFO memory device 130 can store up to sixty7minutes of data, a user could set the default increment of time 134 to be equal to five minutes and indicate that the increment of time 134 should be used as both an increment of time before the timestamp 136 and an increment of time after the timestamp 138. Thereafter, the user could provide a user input 122, e.g., via user interaction with a button or switch on the near-eye display7system 102 that creates a timestamp 132 within the audio data 124 and / or video data 126 being stored in the FIFO memory device 130. Since the FIFO memory device 130 can store the audio data 124 and / or video data 126 for a total period of sixty minutes, once the increment of time after the timestamp 138 has elapsed, a discrete recording 128 is generated that includes the audio data 124 and / or video data 126 obtained between the increment of time before the timestamp 136 and the increment of time after the timestamp 138 (e.g., five minutes before and five minutes after the timestamp 132). This would generate a discrete recording 128 that is ten minutes long that represents the five minutes before the creation of the timestamp 132 and five minutes after the creation of the timestamp. Although not shown in FIG. 5, it should be appreciated that, prior to entering a default recording mode DRM, the user can provide multiple indications via user configurable settings that would allow for the increment of time before the timestamp 136 and the increment of time after the timestamp 138 to be different values. For example, the user mayestablish, via user configurable settings, that the increment of time before the timestamp 136 should be one minute while the increment of time after the timestamp 138 should be five minutes.

[0041] As mentioned above, the augmented reality display system 100 is also operable within a numerically driven recording mode NDM, as the visual representation of this example implementation is the same as the example implementation for the default recording mode DRM, FIGS. 3-5 continue to illustrate the concepts described herein. For example, in a numerically driven recording mode NDM, the user may (prior to providing a user input 122 that is operable to create a timestamp 132 and prior to recording any data) save, as a user-configurable setting, an indication as to whether an increment of time 134 should be used as an increment of time before the timestamp 136, an increment of time after the timestamp 138, or both an increment of time before the timestamp 136 and an increment of time after the timestamp 138. While in the numerically driven recording mode and while the device is obtaining audio data 124 and / or video data 126 and storing that data in the FIFO memory device 130, the user can provide a user input 122 operable to create or generate a timestamp 132 that includes a numerical value 140 (shown in FIG. 2) that can be used to determine the temporal parameters of a discrete recording 128 relative to the timestamp 132.

[0042] For example, as shown in FIG. 3. where the FIFO memory device 130 can store up to sixty minutes of data, a user could indicate that the increment of time 134 should be used as an increment of time before the timestamp 136. Thereafter, and within a numerically driven recording mode NDM, the user could provide a user input 122 via one or more sensors 116, e.g.. a spoken / verbal command such as “Record 5” via a microphone of the near-eye display system 102. One or more devices of the augmented reality display system 100 can then receive the user input 122 (i.e., the verbal command) and parse the verbal command using known speech / audio recognition techniques to determine that a numerical value 140 of “five” had been spoken within the command and uses the numerical value 140 to determine the increment of time before the timestamp 136. Contemporaneously with receiving the user input 122 (e.g., the verbal command), the augmented reality display system 100 creates a timestamp 132 within the audio data 124 and / or video data 126 being stored in the FIFO memory device 130. Since the FIFO memory device 130 has stored the audio data 124 and / or video data 126 of the previous sixty minutes, a discrete recording 128 is generated that includes the audio data 124 and / or video data 126 obtained between the increment of time before the timestamp 136 (i.e., five minutes before the timestamp 132) and the timestamp 132. This would generate a discrete recording 128 that is five minutes long that represents the previous five minutes of time prior to the spoken verbal command and the creation of the timestamp 132.

[0043] In additional examples shown in FIG. 4, where the FIFO memory device 130 can store up to sixty minutes of data, a user could indicate that the increment of time 134 should be used as an increment of time after the timestamp 138. Thereafter, and during a numerically driven recording mode NDM. the user could provide a user input 122 via one or more sensors 116, e.g.. a spoken / verbal command such as "Record 5” via a microphone of the near-eye display system 102. One or more devices of the augmented reality display system 100 can then receive the user input 122 (i.e., the verbal command) and parse the verbal command using known speech / audio recognition techniques to determine that a numerical value 140 of “five” had been spoken within the command and uses the numerical value 140 to determine the increment of time after the timestamp 138. Contemporaneously with receiving the user input 122 (e.g., the verbal command), the augmented reality display system 100 creates a timestamp 132 within the audio data 124 and / or video data 126 being stored in the FIFO memory device 130. Since the FIFO memory device 130 can store the audio data 124 and / or video data 126 for a total period of sixty minutes, once the increment of time after the timestamp 138 has elapsed, a discrete recording 128 is generated that includes the audio data 124 and / or video data 126 obtained between the timestamp 132 and the increment of time after the timestamp 138 (i.e., five minutes after the timestamp 132). This would generate a discrete recording 128 that is five minutes long that represents the next five minutes of time after the spoken verbal command and the creation of the timestamp 132.

[0044] Within the numerically driven recording mode NDM. and as shown in FIG. 5, a combination of those exemplary embodiments is possible. For example, where the FIFO memory device 130 can store up to sixty minutes of data, a user could indicate that the increment of time 134 should be used as both an increment of time before the timestamp 136 and as an increment of time after the timestamp 138. Thereafter, the user could provide a user input 122 via one or more sensors 116, e.g.. a spoken / verbal command such as “Record 5” via a microphone of the near-eye display system 102. One or more devices of the augmented reality display system 100 can then receive the user input 122 (i.e., the verbal command) and parse the verbal command using known speech / audio recognition techniques to determine that a numerical value 140 of “five” had been spoken within the command and uses the numerical value 140 to determine the increment of time before the timestamp 136 and the increment of time after the timestamp 138. Contemporaneously with receiving the user input 122 (e.g., the verbal command), the augmented reality display system 100 creates a timestamp 132 w ithin the audio data 124 and / or video data 126 being stored in the FIFO memory device 130. Since the FIFO memory' device 130 can store the audio data 124 and / or video data 126 for a total period of sixty minutes, once the increment of time after the timestamp 138 has elapsed, a discrete recording 128 is generated that includes the audio data 124 and / orvideo data 126 obtained between the timestamp 132 and the increment of time after the timestamp 138 (i.e., five minutes after and five minutes before the timestamp 132). This would generate a discrete recording 128 that is ten minutes long that represents the five minutes before the spoken verbal command (and the creation of the timestamp 132) and five minutes after the spoken command (and the creation of the timestamp 132). It should be appreciated that, when receiving a user input 122, the input can include multiple numerical values 140, e.g., the user can provide multiple numerical values 140 sequentially within the same command that would allow for the increment of time before the timestamp 136 and the increment of time after the timestamp 138 to be different values. For example, the user may establish, via a user input 122, that the increment of time before the timestamp 136 should be one minute while the increment of time after the timestamp 138 should be five minutes, e.g., by speaking the verbal command “Record 1 then 5”.

[0045] It should also be appreciated that, in any of the foregoing exemplary embodiments, the total recording time of any discrete recording 128 can be lessened or cut short via a subsequent user input 122. For example, within the increment of time after the timestamp 138, should the user provide a user input 122, e.g.. verbal command, button push, hand gesture, or head motion, the discrete recording 128 would be generated representing the data obtained between the increment of time before the timestamp 136 and the receipt of the user input 122, or in the case that the user has indicated that the increment of time 134 is only an increment of time after the timestamp 136, the recording 128 would be generated representing data obtained between the timestamp 132 and the user-input 122. Furthermore, any recording 128 and / or data obtained by the near-eye display 102 can be transmitted, directly or via internet I (shown in FIG. 1), to an external application, to peripheral device 104, or cloud-based portal for further editing and cataloging. For example, a user may create or generate multiple discrete recordings 128 within a given time range. The user can then upload all of the discrete recording files to a cloud-based application that can store, sort, edit, and replay, the recordings back to the user at their convenience. The cloud-based application can be used to combine or concatenate all recordings within a certain window, e.g., within a given day or week into a larger discrete file for viewing all key moments or events in a given day / week. In other examples, an artificial intelligence program could splice, cut, concatenate or combine any of the recordings uploaded to the cloud-based portal, e.g., to combine and organize the most relevant aspects of a work -place accident. Additionally, it should be appreciated that in examples where the user has indicated that the increment of time 134 should be used as both an increment of time before the timestamp 136 and an increment of time after the timestamp 138, rather than create one complete recording 128 that include both time periods, augmented reality display system 100 can be configured such that two discrete recordings 128 are created, one recording128 representing the data recorded between the increment of time before the timestamp 136 and the timestamp 132, and a second recording representing the data obtained between the timestamp 132 and the increment of time after the timestamp 138. In these examples, the processor 110 can be configured to concatenate these two files prior to or after transmitting the data / recordings to an external source, e.g.. to a peripheral device 104 or a cloud-based application.

[0046] The foregoing exemplary embodiments of augmented reality display system 100 are especially useful to capture or record key moments or events happening around the user but potentially after the user recognizes the importance of the event or the actions or context leading up to an event. For example, a user may realize after the conclusion of an event, that it would have been desirable to record the event in its entirety or record the event and the context or build-up of events prior to the event. As an illustration of one example use, you may realize after your child blows out the candles on their birthday cake that it would have been desirable to record for a keepsake. Because the augmented reality display system 100 records a running sixty-minute’s worth of audio data 124 and / or video data 126, a wearer or a user wearing the near eye display 102 can simply press a button or speak a verbal command “Record 5” or “Record 10” and a discrete recording file 128 will be created and saved that encompasses the last five or ten minutes around the user (and / or 5-10 minutes into the future), potentially capturing the key moment or event within the recording 128 for future use. The augmented reality display system 100 according to these exemplary embodiments is also useful for journalling, i.e., for keeping track of interesting things that happen to the user throughout the day, week, etc. The augmented reality display system 100 is also useful to record and catalog context and the build-up prior to, during, and after an accident in a user’s private or work life.

[0047] For example, as illustrated in FIGS. 6-7, a user wearing a near-eye display 102 of augmented reality7display sy stem 100 could witness a workplace accident. As shown in FIG. 6, which illustrates a partial schematic view of from the perspective of a user wearing near-eye display 102, the user may witness a co-worker operating a forklift to move items between locations on a job site or in warehouse. As shown in FIG. 7, which illustrates a partial schematic view of what a user wearing near-eye display 102 would see moments later when an accident occurs, the user may witness the accident and wish to record it. In these circumstances, if the augmented reality display system 100 was in a recording mode RM, e.g., a default recording mode DRM or a numerically driven recording mode NDM, the user could simply provide a user input 122 (e.g., a button press or verbal command) and generate a recording of wftat he / she witnessed over the last five or ten minutes. This recording could be useful for insurance reporting purposes or for conducting workplace safety audits or training.

[0048] In some examples, the function(s) of the user input 122 described above, could be replaced by an external command 142 received by the near-eye display 102 from a source external to the near-eye display 102. For example, in a workplace environment, it may be desirable for a supervisor or foreman to be able to remotely trigger the recording 128 of a workplace accident or event. As such, the supervisor would be able to, using a remote device (e.g.. peripheral device 104 or other remote device), generate a wireless external command 142 that creates a timestamp 132 within the FIFO memory device 130 of a given near-eye display 102 and triggers recording process. It should be appreciated that the external command 142 can carry with it all of the information needed to generate a recording, e.g., it can cany’ with it instructions to create a timestamp 132 within a given set of data, what the value of the increment of time 134 is and whether the increment of time 134 should be applied before, after, or before and after the timestamp 132 to establish the temporal parameters of the recording 128. In some examples, the augmented reality display system 100 may include multiple near-eye displays 102, for example, where multiple workers are using near-eye displays 102 within a single environment such as but not limited to warehousing applications. In these examples, it may be beneficial to obtain recordings of a single event from multiple perspectives. As such the supervisor or foreman could send external command 142 to all near-eye devices 102 within a given range or area triggering each device to create a discrete recording 128 in accordance with the principles described herein.

[0049] Utilization of a device or system that continuously records, e g., as described in association with certain exemplary embodiments of the present invention, may raise concerns about use of a person’s likeness and / or the privacy of individuals whose likenesses are captured by the recording 128. As such the present disclosure includes several methods and / or devices that minimize and / or eliminate privacy concerns around the use of a device that constantly records the environment around a user. As such, in some examples, and as mentioned above with reference to FIG. 1, augmented reality display system 100 can also include a privacy proximity device 108.

[0050] As described above, privacy proximity device 108 is intended to be an object or collection of objects that is / are capable of transmitting one or more wireless signals, e g., privacy signal 144 to indicate proximity and / or location of a person or place that does not wish to be, or should not be, the subject of a data recording, e.g., recording 128. As shown in FIG. 1, privacy proximity' device 108 may take the form of a key fob. key chain, puck, or other device capable of at least sending the privacy signal 144 described herein. Privacy proximity device 108 may include its own power source or is capable of connecting to and utilizing an external power source. Privacy signal 144 is intended to be a wireless signal that utilizes one or more wireless data protocols, e.g., a Bluetooth data protocol, to periodically or continuously broadcast wireless data within a signalradius 146 (described below). In some examples, the privacy signal 144 is an advertising packet that can include one or more pieces of data that when received by the near-eye display 102 are recognized as a privacy signal 144. As such the communications module 118 of the near-eye display 102 is configured to receive broadcast advertising packet. In some examples, the privacy signal 144 is a Bluetooth or Bluetooth Low Energy advertising packet that includes a custom header or other custom component. The custom header may include a specific alphanumerical sequence that when received by the near-eye display 120 (e.g., via the communications module 118) is determined to be a privacy signal 144. If near-eye display 102 is in a recording mode RM when it receives a privacy signal 144, e.g., if near-eye display 102 is in a default recording mode DRM or a numerically driven recording mode NDM, the near-eye display 102 is configured to automatically exit the recording mode so long as it continues to receive the privacy signal 144. Privacy proximity7device 108 is configured to broadcast privacy signal 144 for a certain duration and on a certain cadence, e.g., it may broadcast the privacy signal 144 for one second every7ten seconds.

[0051] FIG. 8, illustrates a top-plan schematic view of a crosswalk showing a user wearing near- eye display 102 and another user (data subject S) holding a privacy proximity device 108. During operation, and as shown, the privacy proximity device 108 will periodically (e.g., for one second every ten seconds) broadcast the privacy signal 144 (e.g., a Bluetooth advertising packet with custom header) to a broadcast radius or radius of detection 146. In some examples the radius of detection 146 can be 100 m. 50 m, 25 m, 20 m. 15 m, 10 m. 5 m, 1 m, or any distance between any7of the foregoing distances (e.g., in meters). As shown in FIG. 8, as the data subject S continues toward the stationary user wearing the near-eye display 102 (positioned on the sidewalk), the radius of detection 146 also continues toward the stationary7user. Once the near-eye display 102 of the stationary user receives the privacy signal 144 the near-eye display 102 is ejected out of a recording mode RM, e.g., a default recording mode DRM or a numerically driven recording mode NDM, is prevented from initializing a recording mode RM so long as near-eye display 102 continues to receive the privacy signal 144.

[0052] Privacy proximity7device 108 may be a portable device such as, but not limited to, a key fob or puck that the user can carry in their pocket or could be a stationary7device that sits in sensitive areas such that the radius of detection 146 encompasses the sensitive area, e.g., schools, sports arenas, public bathrooms, etc.

[0053] In some examples, as shown in FIG. 9. rather than kicking the near-eye display 102 out of a recording mode RM upon receipt of a privacy7signal 144, receipt of the privacy signal 144 operates to cause the near-eye display 102 to enter a privacy mode PM that uses known facialrecognition techniques to identify the faces of one or more individuals (e.g., data subject S) within the data stream, e.g., video data 126, and blurs, removes, or blacks-out (as illustrated with dotted lines in FIG. 9) at least a portion of the subject’s S body or face within the resulting recording 128. For example, FIG. 9 illustrates a forklift operator (data subject S) as shown in previous examples; however, should that forklift operator be in a location that has a privacy proximity device 108 present, or he / she is carrying a privacy proximity device 108 on their person, the neareye display 102 will receive the privacy signal 144 and enter into a privacy mode PM. When observed by the near-eye display 102 in the privacy mode PM, the forklift operator will be observed normally by the user, but the recording 128, when viewed later by the user, will illustrate the forklift operator as blurred out, in such a way that the identity of the forklift operator remains unknowable.

[0054] Rather than a wireless privacy signal 144 signal, augmented reality system 100 can use one or more real-time video processing techniques to recognize visible markings on the exterior or a building or on another w earable device that will cause near-eye display 102 to be ejected out of a recording mode RM. prevented from entering a recording mode RM, or cause near-eye device to enter into a privacy mode PM as described above. For example, signage on a building could include a bar code, quick response (QR) code, or other visual indicator that w hen observed by a camera of the near-eye display 102, operates to eject the near-eye display 102 out of a recording mode RM, prevent it from entering a recording mode RM. or cause it to enter into a privacy mode PM as described above.

[0055] In some example embodiments, rather than a puck or key fob, privacy proximity device 108 can be a peripheral device, e.g., peripheral device 104. In those examples, the peripheral device 104 is operable to send, continuously or in response to user input 122, the privacy signal 144. In these examples, the privacy signal functionality can be implemented by one or more applications that are installed and active on the peripheral device 104. As such, the application can pass the specific custom header information to a communications module or transceiver of the peripheral device 104 which can then be broadcast into the environment around the user. This application, if placed on multiple peripheral devices, could also be used to prevent devices other than near eye display 102 from recording audio or video data. For example, peripheral devices 104, e.g., smart phones, could download an application to their devices and run them. While running, the devices are configured to broadcast the privacy signal 144 in a surrounding radius, i.e., radius of detection 146. Should any other device running the application within the radius of detection 146 receive the privacy signal, the other device will be prevented from recording orejected out of an ongoing recording so long as the device continues to receive the privacy signal 144.

[0056] FIG. 10 illustrates steps of an exemplary method 200 according to the present disclosure. As shown, method 200 includes: recording, via one or more sensors 116 of a near-eye display 102 audio data 124 and / or video data 126 (step 202); receiving a user input 122 (step 204); creating and storing a timestamp 132 associated with the audio data 124 and / or the video data 126 that temporally corresponds with the receipt of the user input 122 (step 206); and storing, as a discrete file, at least one recording 128 that includes the audio data 124 and / or the video data recorded between an increment of time prior to the timestamp 136 and the timestamp 132 (step 208A); or optionally, storing, as a discrete file, at least one recording 128 that includes the audio data 124 and / or the video data recorded between timestamp 132 and an increment of time after the timestamp 138 (step 208B).

[0057] One or more features of the embodiments described herein may be combined to create additional embodiments which are not depicted. While various embodiments have been described in detail above, it should be understood that they have been presented by w ay of example, and not limitation. It will be apparent to persons skilled in the relevant arts that the disclosed subject matter may be embodied in other specific forms, variations, and modifications without departing from the scope, spirit, or essential characteristics thereof. The embodiments described above are therefore to be considered in all respects as illustrative, and not restrictive. The scope of the invention is indicated by the appended claims, and all changes that come within the meaning and range of equivalents thereof are intended to be embraced therein.

Claims

CLAIMSWhat is claimed is:

1. An augmented reality display system, comprising: a near-eye display comprising: one or more sensors arranged to capture audio data and / or video data; a processor; and a non-transitory computer-readable memory arranged to store a set of non- transitory computer-readable instructions that when executed by the processor are configured to: receive the audio data and / or the video data from the one or more sensors; receive a user input; generate and store a timestamp associated with the audio data and / or the video data that temporally corresponds with the receipt of the user input; and save at least one recording that includes the audio data and / or the video data received between an increment of time before the timestamp and the timestamp; or save at least one recording that includes the audio data and / or the video data received between the timestamp and an increment of time after the timestamp.

2. The augmented reality display system of claim 1, wherein the audio data and / or the video data are stored in a first-in-first-out (FIFO) memory device.

3. The augmented reality display system of claim 2, wherein the FIFO memory device is configured to store at least thirty minutes of the audio data and / or the video data.

4. The augmented reality display system of claim 1, wherein the increment of time is a fixed integer of time selected from one of: one minute, two minutes, three minutes, four minutes, five minutes, ten minutes, fifteen minutes, twenty minutes, twenty-five minutes, or thirty7minutes.

5. The augmented reality display system of claim 1, wherein the user input comprises a numerical value and the increment of time before the timestamp and / or the increment of time after the timestamp are derived from the numerical value.

6. The augmented reality display system of claim 1, wherein the processor is configured to receive a further user input and the at least one recording includes the audio data and / or the video data received between the increment of time before the timestamp and the receipt of the further input from the user.

7. The augmented reality display system of claim 1. wherein the at least one recording includes the audio data and / or the video data received between the increment of time before the timestamp and an increment of time after the timestamp.

8. The augmented reality display system of claim 7, wherein the increment of time before the timestamp and the increment of time after the timestamp are both equal to a numerical value derived from the first input of the user.

9. The augmented reality display system of claim 7, wherein the user input is a sensor signal generated by the one or more sensors, and upon receiving the sensor signal, the increment of time before the timestamp and / or the increment of time after the timestamp is a preset value selected from one of: one minute, two minutes, three minutes, four minutes, five minutes, ten minutes, fifteen minutes, twenty minutes, twenty -five minutes, or thirty minutes.

10. The augmented reality display system of claim 1, wherein the one or more sensors are selected from: a camera, a microphone, a gyroscope, an accelerometer, a magnetometer, a physiological sensor, a Global Positioning System (GPS) sensor, a touch screen, a touch- capacitive sensor, a button, or a switch.

11. The augmented reality display system of claim 1, wherein the user input is a gesture or a voice command.

12. The augmented reality display system of claim 1, further comprising a peripheral device and wherein the near-eye display system further comprises a communications module configured to establish one or more wired or wireless data connections with the peripheral device.

13. The augmented reality display system of claim 1, further comprising a peripheral device, wherein the near-eye display system is configured to transmit the audio data and / or the video data to the peripheral device and the peripheral device is configured to store the transmitted audio data and / or the transmitted video data.

14. The augmented reality display system of claim 1, wherein the user input is a wireless external command received from a peripheral device.

15. The augmented reality display system of claim 14, further comprising at least one additional near-eye display and wherein the near-eye display and the at least one additional neareye display are configured to receive the wireless external command and save at least one respective recording that includes the audio data and / or the video data recorded by the near-eye display and the additional near eye-display.

16. The augmented reality display system of claim 1, wherein the at least one recording includes a first recording including the audio data and / or the video data received between the increment of time before the timestamp and the timestamp, and a second recording including the audio data and / or the video data obtained between the timestamp and an increment of time after the timestamp, wherein the processor is further configured to concatenate the first recording and the second recording into a single combined recording.

17. The augmented reality display system of claim 1, further comprising a privacy proximity’ device configured to broadcast a wireless privacy signal that when received by a communications module of the near-eye display, disables the one or more sensors.

18. The augmented reality display system of claim 16, wherein the wireless privacy signal is a Bluetooth advertising packet.

19. The augmented reality display system of claim 1, further comprising a server or a peripheral device in communication with the near-eye display, the server, peripheral device, or near-eye display being configured to receive the audio data and / or the video data related to the at least one recording and use one or more image or audio processing techniques to identify one or more persons from within the audio data and / or the video data, and obscure, blur, or remove at least a portion of the audio data and / or the video data associated with the one or more persons.

20. A method of recording audio or video data using a near-eye display, comprising: recording, via one or more sensors of the near-eye display audio data and / or video data; receiving a user input; creating and storing a timestamp associated with the audio data and / or video data that temporally corresponds with the receipt of the user input; and storing, as a discrete file, at least one recording that includes the audio data and / or video data recorded between an increment of time before the timestamp and the timestamp; or storing, as a discrete file, at least one recording that includes the audio data and / or video data recorded between the timestamp and an increment of time after the timestamp.

Images