Repetitive memory mapping operations in smart lenses / augmented glasses

Abstract

A method, computer system, and computer program product for memory mapping are provided. The method may include identifying an augmented reality device and at least one Internet of Things (IoT) device observing at least one biometric parameter. The method may include defining at least one user attention pattern based on the at least one biometric parameter. The method may include predicting the user's attention based on the at least one attention pattern. The method may include recording data from the augmented reality device based on the user's attention dropping below a certain point. The method may include storing the recorded data.

Description

[Technical Field]

[0001] The present invention relates generally to the field of computing, and more particularly to vision-based technologies. [Background technology]

[0002] Smart contact lenses may have the ability to capture video footage of the user's surroundings. For example, based on a particular eye event, the smart contact lenses can begin capturing and storing video footage. At the same time, the user may be distracted and / or generally inattentive to their surroundings (e.g., stressed, absent-minded, etc.). This may mean that the user may not recall certain events despite looking at their surroundings (e.g., they may not recall what they saw and / or witnessed, including, but not limited to, moments of sharp focus such as an introduction, moments when an unusual event occurred, moments when they were interrupted, or moments when text appeared on an online communication or media screen that was too long to remember after reading it, or combinations thereof). Summary of the Invention

[0003] Embodiments of the present invention disclose a method, computer system, and computer program product for memory mapping. The method may include identifying an augmented reality device and at least one Internet of Things (IoT) device observing at least one biometric parameter. The method may include defining at least one user attention pattern based on the at least one biometric parameter. The method may include predicting the user's attention based on the at least one attention pattern. The method may include recording data from the augmented reality device based on the user's attention falling below a certain point. The method may include storing the recorded data.

[0004] Viewed from a first aspect, the present invention provides a method for memory mapping, the method comprising: identifying an augmented reality device and at least one Internet of Things (IoT) device observing at least one biometric parameter; defining at least one attention pattern of a user based on the at least one biometric parameter; predicting an attention level of the user based on the at least one attention pattern; recording data from the augmented reality device based on the attention level of the user falling below a certain point; and storing the recorded data.

[0005] Preferably, the present invention provides a method wherein said augmented reality device is selected from the group consisting of smart contact lenses, smart glasses, and head mounted displays.

[0006] Preferably, the present invention provides a method wherein predicting the user's attention based on the at least one attention pattern further comprises deploying a long short-term memory (LSTM) recurrent neural network (RNN) model to predict the user's attention pattern via an autoencoder.

[0007] Preferably, the present invention provides a method further comprising using principal component analysis (PCA) with a long short-term memory (LSTM) recurrent neural network (RNN) model to determine the user's surrounding context based on at least one biometric parameter.

[0008] Preferably, the present invention provides a method wherein predicting the attentiveness of the user based on the at least one attention pattern further comprises determining that the user is inattentive and activating the augmented reality devices of nearby users.

[0009] Preferably, the present invention provides a method wherein recording data from the augmented reality device further comprises calculating a deviation of the user's attention from a baseline profile of the user based on the user's attention falling below the particular point. Preferably, the present invention provides a method wherein storing the recorded data further comprises storing the recorded data in a connected database and retraining a long short-term memory (LSTM) recurrent neural network (RNN) model.

[0010] Viewed from a second aspect, the present invention provides a computer system for memory mapping, the computer system including one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more tangible storage media to be executed by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of executing a method including identifying an augmented reality device and at least one Internet of Things (IoT) device observing at least one biometric parameter; defining at least one attention pattern of a user based on the at least one biometric parameter; predicting an attentional intensity of the user based on the at least one attentional pattern; recording data from the augmented reality device based on the attentiveness of the user falling below a certain point; and storing the recorded data.

[0011] Preferably, the present invention provides a computer system wherein the augmented reality device is selected from the group consisting of smart contact lenses, smart glasses, and a head-mounted display.

[0012] Preferably, the present invention provides a computer system wherein predicting the user's attention based on the at least one attention pattern further comprises deploying a long short-term memory (LSTM) recurrent neural network (RNN) model to predict the user's attention pattern via an autoencoder.

[0013] Preferably, the present invention provides a computer system further comprising using principal component analysis (PCA) with a long short-term memory (LSTM) recurrent neural network (RNN) model to determine a user's surrounding context based on at least one biometric parameter.

[0014] Preferably, the present invention provides a computer system wherein predicting the attentiveness of the user based on the at least one attention pattern further comprises determining that the user is inattentive and activating the augmented reality devices of nearby users.

[0015] Preferably, the present invention provides a computer system wherein recording data from the augmented reality device further comprises calculating a deviation of the user's attention from a baseline profile of the user based on the user's attention falling below the particular point.

[0016] Preferably, the present invention provides a computer system wherein storing the recorded data further comprises storing the recorded data in a connected database and retraining a long short-term memory (LSTM) recurrent neural network (RNN) model.

[0017] Viewed from another aspect, the present invention provides a computer program product for memory mapping, the computer program product including: one or more non-transitory computer-readable storage media; and program instructions stored on at least one of the one or more tangible storage media, the program instructions being executable by a processor to cause the processor to perform a method including identifying an augmented reality device and at least one Internet of Things (IoT) device observing at least one biometric parameter; defining at least one attention pattern of a user based on the at least one biometric parameter; predicting an attentiveness of the user based on the at least one attention pattern; recording data from the augmented reality device based on the attentiveness of the user dropping below a certain point; and storing the recorded data.

[0018] Preferably, the present invention provides a computer program product wherein said augmented reality device is selected from the group consisting of smart contact lenses, smart glasses, and head mounted displays.

[0019] Preferably, the present invention provides a computer program product, wherein predicting the user's attention based on the at least one attention pattern further comprises deploying a long short-term memory (LSTM) recurrent neural network (RNN) model to predict the user's attention pattern via an autoencoder.

[0020] Preferably, the present invention provides a computer program product further comprising using principal component analysis (PCA) with a long short-term memory (LSTM) recurrent neural network (RNN) model to determine a user's surrounding context based on at least one biometric parameter.

[0021] Preferably, the present invention provides a computer program product, wherein predicting the attentiveness of the user based on the at least one attention pattern further comprises determining that the user is inattentive, and activating the augmented reality devices of nearby users.

[0022] Preferably, the present invention provides a computer program product wherein recording data from the augmented reality device further comprises calculating a deviation of the user's attention from a baseline profile of the user based on the user's attention falling below the particular point.

[0023] Preferably, the present invention provides a method for memory mapping, the method comprising: predicting a user's inattention; identifying a nearby user; and recording a plurality of data observed by the nearby user's augmented reality device based on the predicted user's inattention.

[0024] Preferably, the present invention provides a method, wherein predicting the inattention of the user further comprises deploying a long short-term memory (LSTM) recurrent neural network (RNN) model to predict the attention pattern of the user via an autoencoder, wherein principal component analysis (PCA) is deployed together with the long short-term memory (LSTM) recurrent neural network (RNN) model to determine the user's ambient context based on at least one biometric parameter.

[0025] Preferably, the present invention provides a method for triggering recording by an augmented reality device, the method comprising determining, by a machine learning algorithm, that the augmented reality device should start recording based on predicting inattention of a user of the augmented reality device, wherein the predicted inattention of the user of the augmented reality device is based on an eye event.

[0026] Preferably, the present invention provides a method wherein said augmented reality device is a smart contact lens.

[0027] Preferably, the present invention provides a method, wherein the machine learning algorithm is a long short-term memory (LSTM) recurrent neural network (RNN) model using principal component analysis (PCA).

[0028] These and other objects, features, and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments thereof, which should be read in conjunction with the accompanying drawings, in which various features are not drawn to scale for clarity, and which, together with the detailed description, are intended to facilitate understanding of the invention by those skilled in the art. [Brief explanation of the drawings]

[0029] [Figure 1] FIG. 1 illustrates a networked computer environment according to at least one embodiment. [Figure 2] 1 is an operational flowchart illustrating a process for memory mapping, according to at least one embodiment. [Figure 3] FIG. 2 is a block diagram of internal and external components of the computer and server depicted in FIG. 1 according to at least one embodiment. [Figure 4] FIG. 2 is a block diagram of an exemplary cloud computing environment including the computer system depicted in FIG. 1, according to one embodiment of the present disclosure. [Figure 5] FIG. 5 is a block diagram of functional layers of the exemplary cloud computing environment of FIG. 4, according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0030] Detailed embodiments of the claimed structures and methods are disclosed herein. However, it should be understood that the disclosed embodiments are merely exemplary of the claimed structures and methods, which may be embodied in various forms. However, the present invention may be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0031] The present invention may be a system, a method, or a computer program product, or a combination thereof, at any level of technical detail that allows integration. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0032] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, or the like, and suitable combinations thereof. Computer-readable storage devices, as used herein, should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0033] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). The network may be comprised of copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0034] Computer-readable program instructions for carrying out operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, as a standalone software package, or partially on the user's computer. Alternatively, the computer may be executed partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the computer-readable program instructions in order to carry out aspects of the present invention.

[0035] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0036] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0037] These computer-readable program instructions may also be stored in a computer-readable storage medium connectable to a computer, programmable data processing apparatus, or other device or combination thereof that functions in a particular way, such that the computer-readable program instructions stored therein configure one of a number of products that include instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0038] Computer-readable program instructions, such as instructions to perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams on a computer, other programmable apparatus, or other device, can also be loaded into a computer, other programmable apparatus, or other device to perform a series of operational steps on the computer, other programmable apparatus, or other device to generate a computer-implemented process.

[0039] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of executable implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which constitute one or more executable instructions for implementing the specified logical function(s). In some alternative embodiments, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0040] The exemplary embodiments described below provide systems, methods, and program products for memory mapping. As such, the present embodiments have the ability to improve the field of augmented reality devices by recording and storing visual data collected by the augmented reality device (e.g., smart contact lenses, smart glasses, or head-mounted displays) and continuously pruning the collected data to determine information that may be deemed valuable to a user. More specifically, the present invention may include identifying an augmented reality device and at least one Internet of Things (IoT) device observing at least one biometric parameter. The present invention may include defining at least one user attention pattern based on the at least one biometric parameter. The present invention may include predicting a user's attention based on the at least one attention pattern. The present invention may include recording data from the augmented reality device based on the user's attention dropping below a certain point. The present invention may include storing the recorded data.

[0041] As previously described, smart contact lenses may have the ability to capture video footage of a user's surroundings. For example, based on a particular eye event (e.g., looking up, looking down, looking left, looking right, rolling the eyes, blinking in a particular sequence, closing the eyes, opening the eyes wide, other eye events, etc.), the smart contact lenses can begin capturing and storing video footage. At the same time, the user may be distracted or generally inattentive to their surroundings, or both (e.g., stressed, absent-minded, etc.). This may mean that despite looking at their surroundings, the user may not recall certain events (e.g., they may not recall what they saw and / or witnessed, including, but not limited to, moments of sharp focus such as an introduction, moments when an unusual event occurred, moments when they were interrupted, or moments when text appeared on an online communication or media screen that was too long to remember after reading it, or combinations thereof).

[0042] It may therefore be advantageous to address such memory-constrained scenarios, particularly by providing means capable of detecting a user's short-term memory capabilities and / or forgetfulness patterns, and accordingly, smart contact lenses and / or augmented glasses can automatically capture video depicting the user's surroundings.

[0043] According to at least one embodiment, the smart storage manager may be embedded in a smart lens (i.e., smart contact lens), smart glasses (i.e., augmented glasses, augmented reality glasses), a head-mounted display, or another augmented reality gadget, or a combination thereof. The smart storage manager may employ configurable learning tools to teach levels of selectively and dynamically storing, labeling, or indexing, or a combination of, verbal, textual, and / or visual data, including any attributes related to the user's footage that may be used for future reference.

[0044] According to at least one embodiment, the smart storage management mechanism may include continuous pruning of selective features using iterative training, which may take into account user preferences (e.g., as configured programmatically) and the user's learned behavior to determine the level of information that may be deemed valuable to the user and desired for subsequent use and / or reference.

[0045] According to at least one embodiment, the smart storage manager may trigger the recording function of a device for groups of users, not just a single user, and may use machine learning to learn when to trigger the device to record.

[0046] According to at least one embodiment, the smart storage management mechanism can support retention of temporal events, including, but not limited to, events based on geographic proximity and user location forgetfulness, time of day (e.g., when the user is overly tired), or scenarios where there is a large number of activities warranting the user's attention, or combinations thereof.

[0047] Referring to FIG. 1, an exemplary networked computing environment 100 is depicted, according to one embodiment. The networked computing environment 100 may include a computer 102 having a processor 104 and a data storage device 106 capable of executing a software program 108 and a memory mapping program 110a. The networked computing environment 100 may include a server 112 capable of executing a memory mapping program 110b, which may interact with a database 114 and a communications network 116. The networked computing environment 100 may include multiple computers 102 and servers 112, only one of which is shown. The communications network 116 may include various types of communications networks, such as a wide area network (WAN), a local area network (LAN), a telecommunications network, a wireless network, a public switched network, or a satellite network, or combinations thereof. An augmented reality (AR) device 118 and an Internet of Things (IoT) device 120 are depicted as their own separate entities, but may be integrated into other parts of the computing environment. It should be appreciated that Figure 1 is only intended to provide an illustration of one implementation and is not intended to suggest any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

[0048] The client computer 102 may communicate with the server computer 112 via a communications network 116. The communications network 116 may include connections such as wired, wireless communication links, or fiber optic cables. As described with reference to FIG. 3 , the server computer 112 may include internal components 902a and external components 904a, respectively, and the client computer 102 may include internal components 902b and external components 904b, respectively. The server computer 112 may also operate in a cloud computing service model such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server 112 may also be located in a cloud computing deployment model such as a private cloud, a community cloud, a public cloud, or a hybrid cloud. The client computer 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running programs, accessing a network, and accessing the database 114. According to various implementations of this embodiment, the memory mapping programs 110a, 110b may interact with a database 114 that may be embedded in various storage devices, such as, but not limited to, the computer / mobile device 102, a networked server 112, or a cloud storage service.

[0049] According to this embodiment, a user using client computer 102 or server computer 112 can use memory mapping programs 110a, 110b (respectively) to record and store visual data collected by an augmented reality device (e.g., smart contact lenses, smart glasses, or a head-mounted display) and continuously prune the collected data to determine information that may be deemed valuable to the user. The memory mapping method is described in more detail below with respect to FIG. 2.

[0050] Referring now to FIG. 2, an operational flowchart illustrating an exemplary memory mapping process 200 used by memory mapping programs 110a and 110b is depicted, according to at least one embodiment. At 202, an augmented reality (AR) device is connected to at least one mobile or Internet of Things (IoT) device, or both. Smart contact lenses, smart glasses, or another augmented reality (AR) device including an embedded video camera, or a combination thereof, may be connected to the at least one mobile or Internet of Things (IoT) device using a communications network (e.g., communications network 116) so that a user's predicted level of attention and ability to recall the user's surroundings at a later date can be determined and recordings can be made.

[0051] The IoT can be a system of interrelated computing devices, machines, and digital machines with embedded sensors that can transmit data over the Internet without the need for human-to-human or human-to-computer interaction. Connected IoT devices can be embedded in mobile devices, industrial equipment, or environmental sensors, or a combination thereof, among many others, including, but not limited to, watches, automobiles, thermostats, voice assistants, etc.

[0052] The AR device can be connected to the IoT device via Bluetooth® (Bluetooth and all Bluetooth-based trademarks and logos are trademarks or registered trademarks of Bluetooth SIG, Inc. and / or its affiliates), WiFi, wireless telegraphy, or other communication methods that may allow data (e.g., signals, video streams) to be transmitted between the AR device and the IoT device, or a combination thereof.

[0053] At 204, the user's attention patterns are defined. A knowledge corpus (e.g., a cloud database and / or database 114) may be created to store the user's attention patterns (e.g., identified correlations between the user's biometric parameters, the user's attentiveness to their surroundings, and the ability to recall particular scenarios).

[0054] The memory mapping programs 110a, 110b may deploy a long-short-term memory (LSTM) recurrent neural network (RNN), which can be used to predict a user's attention patterns via an autoencoder. The LSTM-RNN model may consider multivariate time series data (e.g., a series of chronologically indexed data points, possibly acquired at successive equally spaced intervals or time points). The time series data may be time series data about a user and the activities (if any) performed by the user. The time series data may also be collected from a connected augmented reality (AR) device (e.g., smart contact lenses, smart glasses, a head-mounted display, or another augmented reality gadget, or a combination thereof).

[0055] For example, the time series data may include a user gazing in a particular direction, or walking or driving in a particular direction, or both. The time series data may include categorical feature variables (e.g., gaze or walking, among many others). These categorical feature variables may be converted into numerical feature vectors by encoding (e.g., using label encoding or one-hot encoding, among other encoding methods, in which the categorical feature variables may be converted into a numerical form usable by a machine learning algorithm). The numerical feature vectors may be fed into an LSTM-RNN architecture.

[0056] The numerical feature vector may enable the LSTM-RNN model to learn the user's probabilistic activity, if any, and perform attention detection in conjunction with an autoencoder (e.g., an encoder-decoder architecture of a recurrent neural network may be used for sequence-to-sequence prediction problems). Attention detection may be a mechanism and / or layer in a deep learning model that addresses the limitations of the encoder-decoder architecture for long sequences of data and may use the skills of the LSTM-RNN model for sequence-to-sequence prediction problems.

[0057] An autoencoder may be an unsupervised learning technique that can take an image as input (e.g., from a connected camera or video feed, including one connected to an augmented reality device), compress the image into a latent space representation, and reconstruct an output image from the latent space representation. An autoencoder may consist of a bottleneck, which may include an input representation of data (e.g., data x may have an f(x) input representation). A decoder in the autoencoder may then generate a reconstruction of the input data using the encoding of the bottleneck (e.g., the reconstruction of input data x is r = g(f(x))). An autoencoder may learn about any regularities in the input data (e.g., video data).

[0058] Historical parameters may be collected by the user's various devices (e.g., augmented reality devices or IoT devices, or both) and may be taken into account to predict the user's attention patterns. The historical parameters may include data collected about the user's attention level and the user's ability to recall scenarios (including whether the user's recollection was made while the user was in a poor cognitive state (e.g., sad, stressed, or health issues, or a combination thereof, etc.)). The historical parameters may be collected by connected IoT devices and stored in a knowledge corpus (e.g., a cloud database or database 114, or a combination thereof), as described above with respect to step 202 above.

[0059] Principal component analysis (PCA) can also be used with LSTM-RNN models to determine the context around the user based on real-time biometric parameters (e.g., captured via a connected IoT wearable device), the rate of change of the user's biometric parameters, the contextual needs of the user's surroundings, any written text (e.g., phone number, license plate number, information written on a business card, business hours, address, or street name, or a combination thereof), or introductions when someone speaks the user's name, or a combination thereof, or other instances that may be captured by the AR device. PCA is a dimensionality reduction technique used to reduce the dimensionality of a dataset by constructing principal components. A PCA plot can depict clusters together based on similarity.

[0060] When the number of parameters is highly complex and the LSTM-RNN model takes in a large number of inputs, PCA may be used to understand the relationships between the input variables and / or to perform dimensionality reduction (e.g., to reduce the number of input features or feature vectors used in the algorithm).

[0061] The user's current attention level is predicted at 206. The LSTM-RNN model with PCA used for feature analysis and pruning (described above with respect to step 204) may be taken into account in predicting in which cognitive state the user may miss or be unable to remember certain details related to the user's surroundings.

[0062] A user's cognitive heuristics (e.g., a user's way of processing information, including thinking about problems rationally, logically, intentionally, linguistically, effortlessly, emotionally, or intuitively, or a combination thereof) may be used to identify a user's "norm," and anomalies in a user's behavior, "delta" (e.g., based on the use of particular words, loud sounds, or sudden changes in acceleration, or a combination thereof), may be used to identify a user's attentional abilities at a given time.

[0063] According to at least one embodiment, a user's attentiveness may be measured on a relative scale (e.g., a measured low attentiveness has an attentiveness score close to 0, and a measured high attentiveness has an attentiveness score close to 100). The user's augmented reality (AR) device may begin recording when the user's attentiveness falls below a predefined threshold and / or deviates from the user's behavior by a delta (e.g., below a certain point, such as 50%, calculated on the relative scale).

[0064] For example, a baseline profile of the user may be created (e.g., using the learning mechanism described above with respect to step 204 above), deviations from the user's baseline profile may be identified, and the memory mapping programs 110a, 110b may further identify the user as being attentive or inattentive.

[0065] Certain eye events (e.g., looking up, looking down, looking left, looking right, rolling your eyes, blinking in a particular sequence, closing your eyes, opening your eyes wide, other eye events, etc.) can also trigger the smart contact lenses to begin capturing and storing video footage. By projecting information directly onto the user's retina, an augmented reality (AR) device (e.g., the user's smart contact lenses, smart glasses, or other augmented reality (AR) device, or a combination thereof) may be able to capture gesture-based eye movements to begin capturing and storing video footage.

[0066] A user can search for a number of past events (e.g., E1, E2, . . . , E n Feature pruning (e.g., recursive event pruning, used to reduce complexity in machine learning algorithms) may also be used when trying to recall specific situations and / or events from events (e.g., E m Even if a user has difficulty recalling a past event, they may be able to recall other events. PCA analysis and other feature frame pruning mechanisms can be used to establish the user's context, including developing a pruning strategy for the user (e.g., identifying instances in which the user has difficulty recalling past events). Recurring event pruning may help identify the user's deviations from the baseline profile by establishing a baseline profile for the user and preserving data (e.g., photographs or video data, or both) that may be related to events not previously recalled by the user captured by the user's smart contact lenses, smart glasses, or other augmented reality (AR) device, or a combination thereof.

[0067] At 208, the memory mapping program 110a, 110b determines whether to start recording with the augmented reality (AR) device. For example, based on a particular eye event (e.g., looking up, looking down, looking left, looking right, rolling the eyes, blinking in a particular sequence, closing the eyes, opening the eyes wide, other eye events, etc.), or based on the user's biometric parameters, or both, the smart contact lenses can start capturing and storing video footage.

[0068] The memory model may be placed in parallel with an assist processor and / or a graphics processing unit (GPU) that performs the PCA analysis (as described above with respect to steps 204 and 206 above) of the user's various heuristics.

[0069] For example, if a user is found to be distracted and based on historical data (e.g., including the user's level of attention in previous similar situations), the memory mapping program 110a, 110b may determine that the user does not recall details of what they saw and determine that an augmented reality (AR) device should begin recording. In response to this determination, the user's various devices, including any connected augmented reality (AR) devices and / or IoT devices, may continue to track the user's level of attention and capture the user's surroundings.

[0070] Furthermore, according to at least one embodiment of the present invention, if a user's computing ecosystem connected to the memory mapping program 110a, 110b (e.g., a combination of feeds from the user's mobile device, wearable device, smart contact lens, or IoT sensors, among others) identifies that the user is not attentive or may not recall an event based on the event's similarity to past events that the user did not recall, and at the same time, if the user's focus direction deviates from the target surroundings, the memory mapping program 110a, 110b can interact with at least one other user's (i.e., nearby) paired smart contact lens and / or augmented reality device (e.g., augmented glasses, augmented reality eyeglasses, etc.) to capture the user's surrounding environment.

[0071] The memory mapping programs 110a, 110b may be able to instantly identify and connect to the augmented reality devices of nearby users (e.g., nearby users who also use versions of the memory mapping programs 110a, 110b). Any data captured by the nearby users' augmented reality devices may be stored in a database connected to the users' memory mapping programs 110a, 110b.

[0072] For example, if two friends are traveling together, if one friend is agitated and not paying much attention (e.g., not concentrating on famous places in the area), the memory mapping program 110a, 110b will identify this and activate sensors in the other friend's smart contact lenses and / or augmented reality device to capture the surroundings on behalf of the other friend.

[0073] If it is determined that recording should be started, the memory mapping programs 110a and 110b proceed to step 210 below.

[0074] If it is determined that recording should not be started, the memory mapping programs 110a and 110b are terminated.

[0075] At 210, recording is initiated. A content recording event may begin at time T, at which time the smart contact lenses and / or other augmented reality devices (e.g., augmented glasses, augmented reality eyeglasses, head-mounted displays, etc.) may automatically record video of the user's surroundings based on the determination in step 208 above that the augmented reality (AR) device should begin recording.

[0076] For example, in instances where a user is overly excited or scared, among other emotions that may consume the user's reactions and / or thoughts, video may be automatically captured by memory mapping programs 110a, 110b on smart contact lenses and / or other augmented reality (AR) devices. The user's emotions may also be determined by memory mapping programs 110a, 110b based on connected IoT devices, including but not limited to wearable devices, that may capture the user's biometric data (i.e., biometric parameters).

[0077] The recorded data is stored and the machine learning model is retrained at 212. A training period may be provided.

[0078] At 212, the recorded data is stored and the machine learning model is retrained. A training period may be provided by the memory mapping programs 110a, 110b, where a time series of events (e.g., a series of events E (E1, E2, ..., E n All data collected during time T) relative to time T) may be recorded and stored in a cloud database (e.g., database 114).

[0079] Retraining of the machine learning model may be as previously described with respect to step 204 above.

[0080] It will be understood that Figure 2 provides only an illustration of one embodiment and does not imply any limitations as to how different embodiments may be implemented. Many modifications to the depicted embodiment may be made based on design and implementation requirements.

[0081] Figure 3 is a block diagram 900 of the internal and external components of the computer depicted in Figure 1 in accordance with an exemplary embodiment of the present invention. It should be understood that Figure 3 provides only an illustration of one implementation and is not intended to suggest any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

[0082] Data processing systems 902, 904 are representative of any electronic device capable of executing machine-readable program instructions. Data processing systems 902, 904 may be representative of a smartphone, computer system, PDA, or other electronic device. Examples of computing systems, environments, or configurations, or combinations thereof, that may be represented by data processing systems 902, 904 include, but are not limited to, personal computer systems, server computer systems, thin client, thick client, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0083] The user client computer 102 and the network server 112 may include respective sets of internal components 902a, b and external components 904a, b illustrated in Figure 3. Each set of internal components 902a, b includes one or more processors 906, one or more computer-readable RAMs 908 and one or more computer-readable ROMs 910 on one or more buses 912, one or more operating systems 914 and one or more computer-readable tangible storage devices 916. The one or more operating systems 914, software programs 108, and memory mapping program 110a in the client computer 102 and the memory mapping program 110b in the network server 112 may be stored on one or more computer-readable tangible storage devices 916 for execution by the one or more processors 906 via one or more RAMs 908 (which typically include cache memory). In the embodiment shown in Figure 3, each of the computer-readable tangible storage devices 916 is a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices 916 is a semiconductor storage device such as a ROM 910, an EPROM, a flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0084] Each set of internal components 902a,b also includes a R / W drive or interface 918 for reading from and writing to one or more portable computer-readable tangible storage devices 920, such as CD-ROMs, DVDs, memory sticks, magnetic tapes, magnetic disks, optical disks, or semiconductor storage devices. Software programs, such as software program 108 and memory mapping programs 110a, 110b, can be stored on one or more of the respective portable computer-readable tangible storage devices 920 and can be read via the respective R / W drive or interface 918 and loaded onto the respective hard drive 916.

[0085] Each set of internal components 902a, b may also include a network adapter (or switch port card) or interface 922, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G or 4G wireless interface card, or other wired or wireless communication link. The software program 108 and memory mapping program 110a in the client computer 102 and the memory mapping program 110b in the network server computer 112 may be downloaded from an external computer (e.g., a server) via a network (e.g., the Internet, a local area network, or other wide area network) and their respective network adapters or interfaces 922. From the network adapters (or switch port adapters) or interfaces 922, the software program 108 and memory mapping program 110a in the client computer 102 and the memory mapping program 110b in the network server computer 112 are loaded onto their respective hard drives 916. The network may include copper wire, optical fiber, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof.

[0086] Each of the set of external components 904 a,b can include a computer display monitor 924, a keyboard 926, and a computer mouse 928. The external components 904 a,b can also include touch screens, virtual keyboards, touchpads, pointing devices, and other human interface devices. Each of the set of internal components 902 a,b also includes a device driver 930 for interfacing to the computer display monitor 924, the keyboard 926, and the computer mouse 928. The device driver 930, the R / W drive or interface 918, and the network adapter or interface 922 include hardware and software (stored in the storage device 916 or the ROM 910, or both).

[0087] Although this disclosure includes detailed descriptions of cloud computing, implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the invention can be practiced in conjunction with any other type of computing environment now known or later developed.

[0088] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0089] The characteristics are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without the need for human interaction with the service provider. Broad network access: Computing power is available over the network and can be accessed through standard mechanisms, facilitating use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, PDAs). Resource Pooling: Computing resources from a provider are pooled and offered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated based on demand. Consumers generally have no control or knowledge of the exact location of the resources they are provided with, resulting in a sense of location independence. However, consumers may be able to determine location at a higher level of abstraction (e.g., country, state, data center). Rapid Elasticity: Computing capacity can be provisioned quickly and elastically, sometimes automatically, to instantly scale out and quickly release to instantly scale in. To the consumer, the computing power available for provisioning often appears unlimited, and can be purchased at any time and in any quantity. Metered Services: Cloud systems leverage measurement capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts) to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of utilized services.

[0090] The service model is as follows: Software as a Service (SaaS): The functionality offered to the consumer is the availability of a provider's applications running on a cloud infrastructure that can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., webmail). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functionality, except for limited user-specific application configuration settings. Platform as a Service (PaaS): The capability offered to consumers is to deploy applications they create or acquire using programming languages ​​and tools supported by the provider onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of their hosting environment. Infrastructure as a Service (IaaS): The functionality offered to consumers is the provisioning of processors, storage, networking, and other basic computing resources on which they can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating system, storage, and deployed applications, and in some cases partial control over some network components (e.g., host firewalls).

[0091] The deployment model is as follows: Private Cloud: This cloud infrastructure is dedicated to a specific organization and can be managed by that organization or a third party, and can exist on-premise or off-premise. Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by those organizations or a third party and can exist on-premises or off-premises. Public cloud: This cloud infrastructure is available to the general public or large industry organizations and is owned by an organization that sells cloud services. Hybrid cloud: This cloud infrastructure combines two or more cloud models (private, community, or public), each of which retains its inherent nuances but is bound by standards or specific technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0092] A cloud computing environment is a service-oriented environment that emphasizes statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0093] Referring now to FIG. 4 , an exemplary cloud computing environment 1000 is depicted. As shown, the cloud computing environment 1000 includes one or more cloud computing nodes 100, to which local computing devices used by cloud consumers (e.g., PDAs or cell phones 1000A, desktop computers 1000B, laptop computers 1000C, or automobile computer systems 1000N, or combinations thereof) can communicate. The nodes 100 can communicate with each other. The nodes 100 can be physically or virtually grouped (not shown) in one or more networks, such as, for example, private, community, public, or hybrid clouds, or combinations thereof, as described above. This enables the cloud computing environment 1000 to provide infrastructure, platform, or software as a service, or combinations thereof, for which cloud consumers do not need to maintain resources on their local computing devices. It should be understood that the types of computing devices 1000A-N shown in FIG. 4 are merely exemplary, and that the computing node 100 and cloud computing environment 1000 can communicate with any type of electronic device via any type of network or network-addressable connection (e.g., using a web browser) or both.

[0094] Referring now to Figure 5, there is shown a series of functional abstraction layers 1100 provided by the cloud computing environment 1000. It should be understood in advance that the components, layers, and functions shown in Figure 5 are merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0095] Hardware and software layer 1102 includes hardware and software components. Examples of hardware components include mainframe 1104, reduced instruction set computer (RISC) architecture-based server 1106, server 1108, blade server 1110, storage device 1112, and network and network components 1114. In some embodiments, software components include network application server software 1116 and database software 1118.

[0096] The virtualization layer 1120 provides an abstraction layer from which the following virtual entities can be provided, for example: virtual servers 1122, virtual storage 1124, virtual networks including virtual private networks 1126, virtual applications and operating systems 1128, and virtual clients 1130.

[0097] By way of example, management layer 1132 may provide the following functionality: Resource provisioning 1134 enables dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 1136 enables cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. By way of example, these resources may include application software licenses. Security enables identification and verification of cloud consumers and tasks, as well as protection for data and other resources. User portal 1138 provides consumers and system administrators with access to the cloud computing environment. Service level management 1140 enables allocation and management of cloud computing resources so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 1142 enables advance arrangement and procurement of anticipated future cloud computing resources required according to SLAs.

[0098] The workload layer 1144 provides examples of functionality available to a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include mapping and navigation 1146, software development and lifecycle management 1148, virtual classroom instruction delivery 1150, data analytics processing 1152, transaction processing 1154, and memory mapping 1156. The memory mapping programs 110a, 110b record and store visual data collected by augmented reality devices (e.g., smart contact lenses, smart glasses) and provide users with a way to continually prune the collected data to determine information that may be deemed valuable.

[0099] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope of the disclosed embodiments. The terms used herein were selected to best explain the principles of the embodiments, practical applications or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. 1. A method for a processor to store data in a memory, the method comprising: the processor connecting the augmented reality device to at least one Internet of Things (IoT) device that observes at least one biometric parameter; the processor storing in a memory at least one attention pattern of the user based on the at least one biometric parameter; The processor predicts the user's attention based on the at least one user's attention pattern using a long short-term memory (LSTM) recurrent neural network (RNN) model; capturing data by the augmented reality device based on the processor having the user's attention falling below a certain point, wherein a baseline profile of the user is established based in part on determining a recurring event pruning and a user memory pruning strategy, the user memory pruning strategy being a strategy that identifies instances in which the user has difficulty recalling past events; the processor storing the captured data; and retraining the long short-term memory (LSTM) recurrent neural network (RNN) model using multivariate time series data points of the captured data, the long short-term memory (LSTM) recurrent neural network (RNN) model updating the user memory pruning strategy; A method comprising:

2. the augmented reality device Smart contact lenses, smart glasses, head-mounted displays, 2. The method of claim 1, wherein the compound is selected from the group consisting of:

3. The processor predicting the attentiveness of the user based on the at least one attention pattern of the user includes: The processor predicts the attention of the user via an autoencoder. The method of claim 1 further comprising:

4. the processor using principal component analysis (PCA) with the long short-term memory (LSTM) recurrent neural network (RNN) model to estimate a context of the user's surroundings based on the at least one biometric parameter. The method of claim 3 further comprising:

5. The processor predicting the attentiveness of the user based on the at least one attention pattern of the user includes: the processor determining that the user is inattentive; the processor activating the augmented reality devices of other users; The method of claim 1 further comprising:

6. The processor capturing data with the augmented reality device based on the user's attention falling below the particular point may include: the processor calculating a deviation of the user's attention from the baseline profile of the user. The method of claim 1 further comprising:

7. The processor storing the captured data includes: the processor storing the captured data in a connected database. The method of claim 1 further comprising:

8. 1. A computer system for storing data in a memory, comprising:

1. A computer system comprising: one or more processors; one or more computer-readable memories; one or more computer-readable tangible storage media; and program instructions stored on at least one of the one or more computer-readable tangible storage media to be executed by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of executing a method, the method comprising: connecting the augmented reality device to at least one Internet of Things (IoT) device that observes at least one biometric parameter; storing at least one user attention pattern in a memory based on the at least one biometric parameter; predicting a user's attentional ability based on the at least one user's attentional pattern using a long short-term memory (LSTM) recurrent neural network (RNN) model; and capturing data by the augmented reality device based on the user's attention dropping below a certain point, wherein a baseline profile of the user is established in part based on determining a recurring event pruning and a user memory pruning strategy, the user memory pruning strategy being a strategy that identifies instances in which the user has difficulty recalling past events; storing the captured data; and retraining the long short-term memory (LSTM) recurrent neural network (RNN) model using multivariate time series data points of the captured data, wherein the long short-term memory (LSTM) recurrent neural network (RNN) model updates the user memory pruning strategy; A computer system comprising:

9. the augmented reality device Smart contact lenses, smart glasses, head-mounted displays, 9. The computer system of claim 8, wherein the computer system is selected from the group consisting of:

10. predicting the attentiveness of the user based on the at least one attention pattern of the user, Predicting the attention of the user via an autoencoder The computer system of claim 8 further comprising:

11. using principal component analysis (PCA) with the long short-term memory (LSTM) recurrent neural network (RNN) model to estimate a user's surrounding context based on the at least one biometric parameter. The computer system of claim 10 further comprising:

12. predicting the attentiveness of the user based on the at least one attention pattern of the user, determining that the user is inattentive; activating the augmented reality device of another user; The computer system of claim 8 further comprising:

13. Capturing data by the augmented reality device based on the user's attention dropping below the particular point includes: Calculating a deviation of the user's attention from the baseline profile of the user. The computer system of claim 8 further comprising:

14. Storing the captured data includes: storing said captured data in a connected database; The computer system of claim 8 further comprising:

15. 1. A computer program for storing data in a memory, comprising: The program instructions include program instructions executable by a processor to cause the processor to perform a method, the method comprising: connecting the augmented reality device to at least one Internet of Things (IoT) device that observes at least one biometric parameter; storing at least one user attention pattern in a memory based on the at least one biometric parameter; predicting a user's attentional ability based on the at least one user's attentional pattern using a long short-term memory (LSTM) recurrent neural network (RNN) model; and capturing data by the augmented reality device based on the user's attention dropping below a certain point, wherein a baseline profile of the user is established in part based on determining a recurring event pruning and a user memory pruning strategy, the user memory pruning strategy being a strategy that identifies instances in which the user has difficulty recalling past events; storing the captured data; and retraining the long short-term memory (LSTM) recurrent neural network (RNN) model using multivariate time series data points of the captured data, wherein the long short-term memory (LSTM) recurrent neural network (RNN) model updates the user memory pruning strategy; A computer program comprising:

16. the augmented reality device Smart contact lenses, smart glasses, head-mounted displays, selected from the group consisting of 16. A computer program according to claim 15.

17. predicting the attentiveness of the user based on the at least one attention pattern of the user, Predicting the attention of the user via an autoencoder 16. The computer program of claim 15, further comprising:

18. using principal component analysis (PCA) with the long short-term memory (LSTM) recurrent neural network (RNN) model to estimate a user's surrounding context based on the at least one biometric parameter.

20. The computer program of claim 17, further comprising:

19. predicting the attentiveness of the user based on the at least one attention pattern of the user, determining that the user is inattentive; activating the augmented reality device of another user; 16. The computer program of claim 15, further comprising:

20. Capturing data by the augmented reality device based on the user's attention dropping below the particular point includes: Calculating a deviation of the user's attention from the baseline profile of the user.

16. The computer program of claim 15, further comprising:

21. 1. A method for a processor to store data in a memory, the method comprising: predicting user inattention with a long short-term memory (LSTM) recurrent neural network (RNN) model using a baseline user profile established based on determining a recurring event pruning and user memory pruning strategy, wherein the user memory pruning strategy is a strategy for identifying instances in which the user has difficulty recalling past events; the processor capturing data observed by other users' augmented reality devices based on the predicted inattention of the user; the processor storing the captured data; and retraining the long short-term memory (LSTM) recurrent neural network (RNN) model using multivariate time series data points of the captured data, the long short-term memory (LSTM) recurrent neural network (RNN) model updating the user memory pruning strategy; A method comprising:

22. The processor predicting the inattention of the user comprises: The processor predicts the inattention of the user via an autoencoder, wherein principal component analysis (PCA) is used together with the long short-term memory (LSTM) recurrent neural network (RNN) model to estimate the user's surrounding context based on at least one biometric parameter.

22. The method of claim 21 further comprising:

23. 1. A method for a processor to initiate capture by an augmented reality device, the method comprising: the processor connecting the augmented reality device to at least one Internet of Things (IoT) device that observes eye events, the eye events being selected from the group consisting of looking up, looking down, looking left, looking right, rolling the eyes, blinking in a particular sequence, closing the eyes, and opening the eyes wide; storing, in a memory, at least one user attention pattern based on the eye events; the processor predicting a user's attention based on the at least one user's attention pattern; the processor determining, via a machine learning algorithm, that the augmented reality device should begin capturing when the user's attention falls below a certain point; capturing data by the processor with the augmented reality device, wherein a baseline profile of the user is established in part based on determining a recurring event pruning and a user memory pruning strategy, the user memory pruning strategy being a strategy that identifies instances in which the user has difficulty recalling past events; A method comprising:

24. 24. The method of claim 23, wherein the augmented reality device is a smart contact lens.

25. 24. The method of claim 23, wherein the machine learning algorithm is a long short-term memory (LSTM) recurrent neural network (RNN) model with principal component analysis (PCA).

Images