Iterative memory mapping operations in smart lenses / augmented glasses

Abstract

Methods, computer systems, and computer program products for memory mapping are provided. The invention can include identifying an augmented reality device and observing at least one biometric parameter with at least one Internet of Things (IoT) device. The invention can include defining at least one user attention pattern based on the at least one biometric parameter. The invention can include predicting the user's attention based on the at least one attention pattern. The invention can include recording data from the augmented reality device based on the user's attention falling below a certain point. The invention can include storing the recorded data.

Description

Background Technology

[0001] This invention relates generally to the field of computing, and more specifically to vision-based technologies.

[0002] Smart contact lenses can be capable of capturing video clips of a user's surroundings. For example, based on a specific eye event, a smart contact lens can begin capturing and storing video clips. Meanwhile, the user may be distracted and / or inattentive about their own surroundings (e.g., nervousness, absent-mindedness, etc.). This could mean that even if the user has seen their surroundings, they may not remember specific events (e.g., they may not remember what they saw and / or witnessed, including but not limited to moments of focused attention such as introductions, moments of unusual events, moments of interruption, and / or moments when text is displayed on a media screen during online communication or too long after reading to remember). Summary of the Invention

[0003] This invention discloses a memory mapping method, computer system, and computer program product. The invention may include identifying an augmented reality device and observing at least one Internet of Things (IoT) device with at least one biometric parameter. The invention may include defining at least one user attention pattern based on at least one biometric parameter. The invention may include predicting a user's attention based on at least one attention pattern. The invention may include recording data from the augmented reality device based on a drop in user attention below a specific point. The invention may include storing the recorded data.

[0004] From a first aspect, the present invention provides a method for memory mapping, the method comprising: identifying an augmented reality device and at least one Internet of Things (IoT) device observing at least one biometric parameter; defining at least one user attention pattern based on the at least one biometric parameter; predicting the user's attention based on the at least one attention pattern; recording data from the augmented reality device based on the user's attention dropping below a specific point; and storing the recorded data.

[0005] Preferably, the present invention provides a method wherein the augmented reality device is selected from the group consisting of: a smart contact lens, a pair of smart glasses, and a head-mounted display.

[0006] Preferably, the present invention provides a method in which predicting a user's attention based on at least one attention pattern further includes: deploying a long short-term memory (LSTM) recurrent neural network (RNN) model to predict the user's attention pattern via an autoencoder.

[0007] Preferably, the present invention provides a method that further includes: using principal component analysis (PCA) with a long short-term memory (LSTM) recurrent neural network (RNN) model to determine the context of the user's surrounding environment based on the at least one biometric parameter.

[0008] Preferably, the present invention provides a method in which predicting a user’s attention based on at least one attention pattern further includes: determining that the user’s attention is not focused; and engaging the augmented reality device of a nearby user.

[0009] Preferably, the present invention provides a method in which recording data from the augmented reality device further includes: calculating the deviation of the user's attention from the user's baseline profile based on the user's attention falling below a specific point.

[0010] Preferably, the present invention provides a method in which storing the recorded data further includes: storing the recorded data in a connected database; and retraining a Long Short-Term Memory (LSTM) recurrent neural network (RNN) model.

[0011] From a second aspect, the present invention provides a computer system for memory mapping, comprising: one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more tangible storage media for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising: identifying an augmented reality device and observing at least one Internet of Things (IoT) device with at least one biometric parameter; defining at least one user attention pattern based on the at least one biometric parameter; predicting the user's attention based on the at least one attention pattern; recording data from the augmented reality device based on the user's attention dropping below a specific point; and storing the recorded data.

[0012] Preferably, the present invention provides a computer system wherein the augmented reality device is selected from the group consisting of a smart contact lens, a pair of smart glasses and a head-mounted display.

[0013] Preferably, the present invention provides a computer system in which predicting a user's attention based on at least one attention pattern further includes: deploying a long short-term memory (LSTM) recurrent neural network (RNN) model to predict the user's attention pattern via an autoencoder.

[0014] Preferably, the present invention provides a computer system that further includes: using principal component analysis (PCA) with a long short-term memory (LSTM) recurrent neural network (RNN) model to determine the context of the user's surrounding environment based on the at least one biometric parameter.

[0015] Preferably, the present invention provides a computer system in which predicting a user's attention based on at least one attention pattern further includes: determining that the user's attention is not focused; and engaging the augmented reality device of a nearby user.

[0016] Preferably, the present invention provides a computer system in which, based on the user's attention dropping below a specific point, recording data from the augmented reality device further includes: calculating the deviation of the user's attention from the user's baseline profile.

[0017] Preferably, the present invention provides a computer system in which storing the recorded data further includes: storing the recorded data in a connected database; and retraining a Long Short-Term Memory (LSTM) recurrent neural network (RNN) model.

[0018] In another aspect, the present invention provides a computer program product for memory mapping, comprising: one or more non-transitory computer-readable storage media and program instructions stored on at least one of the one or more tangible storage media, the program instructions being executable by a processor to cause the processor to perform a method comprising: identifying an augmented reality device and observing at least one Internet of Things (IoT) device with at least one biometric parameter; defining at least one user attention pattern based on the at least one biometric parameter; predicting the user's attention based on the at least one attention pattern; recording data from the augmented reality device based on the user's attention dropping below a specific point; and storing the recorded data.

[0019] Preferably, the present invention provides a computer program product, wherein the augmented reality device is selected from the group consisting of a smart contact lens, a pair of smart glasses and a head-mounted display.

[0020] Preferably, the present invention provides a computer program product in which predicting a user's attention based on at least one attention pattern further includes: deploying a long short-term memory (LSTM) recurrent neural network (RNN) model to predict the user's attention pattern via an autoencoder.

[0021] Preferably, the present invention provides a computer program product, further comprising: using principal component analysis (PCA) with a long short-term memory (LSTM) recurrent neural network (RNN) model to determine the context of the user's surrounding environment based on the at least one biometric parameter.

[0022] Preferably, the present invention provides a computer program product in which predicting a user's attention based on at least one attention pattern further includes: determining that the user's attention is not focused; and engaging the augmented reality device of a nearby user.

[0023] Preferably, the present invention provides a computer program product in which, based on the user's attention dropping below a specific point, recording data from the augmented reality device further includes: calculating the deviation between the user's attention and the user's baseline profile.

[0024] Preferably, the present invention provides a method for memory mapping, the method comprising: predicting a user’s inattention; identifying nearby users; and recording multiple data points observed by the nearby user’s augmented reality device based on the predicted user’s inattention.

[0025] Preferably, the present invention provides a method wherein predicting the user's inattention further comprises: deploying a Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN) model to predict the user's attentional patterns via an autoencoder, wherein Principal Component Analysis (PCA) is deployed together with the LSTM Recurrent Neural Network (RNN) model to determine the context of the user's surrounding environment based on at least one biometric parameter.

[0026] Preferably, the present invention provides a method for recording triggered by an augmented reality device, the method comprising: determining, by means of a machine learning algorithm, that the augmented reality device should begin recording based on a prediction of user inattention on the augmented reality device, wherein the prediction of user inattention on the augmented reality device is based on eye events.

[0027] Preferably, the present invention provides a method in which the augmented reality device is a smart contact lens.

[0028] Preferably, the present invention provides a method in which the machine learning algorithm is a Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN) model with Principal Component Analysis (PCA). Attached Figure Description

[0029] These and other objects, features, and advantages of the invention will become apparent from the following detailed description of exemplary embodiments of the invention, which will be read in conjunction with the accompanying drawings. Various features in the drawings are not to scale, as these illustrations are intended to facilitate understanding of the invention by those skilled in the art in conjunction with the detailed description. In the drawings:

[0030] Figure 1 A networked computer environment according to at least one embodiment is shown;

[0031] Figure 2 This is an operational flowchart illustrating a process for memory mapping according to at least one embodiment;

[0032] Figure 3 According to at least one embodiment Figure 1 A block diagram depicting the internal and external components of a computer and server;

[0033] Figure 4 Embodiments according to this disclosure include Figure 1 A block diagram illustrating a cloud computing environment for a computer system.

[0034] Figure 5 According to embodiments of this disclosure Figure 4 A block diagram illustrating the functional layers of an illustrative cloud computing environment. Detailed Implementation

[0035] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it should be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, and they may be implemented in different forms. The invention can be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this disclosure thorough and complete, and to fully convey the scope of the invention to those skilled in the art. Details of well-known features and techniques may be omitted in the description to avoid unnecessarily obscuring the presented embodiments.

[0036] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0037] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0038] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0039] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0040] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0041] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0042] These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, a programmable data processing device, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0043] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0044] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0045] The exemplary embodiments described below provide a system, method, and program product for memory mapping. Accordingly, this embodiment has the capability to improve the field of augmented reality devices by recording and storing visual data collected by augmented reality devices (e.g., smart contact lenses, smart glasses, or head-mounted displays) and continuously trimming the collected data to determine information that can be considered valuable to a user. More specifically, the invention may include at least one Internet of Things (IoT) device that identifies the augmented reality device and observes at least one biometric parameter. The invention may include defining at least one user attention pattern based on at least one biometric parameter. The invention may include predicting a user's attention based on at least one attention pattern. The invention may include recording data from the augmented reality device based on a user's attention dropping below a specific point. The invention may include storing the recorded data.

[0046] As previously mentioned, smart contact lenses can have the ability to capture video clips of the user's surroundings. For example, based on specific eye events (e.g., looking up, looking down, looking left, looking right, eye movement, blinking in a specific order, closing eyes, widening eyes, and other eye events), a smart contact lens can begin capturing and storing video clips. Meanwhile, the user may be distracted and / or inattentive about their own surroundings (e.g., nervousness, absent-mindedness, etc.). This could mean that even if the user has seen their surroundings, they may not remember specific events (e.g., they may not remember what they saw and / or witnessed, including but not limited to moments of focused attention such as introductions, moments of unusual events, moments of interruption, and / or moments of text displayed on a media screen during online communication or too long after reading to remember).

[0047] Therefore, it is particularly advantageous to address such memory-limited scenarios by providing means that can detect the user's short-term memory capacity and / or forgetfulness patterns, and thus, smart contact lenses and / or enhanced glasses can automatically capture video depicting the user's surrounding environment.

[0048] According to at least one embodiment, an intelligent storage management mechanism can be embedded in a smart lens (i.e., a smart contact lens), a pair of smart glasses (i.e., augmented reality glasses), a head-mounted display, and / or another augmented reality gadget. The intelligent storage management mechanism can employ configurable learning tools to teach the selective and dynamic storage, tagging, and / or indexing of verbal, textual, and / or visual data (including any attributes associated with captured fragments by the user) at levels that can be used for future citations.

[0049] According to at least one embodiment, the intelligent storage management mechanism may include using iterative training to continuously prune selective features, the iterative training taking into account user preferences (e.g., preferences configured in the program) and the user's learned behavior to determine the information levels that may be considered valuable to the user and the information levels that may be expected to be used and / or referenced later.

[0050] According to at least one embodiment, the intelligent storage management mechanism can trigger the device's recording capability for individual users and user groups, and can use machine learning to learn when to trigger the device to record.

[0051] According to at least one embodiment, the intelligent storage management mechanism can help retain time-related events, including but not limited to events based on geographic proximity and when the user forgets their location, at a certain time of day (e.g., when the user is overly tired), and / or in scenarios where there is a large amount of activity requiring the user's attention.

[0052] Reference Figure 1 This describes an exemplary networked computer environment 100 according to one embodiment. The networked computer environment 100 may include a computer 102 having a processor 104 and a data storage device 106 and a memory-mapped program 110a capable of running software program 108. The networked computer environment 100 may also include a server 112 capable of running a memory-mapped program 110b that can interact with a database 114 and a communication network 116. The networked computer environment 100 may include multiple computers 102 and servers 112, only one of which is shown. The communication network 116 may include different types of communication networks, such as wide area networks (WANs), local area networks (LANs), telecommunications networks, wireless networks, public switched networks, and / or satellite networks. Augmented reality (AR) devices 118 and Internet of Things (IoT) devices 120 are depicted as their own independent entities as shown, but may be integrated into another part of the computer network environment. It should be understood that... Figure 1 This illustration provides only one possible implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications can be made to the depicted environment based on design and implementation requirements.

[0053] Client computer 102 can communicate with server computer 112 via communication network 116. Communication network 116 may include connections such as wired, wireless communication links, or fiber optic cables. (See reference...) Figure 3 As discussed, server computer 112 may include internal component 902a and external component 904a, and client computer 102 may include internal component 902b and external component 904b. Server computer 112 may also operate in a cloud computing service model (such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS)). Server 112 may also reside in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud. Client computer 102 may be, for example, a mobile device, telephone, personal digital assistant, netbook, laptop computer, tablet computer, desktop computer, or any type of computing device capable of running programs, accessing networks, and accessing database 114. Depending on the implementation of this embodiment, memory-mapped programs 110a, 110b may interact with database 114, which may be embedded in different storage devices, such as, but not limited to, computer / mobile device 102, networked server 112, or cloud storage service.

[0054] According to this embodiment, a user using client computer 102 or server computer 112 can use memory mapping programs 110a and 110b (respectively) to record and store visual data collected by augmented reality devices (e.g., smart contact lenses, a pair of smart glasses, or a head-mounted display) and continuously trim the collected data to determine information that can be considered valuable to the user. (See below for further details.) Figure 2 A more detailed explanation of memory mapping methods.

[0055] Now for reference Figure 2 The document describes an operational flowchart illustrating an exemplary memory mapping process 200 used by memory mapping programs 110a and 110b according to at least one embodiment.

[0056] At 202, the augmented reality (AR) device is connected to at least one mobile and / or internet of things (IoT) device. A smart contact lens, a pair of smart glasses, and / or another augmented reality (AR) device including an embedded camera can be connected to at least one mobile and / or IoT device using a communication network (e.g., communication network 116), enabling the determination of the user's predicted level of attention and the ability to remember the user's surroundings at a later date, and enabling recording.

[0057] The Internet of Things (IoT) can be a system of interconnected computing devices, machines, and digital machines with embedded sensors that can transmit data over the Internet without requiring human-to-human or human-to-computer interaction. Connected IoT devices can be embedded in mobile devices, industrial equipment, and / or environmental sensors, including but not limited to watches, cars, thermostats, and voice assistants.

[0058] AR devices can be used (Bluetooth and all Bluetooth-based trademarks and logos are trademarks or registered trademarks of Bluetooth-SIG, Inc. and / or its affiliates), WiFi, radio and / or other communication methods connect to IoT devices, which can transfer data (e.g., signals, video streams) between AR devices and IoT devices.

[0059] In section 204, define the user's attention patterns. A knowledge corpus (e.g., a cloud database and / or database 114) can be created to store user attention patterns (e.g., identified correlations between user biometric parameters, attention toward the user's surrounding environment, and the ability to remember specific situations).

[0060] Memory mapping procedures 110a and 110b can deploy a Long Short-Term Memory (LSTM) recurrent neural network (RNN), which can be used to predict a user's attentional patterns via an autoencoder. The LSTM-RNN model can take into account multivariate time-series data (e.g., a series of data points indexed in chronological order, which may have been considered at consecutive, equidistant intervals, or points in time). The time-series data can be time-series data about the user and activities performed by the user (if any). The time-series data can be collected from connected augmented reality (AR) devices (e.g., smart contact lenses, a pair of smart glasses, a head-mounted display, and / or another augmented reality gadget).

[0061] For example, time series data might include a user gazing in a specific direction and / or walking or driving in that direction. Time series data can include categorical feature variables (e.g., gazing or walking, and many other things). These categorical feature variables can be encoded (e.g., using label encoding or one-hot encoding, where categorical feature variables can be converted into a numerical form that can be used by machine learning algorithms, as well as other encoding methods) into numerical feature vectors. These numerical feature vectors can then be fed into an LSTM-RNN architecture.

[0062] Numerical feature vectors enable LSTM-RNN models to learn possible user activities (if any) and perform attention detection together with autoencoders (e.g., encoder-decoder architectures used in recurrent neural networks can be used for sequence-to-sequence prediction problems). Attention detection can be a mechanism and / or layer in deep learning models that address the limitations of encoder-decoder architectures on long data sequences and can utilize the skills of LSTM-RNN models regarding sequence-to-sequence prediction problems.

[0063] An autoencoder can be an unsupervised learning technique that takes an image as input (e.g., from a connected camera and / or video feed, including those connected to an augmented reality device), compresses the image into a latent spatial representation, and reconstructs an output image from the latent spatial representation. An autoencoder may include a bottleneck, which may contain an input representation of the data (e.g., data x may have an input representation of f(x)). A decoder in the autoencoder can then use the encoding in the bottleneck to produce a reconstruction of the input data (e.g., the reconstruction of input data x is r = g(f(x))). The autoencoder can learn any regularities in the input data (e.g., video data).

[0064] Historical parameters can be collected from the user's various devices (e.g., augmented reality devices and / or IoT devices) and can be considered here to predict the user's attention patterns. Historical parameters may include collected data on the user's attention levels and the user's ability to remember situations, including whether the user's recall occurred while the user was in an adverse cognitive state (e.g., sadness, stress, and / or health problems). Historical parameters may be collected by connected IoT devices, as previously described with reference to step 202 above, and may be stored in a knowledge corpus (e.g., a cloud database and / or database 114).

[0065] Principal Component Analysis (PCA) can also be used in conjunction with LSTM-RNN models to base data on real-time biometric parameters (e.g., captured via connected IoT wearables), the rate of change of a user's biometric parameters, the contextual needs surrounding the user, any written text (e.g., phone numbers, license plate numbers, information written on business cards, business hours, addresses, and / or street names), and / or introductions when someone states their name, along with other instances that can be captured by AR devices. PCA can also be a dimensionality reduction method used to reduce the dimensionality of a dataset by constructing principal components. PCA graphs can depict clusters together based on similarity.

[0066] When the number of parameters is very complex and the LSTM-RNN model ingests multiple inputs, PCA can be used to understand the relationships between input variables and / or perform dimensionality reduction (e.g., reducing the number of input features or feature vectors that will be used with the algorithm).

[0067] At 206, predict the user's current attention. When predicting the cognitive state in which the user might ignore or fail to remember certain details related to the user's surrounding environment, an LSTM-RNN model with PCA (as described above relative to step 204) can be considered for feature analysis and pruning.

[0068] Cognitive heuristics for users (e.g., methods used by users to process information, including thinking about problems rationally, logically, intentionally, verbally, effortlessly, emotionally, and / or intuitively) can be used to identify users’ “normal behavior” and anomalous behaviors, while “increments” (e.g., based on the use of certain words, loud voices, and / or drastic accelerometer changes) can be used to identify users’ attention at a given time.

[0069] According to at least one embodiment, a user's attention can be measured on a relative scale (e.g., where lower measured attention has an attention score closer to 0 and higher measured attention scores closer to 100). The user's augmented reality (AR) device can begin recording when the user's attention drops below a predefined threshold and / or deviates from the user's behavior by an increment (e.g., drops below a certain point on the relative scale, such as 50% of the calculated value).

[0070] For example, a baseline profile of the user can be created (e.g., using the learning mechanism previously described with respect to step 204 above), and deviations from the user's baseline profile can be identified, wherein memory mapping procedures 110a, 110b further identify the user as attentive or inattentive.

[0071] Specific eye events (e.g., looking up, looking down, looking left, looking right, moving the eyes, blinking in a specific order, closing the eyes, opening the eyes, and other eye events) can also trigger the smart contact lens to begin capturing and storing video clips. By projecting information directly onto the user's retina, augmented reality (AR) devices (e.g., the user's smart contact lens, a pair of smart glasses, and / or other augmented reality (AR) devices) can be able to capture gesture-based eye movements to initiate the capture and storage of video clips.

[0072] Feature pruning (e.g., iterative event pruning to reduce complexity in machine learning algorithms) can also be utilized when a user attempts to recall a situation and / or event from a multitude of past events (e.g., E1, E2, ..., En). A user may experience difficulty recalling an event (e.g., Em) but may be able to remember others. PCA analysis and other feature frame pruning mechanisms can be used to build context for the user, including by developing pruning strategies for the user (e.g., identifying instances where the user experiences difficulty recalling past events). Iterative event pruning can help build a baseline profile of the user and identify deviations from that profile by preserving data captured by the user's smart contact lens, a pair of smart glasses, and / or other augmented reality (AR) devices (e.g., photo and / or video data, etc.), which may relate to events the user previously did not remember.

[0073] At 208, memory mapping procedures 110a, 110b determine whether recording on the augmented reality (AR) device should begin. For example, based on specific eye events (e.g., looking up, looking down, looking left, looking right, moving the eyes, blinking in a specific order, closing the eyes, opening the eyes, and other eye events) and / or the user's biometric parameters, the smart contact lens can begin capturing and storing video clips.

[0074] The memory model can be co-located with an auxiliary processor and / or a graphics processing unit (GPU) that performs PCA analysis using a trial-and-error approach based on user changes (as previously described with respect to steps 204 and 206 above).

[0075] For example, user distraction and inattention can be detected, and based on historical data (e.g., user attention levels in previous similar situations), memory mapping procedures 110a, 110b can determine the details of what the user will not remember and should begin recording on the augmented reality (AR) device. Based on this determination, the user's various devices (including any connected AR and / or IoT devices) can continue to track the user's attention levels and also capture the user's surrounding environment.

[0076] Furthermore, according to at least one embodiment of the present invention, if the computing ecosystem of a user connected to memory mapping programs 110a, 110b (e.g., a combination of the user's mobile devices or wearable devices that feed data from smart contact lenses and / or IoT sensors, etc.) identifies that the user is not focused or may not remember the event based on the similarity between the event and a past event that the user does not remember, and at the same time the user's focus is distracted from the target's surrounding environment, then memory mapping programs 110a, 110b may interact with at least one other user's (i.e., nearby user's) paired smart contact lenses and / or augmented reality devices (e.g., augmented glasses, augmented reality glasses, etc.) to capture the user's surrounding environment.

[0077] Memory mapping programs 110a and 110b can immediately identify and connect to the augmented reality devices of nearby users (e.g., nearby users who also use versions of memory mapping programs 110a and 110b). Any data captured by the augmented reality devices of nearby users can be stored in the database of memory mapping programs 110a and 110b connected to the user.

[0078] For example, two friends are traveling together. If one friend is unhappy and not very focused (e.g., not concentrating on famous sites in the area), memory mapping programs 110a, 110b will recognize this and will combine the other friend's smart contact lens and / or the sensors of augmented reality device(s) to capture the surrounding environment on behalf of the other friend.

[0079] If it is determined here that recording should begin, the memory mapping procedures 110a, 110b proceed to step 210 below.

[0080] If it is determined here that recording should not begin, then memory mapping procedures 110a and 110b end.

[0081] At point 210, recording begins. A content recording event can be initiated at time T. At that time, the smart contact lens and / or other augmented reality devices (e.g., augmented glasses, augmented reality glasses, head-mounted displays, etc.) can automatically record video of the user's surroundings based on the above determination at step 208 that recording should begin on the augmented reality (AR) device.

[0082] For example, in instances where the user is excessively excited or fearful, or in other emotional states that may deplete the user's responsiveness and / or thoughts, the video may be automatically captured by memory mapping procedures 110a, 110b on smart contact lenses and / or other augmented reality (AR) devices. The user's emotions may be determined by memory mapping procedures 110a, 110b based on connected IoT devices, including but not limited to wearable devices capable of capturing the user's biometric data (i.e., biometric parameters).

[0083] In step 212, the recorded data is stored and the machine learning model is retrained. Training cycles can be combined.

[0084] At 212, the recorded data is stored and the machine learning model is retrained. Training periods can be joined by memory mapping procedures 110a, 110b, and all data collected during time series events (e.g., time T of a series of events E (E1, E2, ..., En)) can be recorded and stored in a cloud database (e.g., database 114).

[0085] The retraining of the machine learning model can be performed as previously described with respect to step 204 above.

[0086] Understandable Figure 2 This is merely an illustration of one embodiment and does not imply any limitation on how different embodiments may be implemented. Many modifications can be made to the depicted embodiment based on design and implementation requirements.

[0087] Figure 3 This is an illustrative embodiment of the present invention. Figure 1 Block diagram 900 depicts the internal and external components of a computer. It should be understood that... Figure 3 This illustration provides only one possible implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications can be made to the depicted environment based on design and implementation requirements.

[0088] Data processing systems 902 and 904 represent any electronic device capable of executing machine-readable program instructions. Data processing systems 902 and 904 may represent smartphones, computer systems, PDAs, or other electronic devices. Examples of computing systems, environments, and / or configurations that data processing systems 902 and 904 may represent include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0089] The user client computer 102 and the network server 112 may include Figure 3 The internal components 902a,b and external components 904a,b of the corresponding groups are shown. Each group of internal components 902a,b includes one or more processors 906, one or more computer-readable RAMs 908 and one or more computer-readable ROMs 910 on one or more buses 912, as well as one or more operating systems 914 and one or more computer-readable tangible storage devices 916. One or more operating systems 914, software programs 108 and memory-mapped programs 110a in client computer 102 and memory-mapped programs 110b in network server 112 may be stored on one or more computer-readable tangible storage devices 916 for execution by one or more processors 906 via one or more RAMs 908 (which typically include cache memory). Figure 3 In the embodiment shown, each computer-readable tangible storage device 916 is a disk storage device of an internal hard disk drive. Alternatively, each computer-readable tangible storage device 916 is a semiconductor storage device, such as ROM 910, EPROM, flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0090] Each set of internal components 902a,b also includes an R / W drive or interface 918 for reading from and writing to one or more portable computer-readable tangible storage devices 920, such as CD-ROM, DVD, Memory Stick, magnetic tape, disk, optical disc, or semiconductor storage devices. Software programs (such as software program 108 and memory-mapped programs 110a and 110b) may be stored on one or more corresponding portable computer-readable tangible storage devices 920, read from and loaded into corresponding hard disk drives 916 via the corresponding R / W drive or interface 918.

[0091] Each set of internal components 902a,b may also include a network adapter (or switch port card) or interface 922, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G or 4G wireless interface card, or other wired or wireless communication links. The software program 108 and memory mapping program 110a in the client computer 102 and the memory mapping program 110b in the network server computer 112 may be downloaded from an external computer (e.g., a server) via a network (e.g., the Internet, a local area network, or another wide area network) and the corresponding network adapter or interface 922. From the network adapter (or switch port adapter) or interface 922, the software program 108 and memory mapping program 110a in the client computer 102 and the memory mapping program 110b in the network server computer 112 are loaded into the corresponding hard disk drive 916. The network may include copper wire, fiber optic, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers.

[0092] Each set of external components 904a,b may include a computer display monitor 924, a keyboard 926, and a computer mouse 928. External components 904a,b may also include a touchscreen, virtual keyboard, touchpad, pointing device, and other human-computer interface devices. Each set of internal components 902a,b also includes a device driver 930 connected to the computer display monitor 924, keyboard 926, and computer mouse 928. The device driver 930, R / W driver or interface 918, and network adapter or interface 922 include hardware and software (stored in storage device 916 and / or ROM 910).

[0093] It should be understood in advance that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0094] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0095] The features are as follows:

[0096] On-demand self-service: Cloud consumers can automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0097] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0098] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0099] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0100] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0101] The service model is as follows:

[0102] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0103] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0104] Infrastructure as a Service (IaaS): The capabilities offered to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0105] The deployment model is as follows:

[0106] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0107] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0108] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0109] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0110] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0111] Now for reference Figure 4 This describes an illustrative cloud computing environment 1000. As shown, the cloud computing environment 1000 includes one or more cloud computing nodes 100 that can communicate with local computing devices used by cloud consumers, such as, for example, personal digital assistants (PDAs) or cellular phones 1000A, desktop computers 1000B, laptop computers 1000C, and / or automotive computer systems 1000N. The nodes 100 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 1000 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 4 The types of computing devices 1000A-N shown are intended to be illustrative only, and computing node 100 and cloud computing environment 1000 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0112] Now for reference Figure 5 This illustrates a set of functional abstraction layers 1100 provided by the cloud computing environment 1000. It should be understood in advance that... Figure 5The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0113] The hardware and software layer 1102 includes hardware and software components. Examples of hardware components include: a mainframe 1104; a server 1106 based on a RISC (Reduced Instruction Set Computer) architecture; a server 1108; a blade server 1110; a storage device 1112; and a network and networking component 1114. In some embodiments, the software components include network application server software 1116 and database software 1118.

[0114] The virtualization layer 1120 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 1122; virtual storage 1124; virtual network 1126, including virtual private network; virtual application and operating system 1128; and virtual client 1130.

[0115] In one example, management layer 1132 may provide the functionality described below: Resource Provisioning 1134 Provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 1136 Provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security Provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 1138 Provides consumers and system administrators with access to the cloud computing environment. Service Level Management 1140 Provides allocation and management of cloud computing resources to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 1142 Provides pre-scheduling and procurement of cloud computing resources, anticipating future requirements for those resources according to the SLA.

[0116] Workload layer 1144 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 1146; software development and lifecycle management 1148; virtual classroom education delivery 1150; data analytics and processing 1152; transaction processing 1154; and memory mapping 1156. Memory mapping programs 110a and 110b provide methods for recording and storing visual data collected by augmented reality devices (e.g., smart contact lenses, a pair of smart glasses) and continuously trimming the collected data to determine how it can be considered information valuable to the user.

[0117] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for memory mapping, the method comprising: Identify augmented reality devices and observe at least one Internet of Things (IoT) device with at least one biometric parameter; Define at least one user attention pattern based on the at least one biometric parameter; The user's attention is predicted using a Long Short-Term Memory Recurrent Neural Network (LSTM RNN) model based on at least one user attention pattern. Based on the user's attention dropping below a specific point, data from the augmented reality device is recorded, in part based on iterative event pruning and the determination of the user's memory pruning strategy, to establish the user's baseline profile, wherein the user's memory pruning strategy is a strategy for identifying instances in which the user experiences difficulty recalling past events; Store the recorded data; as well as The LSTM RNN model is retrained using multivariate time series data points of the recorded data, wherein the LSTM RNN model updates the user's memory pruning strategy.

2. The method according to claim 1, wherein, The augmented reality device is selected from the group consisting of: Smart contact lenses, smart glasses, and head-mounted displays.

3. The method according to claim 1, wherein, The prediction of the user's attention based on the at least one user attention pattern further includes: Deploy the LSTM RNN model to predict the at least one user attention pattern via an autoencoder.

4. The method of claim 3, further comprising: Principal component analysis (PCA) with the LSTM RNN model is used to determine the context of the user's surrounding environment based on the at least one biometric parameter.

5. The method according to claim 1, wherein, The prediction of the user's attention based on the at least one user attention pattern further includes: Determine that the user is not paying attention; and The augmented reality device is connected to the nearby user.

6. The method according to claim 1, wherein, The step of recording the data from the augmented reality device based on the user's attention dropping below the specific point further includes: Calculate the deviation between the user's attention and the user's baseline profile.

7. The method according to claim 1, wherein, The data recorded in the storage further includes: The recorded data is stored in the connected database.

8. A computer system for memory mapping, comprising: One or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more tangible storage media for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method including the following steps: Identify augmented reality devices and observe at least one Internet of Things (IoT) device with at least one biometric parameter; Define at least one user attention pattern based on the at least one biometric parameter; The user's attention is predicted using a Long Short-Term Memory Recurrent Neural Network (LSTM RNN) model based on at least one user attention pattern. Based on the user's attention dropping below a specific point, data from the augmented reality device is recorded, in part based on iterative event pruning and the determination of the user's memory pruning strategy, to establish the user's baseline profile, wherein the user's memory pruning strategy is a strategy for identifying instances in which the user experiences difficulty recalling past events; Store the recorded data; as well as The LSTM RNN model is retrained using multivariate time series data points of the recorded data, wherein the LSTM RNN model updates the user's memory pruning strategy.

9. The computer system according to claim 8, wherein, The augmented reality device is selected from the group consisting of: Smart contact lenses, smart glasses, and head-mounted displays.

10. The computer system according to claim 8, wherein, The prediction of the user's attention based on the at least one user attention pattern further includes: The LSTM RNN model is deployed to predict the user attention pattern via an autoencoder.

11. The computer system of claim 10, further comprising: Principal component analysis (PCA) with the LSTM RNN model is used to determine the context of the user's surrounding environment based on the at least one biometric parameter.

12. The computer system according to claim 8, wherein, The prediction of the user's attention based on the at least one user attention pattern further includes: Determine that the user is not paying attention; and The augmented reality device is connected to the nearby user.

13. The computer system according to claim 8, wherein, The step of recording the data from the augmented reality device based on the user's attention dropping below the specific point further includes: Calculate the deviation between the user's attention and the user's baseline profile.

14. The computer system according to claim 8, wherein, The data recorded in the storage further includes: The recorded data is stored in the connected database.

15. A computer program product for memory mapping, comprising: Program instructions, executable by a processor, to cause the processor to perform a method comprising: Identify augmented reality devices and observe at least one Internet of Things (IoT) device with at least one biometric parameter; Define at least one user attention pattern based on the at least one biometric parameter; The user's attention is predicted using a Long Short-Term Memory Recurrent Neural Network (LSTM RNN) model based on at least one user attention pattern. Based on the user's attention dropping below a specific point, data from the augmented reality device is recorded, in part based on iterative event pruning and the determination of the user's memory pruning strategy, to establish the user's baseline profile, wherein the user's memory pruning strategy is a strategy for identifying instances in which the user experiences difficulty recalling past events; Store the recorded data; and The LSTM RNN model is retrained using multivariate time series data points of the recorded data, wherein the LSTM RNN model updates the user's memory pruning strategy.

16. The computer program product according to claim 15, wherein, The augmented reality device is selected from the group consisting of: Smart contact lenses, smart glasses, and head-mounted displays.

17. The computer program product according to claim 15, wherein, The prediction of the user's attention based on the at least one user attention pattern further includes: The LSTM RNN model is deployed to predict the user attention pattern via an autoencoder.

18. The computer program product of claim 17, further comprising: Principal component analysis (PCA) with the LSTM RNN model is used to determine the context of the user's surrounding environment based on the at least one biometric parameter.

19. The computer program product according to claim 15, wherein, The prediction of the user's attention based on the at least one user attention pattern further includes: Determine that the user is not paying attention; and The augmented reality device is connected to the nearby user.

20. The computer program product according to claim 15, wherein, The step of recording the data from the augmented reality device based on the user's attention dropping below the specific point further includes: Calculate the deviation between the user's attention and the user's baseline profile.

21. A method for memory mapping, the method comprising: The user's baseline profile is used to predict the user's inattention. The user's baseline profile is established based on iterative event pruning and the determination of the user's memory pruning strategy, wherein the user's memory pruning strategy is a strategy to identify instances in which the user experiences difficulty recalling past events. Identify nearby users; Data observed by the augmented reality devices of the nearby users is recorded based on the predicted user's inattention. as well as The LSTM RNN model is retrained using multivariate time series data points of the recorded data, wherein the LSTM RNN model updates the user's memory pruning strategy.

22. The method according to claim 21, wherein, The prediction of the user's lack of concentration further includes: The LSTM RNN model is deployed to predict the user's attention patterns via an autoencoder, wherein principal component analysis (PCA) is used in conjunction with the LSTM RNN model to determine the context of the user's surrounding environment based on at least one biometric parameter.

23. A method for triggering recording by an augmented reality device, the method comprising: The augmented reality device and at least one Internet of Things (IoT) device that observes eye events are identified, wherein the eye events are selected from the group including looking up, looking down, looking left, looking right, turning the eyes, blinking in a specific order, closing the eyes, and opening the eyes wide. Define at least one user attention pattern based on the eye events; The augmented reality device begins recording when the user's attention drops below a certain point, determined using a machine learning algorithm. as well as Data is recorded from the augmented reality device, wherein a baseline profile of the user is established based on iterative event pruning and the determination of a memory pruning strategy for the user, wherein the user's memory pruning strategy is a strategy for identifying instances in which the user experiences difficulty recalling past events.

24. The method according to claim 23, wherein, The augmented reality device is a smart contact lens.

25. The method according to claim 23, wherein, The machine learning algorithm is a Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN) model with Principal Component Analysis (PCA).

Images