Systems and methods for contextual memory capture and recall

Abstract

Systems and methods for contextual memory capture and recall are provided. The contextual memory capture and recall systems and methods help users create, store, and recall memory information associated with identified activities. The contextual memory capture and recall systems and methods are able to identify user activities for which a memory query can be desired, create a memory query with recommended memory actions based on the activity, provide the memory query to the user, and automatically link accepted memory actions, as well as any received memory information for the memory actions, with the identified activity.

Description

[0001] Related application citation

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2017 / 066233, international application date of December 14, 2017, entry into the Chinese national phase date of June 20, 2019, Chinese national application number 201780079304.3, and invention title "System and method for contextual memory capture and recall". Technical Field

[0003] Embodiments of this disclosure relate to systems and methods for contextual memory capture and retrieval. Background Technology

[0004] Language understanding systems, personal digital assistants (PDAs), agents, and artificial intelligence (AI) are transforming how users interact with computers. Developers of computers, web services, and / or applications are constantly striving to improve human-computer interaction. Language understanding systems, PDAs, agents, and AI are commonly used to communicate with users and / or perform basic tasks.

[0005] These and other general considerations have been addressed in the various aspects disclosed herein. Moreover, although relatively specific problems have been discussed, it should be understood that these aspects should not be limited to solving the specific problems identified in the background art or elsewhere in this disclosure. Summary of the Invention

[0006] In summary, this disclosure generally relates to systems and methods for contextual memory capture and retrieval. Contextual memory capture and retrieval systems and methods assist users in creating, storing, and recalling memory information associated with identified activities. Contextual memory capture and retrieval systems and methods are capable of identifying user activities and determining whether a memory query is expected for each identified activity. In response to determining that a memory query is expected for an activity, contextual memory capture and retrieval systems and methods are also capable of creating a memory query with recommended memory actions based on the activity, providing the memory query to the user, and automatically linking the accepted memory action and any received memory information for the memory action to the identified activity.

[0007] One aspect of this disclosure relates to a system for context memory capture and retrieval. The system includes at least one processor and a memory. The memory encodes computer-executable instructions that, when executed by the at least one processor, are operable for:

[0008] Collect user context signals from the user's client computing device;

[0009] Leverage world knowledge to enrich user context elements from user context signals, thereby creating a richer context element;

[0010] The user's activity is identified based on rich contextual elements, where the activity is a future activity;

[0011] Evaluate activities based on query rules;

[0012] The desired memory retrieval for the activity is determined based on the evaluation of the activity;

[0013] Create appropriate memory queries based on activities and creation rules;

[0014] In response to determining the expectation to perform a memory query, rich contextual elements are evaluated based on notification rules;

[0015] Collect memory input from the client computing device in response to memory queries;

[0016] Memory actions are created based on memory input, where memory actions include the retrieval of memory information;

[0017] Link the memory action to the first boundary of the activity;

[0018] Detecting the first boundary based on rich contextual elements; and

[0019] In response to the detection of the first boundary, an instruction is sent to the client computing device to provide the user with memory information.

[0020] Memory input includes memory information.

[0021] On the other hand, a method for contextual memory capture and retrieval is disclosed. This method includes:

[0022] Collect context signals from at least one of the user's client computing devices;

[0023] Utilize world knowledge to enrich contextual elements from contextual signals to form a richer contextual element;

[0024] Use rich contextual elements to identify user activities;

[0025] The desired memory retrieval for the activity is determined based on the evaluation of the activity;

[0026] Provide users with memory retrieval;

[0027] Collect memory input from the user in response to memory queries;

[0028] Memory actions are created in response to memory input, where memory actions include the retrieval of memory information; and

[0029] Link the memory input to at least one boundary of the activity.

[0030] Memory input includes memory information entered by the user.

[0031] In another aspect of the invention, this disclosure relates to a system for context memory capture and retrieval. The system includes at least one processor and a memory. The memory encodes computer-executable instructions that, when executed by the at least one processor, are operable for:

[0032] Collect context signals from at least one of the user's client computing devices;

[0033] Utilize world knowledge to enrich contextual signals and form rich contextual elements;

[0034] Use rich contextual elements to identify user activities;

[0035] The desired memory retrieval for the activity is determined based on the evaluation of the activity;

[0036] In response to the determination that a memory query is expected to be performed for an activity, a memory query is created based on the activity;

[0037] Provide memory lookup to at least one client computing device;

[0038] Collect memory input from at least one client computing device in response to a memory query; and

[0039] It links memory actions with activity boundaries in response to memory input.

[0040] This "Summary" is provided to introduce some concepts in a simplified form, which will be further described in the "Detailed Description" below. This "Summary" is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0041] Non-limiting and non-exhaustive embodiments are described with reference to the following figures.

[0042] Figure 1A This is a schematic diagram illustrating a context memory capture and recall system on a client computing device according to various aspects of this disclosure.

[0043] Figure 1B This is a schematic diagram illustrating a context memory capture and recall system on a server computing device used by a user via a client computing device, according to various aspects of this disclosure.

[0044] Figure 2This is a simplified schematic block diagram illustrating the use of a context memory capture and recall system according to various aspects of this disclosure.

[0045] Figure 3A This is a simplified schematic diagram illustrating a user interface of a client computing device displaying a first memory query for an activity targeting a first identifier of a user, according to various aspects of this disclosure.

[0046] Figure 3B This illustrates the display of memory information from memory input related to an activity for the first identifier in response to boundary detection of an activity for the first identifier, according to various aspects of this disclosure. Figure 3A A simplified diagram of the user interface.

[0047] Figure 3C This illustrates a second memory query that displays the user's second identifier according to various aspects of this disclosure. Figure 3A A simplified diagram of the user interface.

[0048] Figure 4 This is a block flowchart illustrating a method for contextual memory capture and retrieval according to various aspects of this disclosure.

[0049] Figure 5 This is a block diagram illustrating example physical components of a computing device that can be used to implement various aspects of this disclosure.

[0050] Figure 6A This is a simplified block diagram of a mobile computing device that can be used to implement various aspects of this disclosure.

[0051] Figure 6B It can be used to implement various aspects of this disclosure. Figure 6A A simplified block diagram of a mobile computing device is shown.

[0052] Figure 7 This is a simplified block diagram of a distributed computing system that can be used to implement various aspects of this disclosure.

[0053] Figure 8 A tablet computing device is shown that can be used to implement various aspects of this disclosure. Detailed Implementation

[0054] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and illustrate specific aspects or examples by way of illustration. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the spirit or scope of this disclosure. Therefore, the following detailed description should not be considered limiting, and the scope of this disclosure is defined by the claims and their equivalents.

[0055] Advances in machine learning, language understanding, and artificial intelligence are transforming how users interact with computers. Digital assistant applications such as Siri, Google Now, and Cortana are examples of this shift in human-computer interaction.

[0056] Human memory cannot record and recall everything. It is nearly impossible for a person to remember everything they need to remember every time they complete an activity. Often, people find themselves thinking, "Oh, I didn't ask this person that question," or "I forgot to call this person as promised before my flight took off," and so on.

[0057] Currently, language understanding systems, personal digital assistants, agents, and artificial intelligence are used to communicate with users and / or complete basic tasks. These previously used systems and methods can provide reminders with user input. However, these reminders are only created upon user request. Furthermore, the user must identify the triggering event for displaying the created reminder, such as an activity, location, or time of day. In other words, the user must identify the need for a reminder, input any desired memory information, and / or then link that reminder and information to the desired triggering activity. Moreover, while these previously used systems and methods can suggest actions based on detected user activity, they do not suggest the execution of memory actions, such as creating reminders with memory input associated with currently or future identified activities. Thus, these previously used systems and methods only have user-initiated memory capture and retrieval and / or user-provided associations for memory capture and retrieval.

[0058] Therefore, systems and methods for contextual memory capture and retrieval are disclosed herein. These systems and methods utilize rich user contextual signals to identify user activities and determine whether a memory query for each identified activity is desirable, beneficial, or appropriate for the user. The systems and methods then create appropriate memory queries based at least on the activity and using memory action recommendations. As used herein, a memory query refers to a request for approval to perform a memory action associated with the identified activity. The memory query for the identified activity is presented to the user, and the memory query requests memory input from the user in response. Memory input can be approval of a memory action or memory information provided by the user. The systems and methods automatically link memory actions and / or memory information to the identified activity. The systems and methods described herein create applications such as digital assistants that improve user task completion, enhance recall and capture of required memory items, and improve usability, performance, and / or user interaction with the application, compared to previously used applications that do not provide memory queries in response to user activity identification and / or do not automatically link user-input memory to user activity.

[0059] Figure 1A and 1B Different examples of a context memory capture and retrieval system 100 on a client computing device 104 used by user 102 according to various aspects of this disclosure are shown. The context memory capture and retrieval system 100 is a system that helps a user create, store, and retrieve memory information about activities. The context memory capture and retrieval system 100 includes an activity framework 108, a notification agent 109, and a linking system 111. The context memory capture and retrieval system 100 is capable of determining whether a user expects a memory query for an identified activity, creating a memory query based on the activity that recommends a memory action, providing the memory query to the user, and automatically linking the memory action and any received memory information to the identified activity. The context memory capture and retrieval system 100 is capable of performing the above operations by determining the user's context by analyzing received user context signals generated by one or more devices 104 of the user based on world knowledge 110. In contrast, previously used systems such as digital assistants only provide user-initiated memory capture and retrieval and / or user-provided associations for memory capture and retrieval.

[0060] The context memory capture and retrieval system 100 may also include a memory storage device 106 for storing memory inputs, memory information, rich memory elements (such as rich context elements), memory queries, memory actions, user patterns, and / or user feedback. Alternatively, the memory storage device 106 may be stored on a separate and distinct database 115 from the context memory capture and retrieval system 100.

[0061] The context memory capture and recall system 100 may further include a pattern detection framework 112 for determining user patterns and / or user feedback based on analysis of user context signals, other user signals, and / or world knowledge. Alternatively, the pattern detection framework 112 resides on a separate server 105, distinct from and separate from the context memory capture and recall system 100. In other respects, the pattern detection framework 112 and / or memory storage device 106 are not part of the context memory capture and recall system 100, but may communicate with it to exchange data.

[0062] In some respects, the context memory capture and retrieval system 100 is implemented on the client computing device 104, such as Figure 1A As shown. In the basic configuration, the client computing device 104 is a computer with input and output elements. The client computing device 104 can be any suitable computing device for implementing the context memory capture and recall system 100. For example, the client computing device 104 can be a mobile phone, smartphone, tablet computer, phablet, smartwatch, wearable computer, personal computer, gaming system, desktop computer, laptop computer, etc. This list is merely exemplary and should not be considered as a limitation. Any suitable client computing device 104 for implementing the context memory capture and recall system 100 can be used.

[0063] In other respects, the context memory capture and recall system 100 is implemented on the server computing device 105, such as... Figure 1B As shown. Server computing device 105 can provide data to and / or receive data from client computing device 104 via network 113. In some aspects, network 113 is a distributed computing network, such as the Internet. In other aspects, the context memory capture and recall system 100 is implemented on more than one server computing device 105, such as multiple server computing devices 105 or a network of server computing devices 105. In some aspects, the context memory capture and recall system 100 is a hybrid system, wherein a portion of the context memory capture and recall system 100 is on client computing device 104 and a portion of the context memory capture and recall system 100 is on server computing device 105.

[0064] Figure 2 This is a simplified schematic block diagram example illustrating the use of the context memory capture and recall system 100 according to various aspects of this disclosure. As described above and as... Figure 2 As shown, the contextual memory capture and retrieval system 100 includes an activity detection framework 108. The activity detection framework 108 of the contextual memory capture and retrieval system 100 collects user signals 116, including contextual signals. As used herein, the term "collection" refers to active retrieval of items and / or passive reception of items. The activity detection framework 108 of the contextual memory capture and retrieval system 100 may also collect world knowledge and / or additional user information 128, such as user feedback, user patterns, and / or other user-rich memory elements, from a pattern detection framework 112.

[0065] User signal 116 is generated by signal generator 138. The signal generator is one or more devices 104 of user 102 and / or one or more applications 138 running on user device 104. For example, client computing device 104 may include the user's desktop computer and / or the user's smartphone. In another example, the application 138 on client computing device 104 that sends user signal 116 may include digital assistant applications, voice recognition applications, email applications, social networking applications, collaboration applications, enterprise management applications, messaging applications, word processing applications, spreadsheet applications, database applications, presentation applications, contact applications, gaming applications, e-commerce applications, photo applications, map applications, e-commerce applications, transaction applications, exchange applications, device control applications, web interface applications, calendar applications, etc.

[0066] The activity detection framework 108 of the context memory capture and recall system 100 receives user signals 116 and / or additional user information 128. The activity detection framework 108 includes an enrichment system 118, a detection system 119, a query decision system 120, and a creation system 122.

[0067] The enrichment system 118 of the activity detection framework platform 108 collects user signals 116, including user context signals. User context signals include context elements and / or digital artifacts of user 102. The enrichment system 118 uses world knowledge 110 to transform digital artifacts into context elements. Furthermore, the enrichment system 118 enriches the context elements using world knowledge 110. World knowledge 110, as used herein, includes any information accessible via a network connection, such as search engines and databases. User context signals are specific to and related to a given user 102. User context signals are signals related to the current state of user 102. The current state or user context is the current environment of user 102 and / or the client computing device 104. For example, the current state or user context may be based on the user's current location, current time, current weather, the user's current digital behavior, and / or the user's current physical actions. In another aspect, the enrichment system 118 enriches all user signals to form memory elements and is not limited to enriching user context signals.

[0068] For example, digital artifacts such as GPS coordinates have no contextual value to user 102. However, when searching world knowledge 110, enrichment system 118 can determine that these coordinates are for a Starbucks at a specific address in Seattle, Washington. In response to this determination, enrichment system 118 can convert the digital GPS coordinates into the context elements “Starbucks” and “Seattle”. Furthermore, enrichment system 118 can search world knowledge 110 to enrich the “Starbucks” context element and determine that “Starbucks” is a coffee shop. In this embodiment, enrichment system 118 can enrich the “Starbucks” context element by labeling it as a “coffee shop”. Thus, enrichment system 118 labels context elements and determines additional context elements to form a rich context element set.

[0069] Detection system 119 collects rich contextual elements generated by richness system 118. Detection system 119 analyzes these rich contextual elements to identify user activities. User activities can be future activities (future activities) or currently occurring activities (current activities). For example, identified future activities could be doctor's appointments, trips to New York, relatives' birthdays, work meetings, etc. Identified current activities could be a user's work commute, the first day of a trip to New York, work meetings, etc. In some aspects, detection system 119 analyzes or evaluates rich contextual signals based on a set of contextual rules. In other aspects, detection system 118 also collects additional user information 128, such as user feedback and user patterns. In these aspects, learning algorithms can be used to update the contextual rules based on the additional user information 128. Any learning algorithms mentioned herein can include machine learning and / or statistical modeling techniques. The ability of detection system 118 to update contextual rules based on user feedback and / or user patterns allows detection system 118 to continuously evolve with user 102 based on user patterns and / or feedback.

[0070] The query decision system 120 collects identified user activities. The query decision system 120 evaluates the identified user activities to determine whether a memory query is expected for that activity. The query decision system 120 uses query rules to evaluate the first activity. In some aspects, the query decision system 120 also collects additional user information 128. In these aspects, the set of query rules can be updated using learning algorithms based on the additional user information 128. The ability of the query decision system 120 to update the query rules based on user feedback and / or user patterns allows the query decision system 120 to continuously evolve with the user 102 based on user patterns and / or feedback. If the activity satisfies the query rules, the query decision system 120 determines that a memory query for the identified activity is appropriate. If the activity does not satisfy the query rules, the query decision system 120 determines that a memory query for the identified activity is inappropriate. For example, an identified activity such as a user's work commute may not meet the rules for a memory query. In another example, an identified activity such as a doctor's appointment or an upcoming trip may satisfy the query rules and qualify for a memory query. In some respects, query rules are a list of identified current and future activities that are appropriate or desirable for a memory query. In other respects, query rules are a list of identified current and future activities that are inappropriate or undesirable for a memory query. However, this list of rules is merely exemplary and not restrictive. As those skilled in the art will understand, query decision operation 120 may use any suitable query rules for determining whether a memory query is appropriate.

[0071] In response to query decision system 120 determining that a memory query is desired, creation system 122 creates or forms a memory query. The memory query identifies an activity and requests approval to perform a memory action. As used herein, a memory action refers to any action or instruction generated for performing any action related to the storage and / or retrieval of memory information in association with the identified activity. Retrieval of memory information associated with the identified activity is referred to herein as memory recall. In some aspects, memory recall may include reminders or reminder information about the activity (also referred to herein as memory information). In other aspects, memory recall may include activity-related memory information that a user wants to store in association with the activity, so that the activity-related information can be recalled upon request. Creation system 122 creates memory query 132 based at least on the activity. In some aspects, creation system 122 further creates memory query 132 based on world knowledge and / or additional user information 128. In some aspects, memory query is determined or created by creation system 122 based on which query rule the activity satisfies. For example, each query rule suitable for a memory query or each identified activity may be associated with a predetermined memory query. Pre-defined memory queries can be templates populated based on analysis of activities, world knowledge, and / or additional user information 128. In other respects, pre-defined memory queries are static prompts that do not change.

[0072] In alternative aspects, memory queries are determined based on the analysis of activities, as well as world knowledge and / or additional user information 128, and are not based on predetermined memory queries. In aspects where memory queries are not static, the creation system 122 can analyze or evaluate activities based on a set of creation rules to form memory queries. In other aspects, the creation system 122 also collects additional user information 128 and / or world knowledge. In these aspects, creation rules can be updated using learning algorithms based on the additional user information 128. The ability of the creation system 122 to update creation rules based on user feedback and / or user patterns allows the creation system 122 to continuously evolve with the user 102 based on user patterns and / or feedback.

[0073] Furthermore, in these aspects, the creation system 122 can evaluate other rich memory elements, world knowledge, and / or additional user information to determine, based on creation rules, whether any memory information should be provided (or recommended) in the memory query. If the creation system 122 identifies any recommended memory information for the activity, the recommended memory information is provided in the created memory query. In some aspects, the creation system 122 utilizes a user interface or selectable icons or buttons for accepting and / or rejecting the provided memory queries to create memory queries.

[0074] Additionally, the creation system 122 can add explanations to the provided memory queries based on activities and analysis of additional user information using creation rules. For example, explanations may be provided only the first time a particular type of memory query is provided, or after a predetermined number of times a particular type of memory query has been provided. In other aspects, the creation system 122 provides explanations based on the complexity of the provided memory action within the memory query. These aspects are merely exemplary and not intended to be limiting. As those skilled in the art will understand, explanations may be provided for any suitable reason.

[0075] For example, Figure 3A and 3C Examples of different memory queries 132 displayed on user interface 140 are shown. Figure 3A A memory query 132a is shown that identifies a user activity in response to a future doctor's appointment. The memory query 132a notifies the user of the identified activity by stating, "I noticed you have a doctor's appointment today." The memory query 132a also includes a memory action that provides a reminder of the activity by stating, "Would you like me to remember the questions you want to ask the doctor?" Furthermore, the memory query 132a provides additional relevant elements based on analysis of additional user information related to the identified doctor's appointment by stating, "I can retain... to provide you with your last prescription when you arrive at the doctor's office." Additionally, the memory query 132a includes an accept button 142 and a reject button 142 for the recommended memory action of the memory query 132a.

[0076] Figure 3C A memory query 132b is displayed in response to an identification of a user's activity during a 3-day trip to New York. Unlike memory query 132a, this memory query 132b is for the current activity, such as day 1 of the 3-day trip. Memory query 132a notifies the user of the identified activity by stating, "I noticed you are on a 3-day trip to New York and have just completed your first day." Memory query 132b also includes a request to perform an activity-related memory action by stating, "Would you like me to record a summary of how your first day of this trip went so I can help you recall it at any time or add it to your travel journal at the end of the trip?" Furthermore, memory query 132b provides a description of what will be recorded when accepting memory query 132b by stating, "I can help you record things such as the places you visited and which of those places you liked best." Memory query 132b also includes an accept button 142 and a reject button 144 for the suggested memory action of memory query 132b.

[0077] Notification agent 109 determines a suitable time period for providing notification 130 to user 102. Notification agent 109 sends an instruction to client computing device 104 to provide notification 130 to user 102. Notification 130 may include memory query 132 and / or memory retrieval 134 (also referred to herein as recalled memory input). Notification agent 109 collects memory query 132 and / or recalled memory input 134 from activity detection framework 108.

[0078] In response to the creation of memory query 132 by the creation system 122 of the activity detection framework 108 and / or in response to the determination of the query decision system 120 of the activity detection framework 108 determining that a memory query is expected for an activity, the notification agent 109 determines the appropriate time to provide memory query 132 to user 102. The notification agent 109 collects rich contextual elements. In some aspects, the notification agent 109 also collects world knowledge and / or additional user information 128. In response to the query decision system 120 determining that a memory query is expected or in response to the creation of memory query 132 by the creation system 122, the notification agent 109 evaluates rich contextual signals based on notification rules. The notification rules can be updated using learning algorithms based on world knowledge and / or additional user information 128. The ability of the notification agent 109 to update notification rules based on user feedback and / or user patterns allows the notification agent 109 to continuously evolve with user 102 based on user patterns and / or feedback.

[0079] If notification agent 109 determines that the current time period is suitable for providing a memory query to the user, notification agent 109 provides or sends an instruction to one or more user devices 104 to provide memory query 132 to user 102. One or more client computing devices 104 of the user provide memory query 132 to user 102 in response to receiving the instruction from notification agent 109. Client computing devices 104 may provide notification 130, such as memory query 132, to user 104 via any suitable notification medium (such as visual, auditory, tactile, and / or other sensory outputs). For example, client computing devices 104 may use artificial voice intelligence to display memory query 132 and / or verbally state memory query 132.

[0080] If notification agent 109 determines that the current time period is not suitable for providing memory query 132 to user 102, then notification agent 109 does not send an instruction to one or more user devices 104 to provide memory query 132 to user 102, but instead continues to monitor for an appropriate time period for providing memory query 132 to user 102. For example, if notification agent 109 determines, based on analysis of rich contextual elements, that the user is driving or sleeping, then notification agent 109 can determine that the time period is not suitable for displaying notification 130.

[0081] In response to receiving a memory query 132, a user can input a memory input 134 into the user interface 140 of one or more client computing devices 104. A context memory capture and recall system 100 collects the memory input 134 from one or more client computing devices 104. A link system 111 of the context memory capture and recall system 100 collects the memory input 134. The link system 111 is a memory read and write application programming interface (API). The link system 111 reads or understands the received memory input 132. The memory input 134 includes accepting or rejecting a proposed memory action in the memory query. In some aspects, the memory input 134 includes memory information provided by the user. In other aspects, the receipt of memory information is interpreted by the link system 111 as an implicit acceptance of the memory query 132. If the link agent reads the memory input 134 and determines that the memory input is a rejection of the memory query, the link agent 111 sends a rejection to the memory storage device 106.

[0082] If the linking agent reads memory input 134 and determines that the memory input is an acceptance of a memory query, then the linking agent 111 links or associates the memory action with the activity. In some aspects, in response to acceptance of the proposed memory action, the linking agent 111 also collects memory information about the activity. For example, the linking system 111 may request or send an instruction to one or more user devices 104 to request memory information in response to acceptance. One or more user devices 104 provide a memory information request to the user. The user 102 may provide additional memory information via user interface 140 in response to the memory information request. The user device 104 sends the user-inputted memory information to the context memory capture and retrieval system 100. The linking system 111 of the context memory capture and retrieval system 100 collects the user-inputted memory information from the user device 104. The memory information collected by the linking system 111 after acceptance of memory query 132 is considered part of memory input 134.

[0083] In other respects, the linking system 111 analyzes the memory information in the proposed memory action and / or memory input. Based on this analysis, the linking system 111 can collect additional memory information from world knowledge and / or additional user information. This additional memory information collected by the linking system 111 is considered part of the memory input 134.

[0084] Once the linking system 111 has collected all memory inputs, it creates or writes instructions to perform memory actions. In some aspects, the linking system 111 links memory actions to activities by linking or associating them with one or more boundaries of the activity. In other aspects, the linking system 111 links memory actions to activities by linking or associating the storage of memory information with the activity. Memory actions can be the display of memory information input by the user and / or the display of additional memory information collected by the linking system 111. Alternatively, memory actions can be the recording of a specific memory event, such as the detection and recording of memory elements of an activity associated with an identified activity, such as places visited during a trip, things the user liked during a trip, and / or adding the recorded memory elements to a specific storage device or database.

[0085] Once the linking system 111 has created a memory action and linked it to an activity, it sends that information to the memory storage device 106. The memory storage device 106 stores the created memory action and the associated boundaries of one or more activities. The memory storage device 106 can collect and store any rich memory elements of the user. In some aspects, rich memory elements or a portion thereof may be formed by the context memory capture and retrieval system 100. Rich memory elements may include rich context elements formed by the context memory capture and retrieval system 100. In other aspects, rich memory elements or a portion thereof may be formed by a system separate from the context memory capture and retrieval system 100. The context memory capture and retrieval system 100 and / or the pattern detection system 112 collect information from the memory storage device 106.

[0086] The pattern detection framework 112 collects rich memory elements from the user. These rich memory elements include user feedback. Feedback can be explicit or implicit. Explicit feedback from the user is feedback input by the user. For example, explicit user feedback refers to the user selecting or providing input that indicates whether a particular memory query or memory action is helpful or not. Conversely, implicit feedback is determined by monitoring user behavior in response to provided memory actions. For example, the selection / non-selection of presented memory information, the duration of use, and / or the usage pattern of provided memory actions can be monitored to determine user feedback 148.

[0087] In some aspects, the pattern detection framework 112 can collect user signals from one or more signal generators and enrich the user signals to form rich memory elements. In other aspects, rich memory elements are collected from systems separate from and distinct from the pattern detection framework 112. Rich memory elements are formed by utilizing world knowledge 110 to enrich the elements in the user signals collected from the signal generators. User signals include all types of user elements, such as user context elements.

[0088] The pattern detection framework 112 includes a mapping system 126 and a recommendation system 124. The mapping detection system 126 determines user patterns by mapping a user's rich memory elements to different activities. The recommendation engine 124 can utilize the determined user patterns to recommend additional memory information for memory retrieval and / or updates to the recommendation learning algorithm.

[0089] In some respects, the mapping system 126 of the pattern detection framework analyzes activities. The mapping system 126 analyzes activities by mapping them to rich memory elements of different activities. The recommendation engine 124 identifies related activities based on the analysis of the activities. The recommendation engine 124 links or associates one or more boundaries of related activities with memory actions created for the activities. In some respects, related activities are future activities. The recommendation engine sends recommendations to the activity state detection framework 108 to monitor one or more boundaries of related activities, and provides memory actions for the activities in response to the detection of one or more boundaries of related activities. Alternatively, the recommendation engine sends recommendations to the activity state detection framework 108 to monitor one or more boundaries of related activities, and provides new memory queries related to the activity in response to the detection of one or more boundaries of related activities. The new memory query may request the recording of memory information related to the activity and associate that memory information with the memory information of the activity and / or related activities. Alternatively, the new memory query may request approval to display memory information related to the activity in response to the detection of a boundary of a related activity.

[0090] For example, the activity state detection framework 108 can detect the boundaries of relevant activities occurring during the main activities of a 3-day trip to New York (such as...). Figure 3C (See memory query 132c). In this example, in response to detecting the boundary of a related activity (such as a trip to the Metropolitan Museum of Art), memory information about the related activity (such as pictures taken at the museum) is recorded and associated with the main activity (a 3-day trip to New York) and / or the related activity. Alternatively, in this example, in response to detecting the boundary of a related activity, a memory query is provided that requests permission to record memory information about the related activity and associates the memory information with the related activity and / or the main activity (a 3-day trip to New York).

[0091] In response to receiving recommendations from the recommendation engine 124 of the pattern detection framework 112, the activity detection framework 108 can monitor and detect one or more boundaries of related activities based on rich contextual elements. If the activity detection framework 108 does not detect one or more boundaries of related activities, it continues to monitor one or more boundaries. If the activity detection framework 108 detects one or more boundaries of related activities, it re-executes the memory action created in response to the activity in response to the detection of one or more boundaries of related activities. Alternatively, if the activity detection framework 108 detects one or more boundaries of related activities, it presents a memory query requesting permission to re-execute the memory action created in response to the main activity in response to the detection of one or more boundaries of related activities. If the memory action includes displaying a notification, the activity framework 108 can send the notification to a notification broker as described above.

[0092] Pattern detection framework 112 collects or shares any identified user patterns, user feedback, rich memory elements, and / or recommendations as supplementary user information to memory storage device 106 for storage. Furthermore, pattern detection framework 112 sends any identified user patterns, user feedback, rich memory elements, and / or recommendations as supplementary user information to activity detection framework 108 or shares it with activity detection framework 108.

[0093] Activity detection framework 108 receives instructions to detect one or more boundaries of an activity. Activity boundaries may include the start of an activity, the start time of an activity, the occurrence of an activity, and / or the completion of an activity. Activity detection framework 108 detects one or more activity boundaries based on analysis of rich contextual signals. If activity detection framework 108 does not detect a boundary of an action linked to a memory action, activity state detection framework 108 continues to monitor activity boundaries. If activity detection framework 108 detects a boundary of an activity linked to a memory action, activity state detection framework 108 executes the memory action linked to that activity. If the memory action requires sending a notification to the user, such as displaying memory information, activity state detection framework 108 sends the notification to notification agent 109. The display of memory information may also be referred to herein as memory recall 128.

[0094] As described above, notification agent 109 determines a suitable time period for providing notification 130 to user 102. Notification agent 109 provides or sends an instruction to client computing device 104 to provide notification 130 to the user. Alternatively, notification 140 required for the memory action is automatically provided to client computing device 104 by notification agent 109 upon receipt.

[0095] Figure 3BThe memory input 128 is shown to display a recall of activity for a first identifier in response to boundary detection of activity for the first identifier, according to various aspects of this disclosure. Figure 3A An example of a user interface. As described above, the identified activity is a doctor's appointment. The boundary associated with the doctor's appointment can be the start of the activity, the start time of the activity, or before the activity occurs. The recalled memory input 128b includes memory information entered by the user in response to memory query 132a. The memory information entered by the user includes questions from the doctor regarding his or her doctor's appointment. In some embodiments, when the user searches for an activity, the memory information associated with the memory input recall 128 can be requested and / or edited by the user before the linked activity boundary occurs.

[0096] Figure 4 A flowchart is shown, conceptually illustrating an example of a method 400 for contextual memory capture and retrieval. In some aspects, method 400 is performed by a contextual memory capture and retrieval system 100 as described above. Compared to previously used applications (which do not provide memory queries based on detected activities, or do not link memory actions to detected activities based on the receipt of memory input in response to memory queries), method 400 provides an application that enhances the user's ability to complete tasks, improves the retrieval of desired memory items, and improves the usability, performance, and / or user interaction with the application. For example, even before the user realizes that storing the memory information would be beneficial, method 400 may ask the user whether they expect the memory information for later use.

[0097] Method 400 includes operation 402. In operation 402, user signals, including user context signals, are collected. User signals are collected from a signal generator. In some aspects, the signal generator is at least one of a client device, a server, and / or an application. User signals include memory elements and / or digital artifacts. As used herein, digital artifacts are converted into elements using world knowledge and / or other user information. In some aspects, memory elements and / or digital artifacts in user signals may include user feedback, GPS coordinates, photos, browser history, emails, text messages, social data, notebooks, to-do lists, calendar items, professional data, events, and / or application data. As used herein, digital artifacts are data that the user will not recognize as memory elements before further processing. For example, digital artifacts may include GPS coordinates or computer-coded data. In operation 402, user signals are continuously collected under predetermined conditions and / or after a predetermined amount of time. Thus, new or updated user context, user patterns, user feedback, activities, etc., can be determined based on each newly received user signal during method 400.

[0098] User context signals indicate the current state of the user or user context. Over time, as the environment changes, the user's location changes, user behavior changes, and / or the user's physical actions change, the user context is constantly changing and / or being updated. As mentioned above, the current state of the user or the current user context is based on user context signals such as the user's current location, current time, current weather, the user's current digital behavior, and / or the user's current physical actions.

[0099] Next, method 400 includes operation 404. In operation 404, world knowledge is used to enrich memory elements, including user context elements, to form rich elements. In some aspects, in operation 404, world knowledge and / or other user information are used to convert digital artifacts in the user signal into memory elements, such as context elements. In operation 404, world knowledge and / or other user information may also be used to enrich these memory elements to form rich elements.

[0100] In some aspects, method 400 includes operation 406. In operation 406, rich memory elements are analyzed to determine user patterns and / or user feedback. In some aspects, operation 406 updates or trains machine learning techniques and / or statistical modeling techniques of the learning algorithm based on the determined user feedback and / or user patterns. In other aspects, user feedback and / or user patterns are added to the rich memory elements.

[0101] Those skilled in the art will understand that operations 402, 404, and / or 406 can be performed sequentially by method 400. Furthermore, those skilled in the art will understand that operations 402, 404, and / or 406 can overlap with the execution of other operations of method 400. For example, method 400 can continue to collect and enrich user signals at operations 402 and 404 during the creation of a memory query at operation 410.

[0102] Following operations 404 and / or 406, operation 407 is performed during method 400. In operation 407, user activities are identified based on rich contextual elements, world knowledge, and / or other user information. In some aspects, operation 407 utilizes rich contextual elements, world knowledge, and / or other user information to identify activities related to the main activity. The main activity used herein refers to the activity previously identified at operation 407. Related activities used herein refer to any activity associated with or linked to the main activity. Activities and / or related activities can be future or current activities. Activities used herein refer to any event, place, and / or occasion related to the user. For example, an activity could include a relative's birthday, a future doctor's appointment, a vacation, arriving at a work meeting, etc. Activities are identified by analyzing rich contextual elements, world knowledge, and / or other user information based on a set of contextual rules. In some aspects, learning algorithms are used to update the contextual rules.

[0103] Following operation 407, operation 408 is performed during method 400. In operation 408, the activity identified by operation 407 is evaluated to determine whether the memory query is expected and / or appropriate. In some aspects, in operation 408, the activity is evaluated given a set of query rules. In some aspects, a learning algorithm is used to update the query rules. In these aspects, if the activity satisfies one or more query rules, then the memory query is expected or appropriate for the detected activity. In these aspects, if the activity does not satisfy one or more query rules, then the memory query is unexpected or inappropriate for the detected activity. If it is determined at operation 408 that the memory query is expected or appropriate, then operation 410 is performed. If it is determined at operation 408 that the memory query is unexpected or inappropriate, then operation 410 is not performed and method 400 may continue to perform operation 402.

[0104] Next, operation 410 is performed. In operation 410, in response to determining in operation 408 that a memory query is desired for an activity, a memory query is created or formed based on the activity. The memory query is created by analyzing the activity, world knowledge, and / or other user information using a set of creation rules. In some aspects, a learning algorithm is used to update the creation rules. The memory query created at operation 410 includes the identified activity and a request to perform one or more memory actions related to the identified activity. Memory actions may include a request to collect memory information related to the activity from the user and / or from world knowledge and / or other user information. In other aspects, the memory query includes recommended memory information related to the identified activity. The recommended memory information may be identified based on world knowledge, rich user context elements, and / or other user information. In these aspects, memory actions may include a request to provide recommended memory elements listed in the memory query in the memory recall of the activity. In other aspects, a memory action is a request to identify and store one or more memory elements of an activity for memory recall of an activity. In other aspects, a memory action is a request to identify and store one or more memory elements of a related activity for memory recall of a main activity.

[0105] In some aspects, method 400 includes operation 412. In operation 412, it is determined whether the current time period is suitable for providing a memory query. In these aspects, in operation 412, rich user context elements are evaluated to determine whether the current time period is suitable for providing a memory query. The rich user context elements may be evaluated using notification rules. In some aspects, a learning algorithm is used to update the notification rules. In these aspects, if the rich context elements satisfy one or more notification rules, it is determined that the current time period is suitable for providing a memory query and operation 413 is performed. In these aspects, if the rich context elements do not satisfy one or more notification rules, it is determined that the current time period is not suitable for providing a memory query, and the received rich user context elements are evaluated until an appropriate time period is determined, or until at operation 412 the memory query is no longer appropriate or desired.

[0106] After performing operations 410 and / or 412, operation 413 is performed during method 400. In operation 413, a memory query is provided to the user, or an instruction to provide a memory query to the user is sent to one or more client computing devices.

[0107] Next, in operation 414, in response to a memory query, memory input is collected from at least one client computing device. The memory input is entered by a user into the user interface of the client computing device. The memory input includes accepting or rejecting suggested memory actions included in the memory query. The memory input also includes any memory information entered by the user via the user interface of the client device in response to the memory query. In some aspects, the user's input of memory information in response to the memory query is considered an acceptance of a memory action in the memory query.

[0108] In some aspects, method 400 includes operations 416 and 417. In operation 416, a learning algorithm is used to evaluate the user's memory actions and / or any received memory information, based on world knowledge and / or other user information, to determine if any supplementary memory information should be added to the memory information utilized by the memory action. If supplementary memory information is detected at operation 416, operation 417 is performed. If no supplementary memory information is detected at operation 416, operation 418 is performed.

[0109] In operation 417, add the identified supplementary memory information associated with the activity and store it together with any memory information already associated with the memory action.

[0110] Next, operation 418 is performed. In operation 418, in response to the acceptance of a memory action in the received memory input and / or in response to the creation of a memory query, the memory action is linked to an activity. In other words, the memory action is associated with an activity. In some aspects, the memory action is linked to an activity by linking it to one or more boundaries of the activity. For example, a memory action for recalling memory information can be linked to an activity, so that the memory information can be found by searching the activity, or the memory information can be presented when one or more boundaries of the activity are detected.

[0111] In operation 420, activity boundaries linked to the activity are detected based on rich contextual signals. As mentioned above, activity boundaries can occur before the start of the activity, the start time of the activity, the occurrence of the activity, and / or the completion of the activity. Activity boundaries can be determined by evaluating rich contextual elements using a learning algorithm at operation 420. If one or more boundaries associated with the activity are detected at operation 420, operation 422 is executed. If one or more boundaries associated with the activity are not detected at operation 420, operation 420 continues to monitor newly received rich contextual signals for one or more boundaries.

[0112] In some aspects, method 400 includes operation 422. In operation 422, it is determined whether the current time period is suitable for delivering a memory action. In these aspects, rich user context elements are evaluated at operation 422 to determine whether the current time period is suitable for delivering a memory action. Notification rules may be used to evaluate the rich user context elements. In some aspects, a learning algorithm is used to update the notification rules. In these aspects, if the rich context elements satisfy one or more notification rules, it is determined that the current time period is suitable for performing a memory action and operation 424 is performed. In these aspects, if the rich context elements do not satisfy one or more notification rules, it is determined that the current time period is not suitable for performing a memory action, and operation 422 continues to monitor the received rich user context elements until an appropriate time period is determined or until the memory action is no longer appropriate or desired.

[0113] After performing operations 420 and / or 422, operation 424 is performed during method 400. In operation 424, a memory action is performed or an instruction to perform a memory action is sent to one or more client computing devices.

[0114] In some aspects, method 400 includes operation 426. In operation 426, user feedback is requested and / or determined in response to the performed memory action. As mentioned above, the user feedback can be implicit or explicit. Any determined user feedback at operation 426 is provided to operation 402 as a user signal.

[0115] Figure 5-8 The associated description provides a discussion of various operating environments in which the aspects of this disclosure can be implemented. However, regarding Figure 5-8 The devices and systems shown and discussed are for illustrative purposes only and are not intended to limit the wide range of computing device configurations that may be used to implement various aspects of this disclosure, as described herein.

[0116] Figure 5 This is a block diagram illustrating the physical components (e.g., hardware) of a computing device 500 that can implement various aspects of this disclosure. For example, a context memory capture and recall system 100 may be implemented by the computing device 500. In some aspects, the computing device 500 is a mobile phone, smartphone, tablet computer, phablet, smartwatch, wearable computer, personal computer, desktop computer, gaming system, laptop computer, etc. The computing device components described below may include computer-executable instructions for the context memory capture and recall system 100, which can be executed to employ method 400 to construct and / or use AI that simulates the human brain, as disclosed herein.

[0117] In a basic configuration, computing device 500 may include at least one processing unit 502 and system memory 506. Depending on the configuration and type of the computing device, system memory 506 may include, but is not limited to, volatile storage devices (e.g., random access memory), non-volatile storage devices (e.g., read-only memory), flash memory, or any combination of these memories. System memory 506 may include an operating system 505 and one or more program modules 506 suitable for running software application 520. For example, operating system 505 may be suitable for controlling the operation of computing device 500. Furthermore, aspects of this disclosure may be implemented in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. This basic configuration is in... Figure 7 The components are shown in dashed line 508. The computing device 500 may have additional features or functions. For example, the computing device 500 may also include additional data storage devices (removable and / or non-removable), such as, for example, disks, optical discs, or magnetic tapes. Such additional storage devices... Figure 5 It is illustrated by removable storage device 509 and non-removable storage device 510.

[0118] As described above, multiple program modules and data files can be stored in system memory 504. When executed on processing unit 502, program module 506 (e.g., context memory capture and recall system 100) can perform processing, including but not limited to performing the methods described herein 400. For example, processing unit 502 can implement context memory capture and recall system 100. Other program modules that can be used according to various aspects of this disclosure, and particularly for generating screen content, may include digital assistant applications, speech recognition applications, email applications, social networking applications, collaboration applications, enterprise management applications, messaging applications, word processing applications, spreadsheet applications, database applications, presentation applications, contact applications, game applications, e-commerce applications, e-commerce applications, transaction applications, exchange applications, device control applications, web interface applications, calendar applications, etc. In some aspects, context memory capture and recall system 100 constructs a user-centric memory map for one or more of the above applications.

[0119] Furthermore, aspects of this disclosure can be implemented in circuits including discrete electronic components, packages or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or single chips containing electronic components or microprocessors. For example, aspects of this disclosure can be implemented via a system-on-a-chip (SOC), wherein... Figure 5Each or many of the components shown can be integrated onto a single integrated circuit. Such a SoC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “burned in”) as a single integrated circuit onto a chip substrate. When operating via the SoC, the capabilities described herein regarding the client switching protocol can be operated via application-specific logic integrated with other components of the computing device 500 on the single integrated circuit (chip).

[0120] The aspects of this disclosure can also be implemented using other techniques capable of performing logical operations (such as, for example, AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, the aspects of this disclosure can be implemented within a general-purpose computer or in any other circuit or system.

[0121] The computing device 500 may also have one or more input devices 512, such as a keyboard, mouse, pen, microphone or other sound or voice input device, touch or swipe input device, etc. Multiple output devices 514, such as a monitor, speakers, printer, etc., may also be included. The above devices are examples, and other devices may be used. The computing device 500 may include one or more communication connections 516 that allow communication with other computing devices 550. Examples of suitable communication connections 516 include, but are not limited to, RF transmitters, receivers and / or transceiver circuitry; Universal Serial Bus (USB), parallel and / or serial ports.

[0122] The terms computer-readable medium or storage medium as used herein can include computer storage media. Computer storage media can include volatile and non-volatile removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 504, removable storage device 509, and non-removable storage device 510 are examples of computer storage media (e.g., memory storage devices). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices, or other magnetic storage devices, or any other article of manufacture that can be used to store information and is accessible by computing device 500. Any such computer storage medium may be part of computing device 500. Computer storage media may not include carrier waves or other propagated or modulated data signals.

[0123] Communication media can be implemented by computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and include any information transmission medium. The term "modulated data signal" can describe a signal whose one or more characteristics are set or altered such that information can be encoded in the signal. As an example, and not a limitation, communication media can include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0124] Figure 6A and 6B A mobile computing device 600 is shown, such as a mobile phone, smartphone, tablet computer, phablet, smartwatch, wearable computer, personal computer, desktop computer, gaming system, laptop computer, etc., which can be used to implement various aspects of this disclosure. Reference Figure 6A This illustrates one aspect of a mobile computing device 600 suitable for implementing these aspects. In a basic configuration, the mobile computing device 600 is a handheld computer with input and output elements. The mobile computing device 600 typically includes a display 605 and one or more input buttons 610 that allow the user to input information into the mobile computing device 600. The display 605 of the mobile computing device 600 can also be used as an input device (e.g., a touchscreen display).

[0125] If included, the optional side input element 615 allows for additional user input. The side input element 615 can be a rotary switch, a button, or any other type of manual input element. Alternatively, the mobile computing device 600 can include more or fewer input elements. For example, in some aspects, the display 605 may not be a touchscreen. In yet another alternative aspect, the mobile computing device 600 is a portable telephone system, such as a cellular phone. The mobile computing device 600 may also include an optional keypad 635. The optional keypad 635 can be a physical keypad or a “soft” keypad generated on a touchscreen display.

[0126] In addition to or replacing the touchscreen input device associated with display 605 and / or keypad 635, a Natural User Interface (NUI) may be incorporated into mobile computing device 600. As used herein, NUI includes any interface technology that enables users to interact with the device in a “natural” manner without the artificial constraints imposed by input devices such as a mouse, keyboard, or remote control. Examples of NUI methods include those that rely on speech recognition, touch and stylus recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence.

[0127] In various aspects, output elements include a display 605 for displaying a graphical user interface (GUI). In the aspects disclosed herein, various sets of user information may be displayed on the display 605. Other output elements may include a visual indicator 620 (e.g., a light-emitting diode) and / or an audio transducer 625 (e.g., a speaker). In some aspects, the mobile computing device 600 includes a vibration transducer for providing tactile feedback to a user. In yet another aspect, the mobile computing device 600 includes input and / or output ports for sending signals to or receiving signals from external devices, such as audio inputs (e.g., a microphone jack), audio outputs (e.g., a headphone jack), and video outputs (e.g., an HDMI port).

[0128] Figure 6B This is a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, the mobile computing device 600 may include a system (e.g., architecture) 602 to implement several aspects. In one aspect, the system 602 is implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In other aspects, the system 602 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

[0129] One or more applications 666 and the context memory capture and recall system 100 run on or in association with the operating system 664. Examples of applications include telephone dialers, email programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, etc. System 602 also includes a non-volatile storage area 668 within memory 662. The non-volatile storage area 668 can be used to store persistent information that should not be lost when system 602 is powered off. Applications 666 can use and store information, such as emails or other messages used by email applications, in the non-volatile storage area 668. A synchronization application (not shown) also resides on system 602 and is programmed to interact with a corresponding synchronization application residing on the host computer to keep the information stored in the non-volatile storage area 668 synchronized with the corresponding information stored on the host computer. It should be understood that other applications can be loaded into memory 662 and run on mobile computing device 600.

[0130] System 602 has a power supply 670, which can be implemented as one or more batteries. The power supply 670 may also include an external power source, such as an AC adapter or power docking bracket for supplementing or recharging the batteries.

[0131] System 602 may also include a radio 672 that performs the functions of transmitting and receiving radio frequency communications. Radio 672 supports a wireless connection between system 602 and the "external world" via a communications operator or service provider. Transmissions to and from radio 672 are conducted under the control of operating system 664. In other words, communications received by radio 672 can be propagated to application 666 via operating system 664, and vice versa.

[0132] A visual indicator 620 can be used to provide visual notifications, and / or an audio interface 674 can be used to generate audible notifications via an audio transducer 625. In the illustrated aspect, the visual indicator 620 is a light-emitting diode (LED), and the audio transducer 625 is a speaker. These devices can be directly coupled to a power supply 670 such that they remain on for a duration indicated by the notification mechanism when activated, even if the processor 660 and other components may be turned off to conserve battery power. LEDs can be programmed to remain on indefinitely until the user takes an action to indicate the device's power-on status. The audio interface 674 is used to provide and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 674 can also be coupled to a microphone to receive audible input. System 602 may also include a video interface 676 that enables the operation of the onboard camera 630 to record still images, video streams, etc.

[0133] The mobile computing device 600 implementing system 602 may have additional features or functions. For example, the mobile computing device 600 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage devices... Figure 6B The non-volatile storage region 668 is shown in the middle.

[0134] As described above, data / information generated or captured by mobile computing device 600 and stored via system 602 can be stored locally on mobile computing device 600, or the data can be stored on any number of storage media that can be accessed by the device via radio 672 or via a wired connection between mobile computing device 600 and separate electronic devices associated with mobile computing device 600 (e.g., server computers in distributed computing networks such as the Internet). It should be understood that such data / information can be accessed via radio 672 or via a distributed computing network through mobile computing device 600. Similarly, according to known data / information transmission and storage devices, including email and collaborative data / information sharing systems, such data / information can be easily transferred between computing devices for storage and use.

[0135] Figure 7One aspect of the architecture of a system for processing data received at a computing system from a remote source (such as a general-purpose computing device 704, a tablet computer 706, or a mobile device 708) is shown, as described above. Content displayed and / or utilized at server device 702 can be stored in different communication channels or other storage types. For example, various documents can be stored using a directory service 722, a web portal 724, an email service 726, an instant messaging storage device 728, and / or a social networking site 730. As an example, a context memory capture and retrieval system 100 can be implemented in a general-purpose computing device 704, a tablet computing device 706, and / or a mobile computing device 708 (e.g., a smartphone). In some aspects, server 702 is configured to communicate via, for example, Figure 7 The network 715 shown implements the context memory capture and recall system 100.

[0136] Figure 8 An exemplary tablet computing device 800 is shown in which one or more aspects disclosed herein can be performed. Furthermore, the aspects and functions described herein can operate on a distributed system (e.g., a cloud-based computing system), wherein application functions, memory, data storage and retrieval, and various processing functions can operate remotely to each other via a distributed computing network such as the Internet or an intranet. User interfaces and various types of information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information can be displayed and interacted with on a wall, on which user interfaces and various types of information are projected. Interaction with multiple computing systems that can implement aspects of the present invention includes keystroke input, touchscreen input, voice or other audio input, gesture input, wherein the associated computing device is equipped with detection (e.g., camera) functions for capturing and interpreting user gestures, etc., for controlling the functions of the computing device.

[0137] For example, embodiments of the present disclosure have been described above with reference to block diagrams and / or operational illustrations of methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions recorded in the blocks may not occur in the order shown in any flowchart. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions / actions involved.

[0138] This disclosure describes some embodiments of the present technology with reference to the accompanying drawings, in which only some possible aspects are depicted. However, other aspects may be implemented in many different forms, and the specific aspects disclosed herein should not be construed as limited to the aspects of this disclosure set forth herein. Rather, these exemplary aspects are provided so that this disclosure is thorough and complete and fully conveys the scope of other possible aspects to those skilled in the art. For example, aspects of the various aspects disclosed herein may be modified and / or combined without departing from the scope of this disclosure.

[0139] Although specific aspects have been described herein, the scope of this technology is not limited to these specific aspects. Those skilled in the art will recognize other aspects or improvements within the scope and spirit of this technology. Therefore, specific structures, actions, or media are disclosed only as illustrative aspects. The scope of this technology is defined by the following claims and any of their equivalents.

Claims

1. A system for contextual memory capture and retrieval, the system comprising: At least one processor; as well as Memory for storing and encoding computer-executable instructions, which, when executed by the at least one processor, operate to: Context elements are generated based on context signals associated with the user; The user's activity is identified based on the context element, wherein the activity is a future activity; Based on the query rules and the boundaries of the activity, it is determined that a memory query for the activity is desired, and the memory query includes suggested memory actions; The memory query is created based on the activity; In response to the memory query, memory input is collected, wherein the memory input includes memory information and the user's acceptance of the suggested memory action; In response to the acceptance of the suggested memory action, a memory action is generated based on the collected memory input, wherein the generated memory action includes the recall of the memory information; Detect the first boundary of the activity based on the context element; as well as In response to the detection of the first boundary, the generated memory action is sent to the user's client computing device, wherein the generated memory action includes the recall of the memory information.

2. The system according to claim 1, wherein the at least one processor is further operable to: Enrich the context elements from the aforementioned context signals by utilizing world knowledge to form a rich set of context elements; Determining user patterns based on user signals; and The query rules are updated based on the user pattern using a learning algorithm.

3. The system according to claim 1, wherein the at least one processor is further operable to: Receive additional user information from the user pattern detection framework, wherein the additional user information is at least one of user patterns or user feedback; and The query rules are updated based on the additional user information using a learning algorithm.

4. The system of claim 1, wherein the at least one processor is further operable to: Analyze user signals to determine user feedback; and The query rules are updated based on the user feedback using a learning algorithm.

5. The system of claim 1, wherein the at least one processor is further operable to: Receive user feedback and user patterns from the user pattern detection framework; and The query rules are updated based on the user feedback and user patterns using a learning algorithm.

6. The system of claim 1, wherein the at least one processor is further operable to: The query rules, creation rules, and notification rules are updated using a learning algorithm based on at least one of user patterns and user feedback.

7. The system of claim 1, wherein the at least one processor is further operable to: The generated memory action is linked to the first boundary of the activity, wherein the first boundary is associated with the occurrence of the user's activity, and wherein the first boundary precedes the start of the activity.

8. The system of claim 1, wherein the at least one processor is further operable to: Analyze the memory input; Based on the analysis of the memory input, additional memory information is collected from world knowledge; Link the additional memory information to the first boundary of the activity; as well as In response to the detection of the first boundary, the generated memory action is sent to the user's client computing device, wherein the generated memory action further includes the recall of the additional memory information.

9. The system of claim 1, wherein the at least one processor is further operable to: Analyze the activities described; The relevant activities are determined based on the analysis of the activities described; Link the generated memory action to the second boundary of the related activity; Enrich the context elements from the context signals by utilizing world knowledge to form a rich set of context elements; The second boundary of the relevant activity is detected based on the rich context elements; In response to the detection of the second boundary, the generated memory action is sent to the user's client computing device, wherein the generated memory action includes the retrieval of the memory information.

10. The system of claim 1, wherein the at least one processor is further operable to: Enrich the context elements from the context signals using world knowledge to form a rich set of context elements; Based on the rich contextual elements, a suitable time period for providing the memory query is determined; and The memory query is sent to the user's client computing device within the appropriate time period.

11. A method for contextual memory capture and retrieval, the method comprising: Context elements are generated based on context signals associated with the user; The user's activity is identified based on the context element, wherein the activity is a future activity; Based on the query rules and the boundaries of the activity, it is determined that a memory query for the activity is desired, and the memory query includes suggested memory actions; Collect memory input associated with the memory query, wherein the memory input includes memory information and acceptance of the suggested memory action; In response to the acceptance of the suggested memory action, a memory action is generated, the memory action including recall associated with the memory input; as well as Based on the occurrence of at least one boundary of the activity associated with the generated memory action including the recall, the generated memory action is sent to the user's client computing device, wherein the generated memory action includes the recall of memory information.

12. The method of claim 11, further comprising: Enrich the context elements from the context signals using world knowledge to form a rich set of context elements; Detect the first boundary of the activity based on the rich context elements; as well as In response to the detection of the first boundary, the generated memory action is executed.

13. The method of claim 12, wherein the memory retrieval further includes the activity and a request for memory information associated with the activity.

14. The method of claim 13, wherein the memory input includes the memory information relating to the activity.

15. The method of claim 12, further comprising: Using world knowledge and additional user information, additional memory information related to the activity is identified based on the analysis of the activity. The memory retrieval includes the activity, the additional memory information associated with the activity, and a request for approval to use the additional memory information in memory retrieval related to the activity.

16. The method of claim 15, wherein the memory input comprises: Approval was granted for creating the memory recall for the activity.

17. The method of claim 15, wherein the memory retrieval further comprises a request for memory information related to the activity, and The memory input also includes the memory information related to the activity from the user.

18. A system for contextual memory capture and retrieval, the system comprising: At least one processor; as well as Memory for storing and encoding computer-executable instructions, which, when executed by the at least one processor, operate to: Context elements are generated based on context signals associated with the user; The user's activity is identified based on the context element, and the activity is a future activity; Based on the query rules, it is determined that a memory query for the activity is desired based on the boundaries of the activity, and the memory query includes suggested memory actions; Collect memory input associated with the memory query, wherein the memory input includes memory information and acceptance of the suggested memory action; In response to acceptance of the suggested memory action, a memory action is generated, the memory action including recall associated with the memory information in the memory input; and The generated memory action is sent to the user's client computing device based on the occurrence of the boundary of the activity associated with the generated memory action that includes the memory information.

19. The system of claim 18, further operable to: The context signals are collected from at least one of the user's client computing devices to form the context elements; Enrich the contextual elements with world knowledge to form a richer contextual element set; The boundary is detected based on the rich context elements; as well as In response to the detection of the boundary, an instruction is sent to the at least one client computing device to execute the generated memory action.

20. The system of claim 19, further operable to: Based on additional user information identifiers, the activity is a related activity for the activity, wherein the related activity is a future activity; Link the generated memory action to the second boundary of the related activity; Detect the second boundary; as well as In response to the detection of the second boundary, the generated memory action is executed.

Images