Systems and methods for measuring and enhancing human engagement and cognition

JP2023550846A5Inactive Publication Date: 2024-11-15KINOO INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023549966
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-10-27
Filing Date
2021-10-26
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing educational technologies struggle to accurately assess knowledge acquisition and emotional engagement in real-time, particularly in distance learning environments, leading to superficial understanding and ineffective learning strategies.

Method used

The use of human interaction entities (HIEs) instantiated with artificial intelligence (AI) to monitor and regulate emotional states and knowledge acquisition through real-time interaction analysis, utilizing neural networks and classifiers to identify indicators of engagement and understanding, and provide personalized learning experiences.

Benefits of technology

Enhances emotional and cognitive learning environments by maintaining optimal engagement levels and providing real-time feedback for effective knowledge acquisition, improving learning outcomes and reducing redundancy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Systems and methods are described that implement machine-based simultaneous classification of an individual's emotional and cognitive states in substantially real time. A bootstrap approach is used to identify indicators of emotional states, including boredom, immersion, and feeling overwhelmed, by finding additional interactions temporally adjacent to known indicators. A similar bootstrap approach is used to identify new indicators of knowledge acquisition (or confusion) by finding frequent, temporally adjacent interactions. The systems and methods can enhance instruction by identifying knowledge presentations that result in successful knowledge acquisition and enhance learning by allowing instructors associated with individuals to track material that is understood (or misunderstood). Machine-based tracking of learning can avoid boredom by avoiding unnecessary repetition and facilitate the teaching of synergistic topics related to recently understood material. The systems and methods can provide an enhanced, personalized environment for both emotional and cognitive development.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Related Applications This application claims the benefit of co-pending Provisional Application No. 63 / 106,296, filed October 27, 2020.

[0002] The present invention generally relates to systems and methods for providing supportive emotional engagement and cognitive learning environments using one or more electronic devices or human interaction entities (HIEs). Optionally, the HIE may be instantiated with one or more artificial intelligences (AIs) to discern human intent and / or context in real-time during human-machine interaction. The HIE may also represent an AI personality (AIP) familiar to the user during such interaction. The systems and methods herein utilize techniques within the fields of computer programming, machine learning [including artificial neural networks (ANNs), convolutional neural networks (CNNs), word embeddings, and deep learning], human-machine interface (HMI), telecommunications, and cognitive science (including psychology, linguistics, and learning). The systems and methods herein may also provide one or more users with intuitive machine interfaces and / or companions for 1) monitoring and adjusting emotional engagement and 2) demonstrating expertise to encourage and measure the acquisition of new knowledge.

Background Art

[0003] Within the field of cognitive science, it is widely accepted that emotions, including feelings, play a crucial role in learning, memory, logical thinking, and problem-solving. Some researchers even argue that learning is impossible without emotional "tags." Recent advances in neuroscience have improved our understanding of the role of emotions in learning to the point where we can identify specific neurotransmitters, such as dopamine (i.e., 3,4-dihydroxyphenethylamine) and serotonin (i.e., 5-hydroxytryptamine), as the brain's emotional reward system during learning, cognition, and memory formation.

[0004] On the other hand, excessive and / or prolonged stimulation of the sympathetic nervous system, particularly the release of various catecholamines that are part of the so-called "fight-or-flight" response, leads to a significant reduction in the ability to learn and / or form long-term memories. The so-called "inverted U-shaped" relationship between overall mental performance (including learning) and the emotional "pressure" (a term used in the past) perceived by an individual has been accepted in the field of psychology for over a century. Robert Yerkes and John Dodson first described this inverted U-shaped relationship in 1908 [Yerkes RM, Dodson JD (1908) The relation of strength of stimulus to rapidity of habit-formation, J Comp Neurol Psychol 18:459-482]. As a result, an individual's emotional engagement is generally seen as a crucial element of a productive learning environment.

[0005] Active learning is a complex lifelong process involving the acquisition of new knowledge, behaviors, skills, values, attitudes, and / or preferences. Understanding is an element of knowledge acquisition that relates to both abstract and physical concepts, thereby enabling us to "think about" or consider those concepts. Understanding is often closely linked to the ability to reason (i.e., the steps of logical thinking about such concepts that lead to logical consequences). It is generally accepted that the learning process has at least six dialogue components: attention, memory, language, processing / organization, writing-motor (i.e., writing), and higher-order thinking. These processes interact not only with each other, but also with the physical learning environment, behaviors, social skills, and the individuals involved in the learning process (e.g., parents, friends, mentors).

[0006] Traditionally, and perhaps too general and vague, learning has been categorized as arising from three categories of interaction: 1) instructors (e.g., teachers providing continuing education, parents or other family members, or professional leaders); 2) other learners (i.e., peer communication and / or instruction, with or without the involvement of a formal instructor); and 3) content (currently presented using hardware tools, including a wide range of software applications, books, audio broadcasts, and screen-based and voice-first devices). However, advanced technologies and distance learning approaches have significantly blurred these distinctions.

[0007] As an additional framework for considering the learning process, individual learning styles are generally considered to fall within at least five overlapping categories: 1) visual, 2) auditory, 3) reading and writing, 4) olfactory, and 5) kinesthetic (i.e., movement of body parts). These diverse learning styles highlight the need for personalized modes of presenting new knowledge and review using personalized techniques to assess whether the knowledge has been sufficiently acquired.

[0008] Different forms of instructional dialogue may support different learning styles. In practice, resources introducing common instructional topics are often available in formats that convey such preferences and / or styles. For example, topic descriptions may be available as text in a book (or on one of the many devices that can display text on a screen), pencasts, a series of images, audio-visual clips illustrating knowledge elements with examples, animations illustrating underlying concepts, podcasts, and / or audiobook snippets.

[0009] The formation of long-term memory is an element of learning and an indicator of a healthy emotional environment. Based on magnetic resonance imaging and other brain monitoring techniques, it is generally believed that the organization and formation of long-term memory occurs through the interconnections between the frontal lobe (working memory) with the occipital lobe (visual cognition), the temporal lobe (language and conceptual knowledge), the somatosensory cortex (touch and bodily sensation), and other areas of the cortex (auditory processing). The frontal lobe also plays a crucial role during the concentration and attention required for memory formation and retrieval.

[0010] Other areas of the body, such as the adrenal glands located above each kidney, contribute to an individual's emotional state through the release of neurohormones such as epinephrine (i.e., adrenaline) and norepinephrine (i.e., noradrenaline). Each of these anatomical regions serves multiple functional roles beyond its involvement in regulating emotional states. Recognizing that these anatomical structures are simultaneously involved in their diverse functional roles and specific interconnected pathways that result in identifiable limitations in communication between different regions can help in constructing (and limiting) emotional and cognitive strategies that facilitate the overall development of emotion and intelligence.

[0011] A crucial element in the educational process is assessing whether learners understand the presented knowledge. Traditionally, this has been done through "examinations," often in the form of oral questions and / or written tests related to newly presented material. At least in part, as a result of pressures to increase class size and reduce time, and therefore to reduce costs, associated with formal education, and to facilitate examination and scoring processes in increasingly automated and / or online environments, there has been an increasing reliance on so-called multiple-choice tests (i.e., selecting the correct or best answer from a list of options) to assess knowledge comprehension. The accuracy of such multiple-choice tests in assessing learning and / or true understanding is a matter of considerable discussion, especially since numerous strategies for taking such tests have been published, even if one lacks substantial knowledge of the subject. In addition, memorizing answers to such forms of examinations does not necessarily indicate a deep understanding of the subject matter.

[0012] The need for accurate assessment methods of knowledge acquisition is not only for the benefit of learners, but also for the many people who contribute to formal and informal education (e.g., parents, peers, professors). Reintroduction or revisiting material, and / or presentation of synergistic topics, can greatly support the entire learning process, especially when re-explained using different styles and / or strategies. Identifying and disseminating individual knowledge acquisition (while taking reasonable privacy concerns into account) can help define and / or regulate an acceptable pace (i.e., speed and / or complexity of new topics), scope, and order of instruction. In addition, facilitators involved in an individual's education may synchronize with the presentation of review topics, avoid redundancy, and track the level of academic and intellectual progress across different topic ranges. Such feedback can also help improve methods of knowledge presentation by informing those providing new knowledge about the effectiveness of their teaching strategies.

[0013] A highly praised attribute of distance learning and so-called virtual classrooms is the ability to conduct learning at one's own pace or self-management. The ability to choose when and where to interact with such resources is highly valuable to many. Furthermore, some virtual or electronic classrooms allow for the avoidance of presenting parts of a topic based on the results of online assessments (generally multiple-choice). However, while this process may save time and reduce the possibility of repetition and tedium, it does not guarantee a deep understanding of a topic or subject group, and skipping topics may even result in "guessing" (i.e., not truly understanding) during the assessment. Numerous reinforcement learning and time-saving steps per topic can be effectively implemented if true knowledge understanding can be assessed in real time as the knowledge is presented.

[0014] A new framework is needed to monitor and reinforce the emotional environment, particularly when its components (e.g., instructors, peers, classrooms, learning activities) are separated by distance and / or time, thereby improving the efficiency and effectiveness of learning. Artificial human companions will never completely replace family, friends, teachers, peers, or other individuals involved in the human support and learning process. However, machine-based approaches may have the potential to help present effective educational experiences and subsequently track knowledge comprehension while simultaneously supporting individual cognitive and emotional behavior. Furthermore, the ability to precisely monitor both successful and unsuccessful learning strategies, particularly as a function of topic, age, and / or academic growth, could provide a framework for reinforcement learning that supports individual learners and improves overall educational strategies for both child and adult education. [Overview of the Initiative]

[0015] In light of the foregoing, systems and methods are provided herein for using one or more Human Interaction Entities (HIEs) to monitor and adjust an individual's emotional state, and to encourage and / or enhance both structured learning (e.g., formal, institutional, qualification-based) and unstructured learning (e.g., play, chat, entertainment). The HIE may include one or more portable Human Interaction Accessories (HIAs) that facilitate mobile interaction. The HIE may optionally be instantiated with one or more artificial intelligence (AI) to determine the context and / or intent in real time during human-machine interaction. The AI ​​component may also represent one or more AI “personalities” (AIPs), “characters,” or “companions” that are friendly, socially acceptable, and enable a more effective communication experience. The HIE may manage, identify, and / or conduct the communication experience, maintaining emotional engagement, assessing the learner’s understanding of knowledge, and enhancing the methods for teaching.

[0016] Embodiments may provide one or more users with an intuitive machine interface and / or companion to 1) monitor and / or adjust an individual's emotional state to facilitate engagement, and simultaneously 2) present a knowledge base to structure, encourage, and measure the acquisition of new knowledge. Often, strategies described herein for monitoring and enhancing the emotional element overlap with similar approaches that simultaneously monitor and enhance the cognitive (i.e., learning) element. To avoid repetition, some strategies and diagrams may be described in more detail within either the emotional or cognitive descriptions, although similar strategies may be applied in both settings.

[0017] In exemplary embodiments, considering a first method of monitoring emotional conditions, individuals often create multiple indications of emotional states (usually without much thought). Furthermore, one or more observers of an individual may also readily create multiple indications of the individual's perceived emotional states (e.g., without much thought). These embodiments allow for a nearly automated approach to building a database of 1) dialogue indicators of an individual's emotional states (ranging from boredom to enthusiasm to overwhelming), and 2) dialogue indicators of an individual's perceived emotional states by observers. Dialogues that produce specific emotional states may be scanned for additional indications of emotional states by starting with a few intentional (i.e., easily detectable by a machine) indicators of emotional states (e.g., expressing one or more keywords, pressing a button by either the learner or observer, a predetermined cue gesture). These entire programming and data-building strategies, repeatedly applied to grow and enhance the diversity of indications in the emotional state database, are sometimes referred to as a “bootstrapping” approach.

[0018] The cycle of using known indicators of emotional state to discover further indicators of emotional state may be repeated indefinitely. In addition, as the database expands to include a larger number of dialogues and the analysis is iterated, a step may be taken to remove (i.e., "eliminate") dialogues that are unreliable and / or poorly correlated with a particular emotional state. Indicators of emotional state that are determined to occur infrequently, not to be specific to a particular emotional state, and / or are later found to be invalid (e.g., corrected by one or more observers or the monitored individual) may be removed from the indicators in the emotional state database.

[0019] Along similar lines, individual learners often create multiple indicators of knowledge acquisition or cognitive states, usually without much thought. Furthermore, one or more observers of the learner may also readily create multiple indicators of what they perceived as knowledge acquisition by the learner. These aspects allow for a nearly automated approach to building a database of 1) indicators of knowledge acquisition by the learner (ranging from complete understanding to confusion), 2) indicators of perceived knowledge acquisition created by the learner's observers, and 3) successful knowledge presentations (i.e., instruction) (as opposed to failures).

[0020] By starting with a few intentional (i.e., easily detectable by machine) indicators of knowledge acquisition, dialogues that generate knowledge comprehension may be scanned for additional instructions on understanding. The cycle of finding further indicators of understanding using known indicators of understanding (i.e., bootstrapping) may be repeated indefinitely. The indicators and / or combinations of indicators identified above for dialogues resulting from knowledge acquisition that is determined to occur infrequently, and / or knowledge that is later found not to be well understood (e.g., as a result of conventional tests and / or oral examinations), may be removed from the indicators of the knowledge acquisition database.

[0021] A similar bootstrapping process may be used to construct a database of one or more indicators of knowledge comprehension gaps or confusions. For the purposes of this application, “confusion,” or the degree of knowledge comprehension gaps, may be expressed in a wide variety of detectable forms, including, for example, conveying levels of simply indicating a lack of understanding, demanding clarification, questioning, frustration, and / or even anger. Alternatively or in addition, learner or one or more observer responses indicating learner confusion may be accumulated by considering knowledge comprehension as a continuum ranging from deep understanding at one end of the spectrum to complete lack of understanding, misunderstanding, and / or confusion at the other end of the same spectrum.

[0022] A crucial aspect of both teaching and learning is the ability to recognize when and to what extent knowledge acquisition is occurring. In most formal educational settings, such real-time assessment is often impossible due to the excessive time required to conduct frequent tests in multi-learner settings, the availability of teaching and / or assessment resources, and excessive costs. Formal testing (e.g., exams) and classroom feedback through instructional questioning are typically limited to inquiries about chunked (i.e., multi-topic) knowledge, often well after the initial knowledge presentation and generally after the presentation of review material (i.e., material that may rely on previous understanding) and / or unrelated topics. Machine-based approaches that assess the occurrence and timing of knowledge comprehension in real time can greatly support the processes involved in teaching and knowledge acquisition within both individual and group learning settings.

[0023] In further exemplary embodiments using neural network-based and / or non-neural network classification strategies, a machine-based approach is developed to form a database of one or more labeled learning dialogue components (e.g., video, audio) in which it is known that learner knowledge acquisition is occurring. Such databases may be queried using neural networks and other classifier approaches (e.g., support vector machines [SVM], hyperplane analysis) to identify specific dialogue characteristics or components and / or develop learner-indicator dialogue "templates" that signal success in understanding against gaps or confusion in understanding.

[0024] Different learners, particularly those of different ages and / or different topics, may express their instructions for knowledge acquisition and comprehension in different ways. Therefore, databases used to develop and / or train machine-based approaches to identify comprehension levels may improve classification accuracy by incorporating numerous knowledge acquisition interactions from a wide range of learners across numerous learning environments. Where known, factors such as age, learning environment, native language, cultural context, and subject area of ​​instruction may be used as input to large, comprehensively applicable classification schemes.

[0025] Alternatively, or in addition, the database may be segmented into separate classifiers if such factors are expected to be known at implementation time (i.e., when the classifier scheme is deployed). Separate classification schemes applied to specific subsets of the population and / or learning topic may be simpler and less computationally intensive to implement. On the other hand, more comprehensive classification schemes used when such segmentation factors are not expected to be known at implementation time may require more computational resources and may be less accurate in some cases.

[0026] In further exemplary embodiments, a database of one or more dialogue components that generate knowledge acquisition may be valuable at the learner level, but an additional database of knowledge presentations that have succeeded (and / or failed) in generating new knowledge acquisition may be valuable to teachers and others involved in organizing knowledge in a comprehensible form (e.g., authors, textbook editors, journalists, distributors, media professionals). Thus, a machine-based approach that identifies the earliest time of knowledge acquisition, and consequently one or more knowledge presentations that lead to knowledge comprehension, may be used to expand a database of successful teachings by identifying the presentation material and dialogue components that produced comprehension. A machine-based approach that identifies successful teaching strategies may be used to enhance both individualized and group learning environments. In general, a database of proven successful teachings by instructors and identifiers of knowledge acquisition by learners may provide tools for “learning about learning.”

[0027] In additional exemplary embodiments, the detection of successful knowledge acquisition by a learner may be used to notify others involved in the education and development of the individual learner (i.e., other instructors, parents, facilitators, and / or monitors) via one or more monitoring processors (e.g., mobile devices, laptops, information terminals) and, if necessary, telecommunications. This facilitates a number of enhanced teaching strategies, including avoiding repetition of a particular topic by multiple instructors and / or synergistically presenting additional material related to a topic just learned. Machine-based systems may also be deployed to automatically (or selectively, if desired, based, for example, on the subject matter of the topic and / or privacy concerns) notify all monitors (e.g., those involved in regular progress), facilitators (e.g., parents or guardians), or any subset of individuals involved in the education and development of the learner, when knowledge acquisition occurred and knowledge presentations that would lead to learning success.

[0028] According to an exemplary embodiment, there is provided an electronic device including an interaction processor instantiated with artificial intelligence, one or more interaction output devices operably coupled to the interaction processor, and one or more sensors operably coupled to the interaction processor, proximate to a first human, expressing knowledge that the first human would not know by one or both of the electronic device and a second human to the first human, obtaining dialogue data from a dialogue between the electronic device and the first human from the one or more sensors, classifying, within the dialogue data, one or more indications of the degree of emotional engagement by the first human by one or more of the interaction processor and one or more remote processors, and classifying, within the dialogue data, one or more indications of the degree of knowledge acquisition by the first human by one or more of the interaction processor and one or more remote processors, a method of monitoring human learning using artificial intelligence is provided.

[0029] According to another exemplary embodiment, there is provided a system for monitoring the emotional state of a human during learning, including an electronic device including an interaction processor, one or more interaction output devices operably coupled to the interaction processor, and one or more sensors operably coupled to the interaction processor, configured to be disposed proximate to a first human, wherein one or both of the electronic device and a second human express knowledge that the first human would not know to the first human using the one or more interaction output devices, the one or more sensors obtain dialogue data, and one or more of the interaction processor and one or more remote processors classify one or more indications of the degree of emotional engagement by the first human within the dialogue data, and one or more of the interaction processor and one or more remote processors classify one or more indications of the degree of knowledge acquisition by the first human within the dialogue data.

[0030] According to yet another exemplary embodiment, there is provided an electronic device including a dialogue processor, one or more dialogue output devices operatively coupled to the dialogue processor, and one or more sensors operatively coupled to the dialogue processor, proximate to a first human, expressing knowledge that the first human would not know by one or both of the one or more dialogue output devices and a second human to the first human, obtaining dialogue data from the one or more sensors, classifying, within the dialogue data, one or more indications of the degree of emotional involvement of the first human by one or more of the dialogue processor and one or more remote processors, and classifying, within the dialogue data, one or more indications of the degree of knowledge acquisition of the first human by one or more of the dialogue processor and one or more remote processors, a method for monitoring the emotional state of a learning human is provided.

[0031] According to an additional exemplary embodiment, there is provided a first electronic device including a first processor and one or more first output devices operatively coupled to the first processor, proximate to a first human, providing a second electronic device including a second processor and one or more sensors operatively coupled to the second processor, proximate to a second human, expressing knowledge that the first human would not know by one or both of the first electronic device and the second human to the first human, obtaining dialogue data from the one or more sensors from a dialogue between the second electronic device and the second human, classifying, within the dialogue data, one or more indications of the degree of emotional involvement of the first human by one or more of the second processor and one or more remote processors, and classifying, within the dialogue data, one or more indications of the degree of knowledge acquisition of the first human by one or more of the second processor and one or more remote processors, a method for monitoring an observer of human learning is provided.

[0032] According to another additional exemplary embodiment, a system is provided for monitoring a human learning observer, comprising a first electronic device including a first processor and one or more first output devices operably coupled to the first processor, configured to be positioned in close proximity to a first human, and a second electronic device including a second processor and one or more sensors operably coupled to the second processor, wherein the system is configured such that one or both of the second human and the first output devices represent knowledge that the first human would not know to the first human, one or more sensors acquire dialogue data, the second processor and one or more of the one or more remote processors classify one or more indications of the degree of emotional engagement by the first human in the dialogue data, and the second processor and one or more of the one or more remote processors classify one or more indications of the degree of knowledge acquisition by the first human in the dialogue data.

[0033] A further additional exemplary embodiment provides a method for monitoring the emotional state of a learning human, comprising: providing a first electronic device in close proximity to a first human, including a first processor and one or more first output devices operably coupled to the first processor; providing a second electronic device in close proximity to a second human, including a second processor and one or more sensors operably coupled to the second processor; representing to the first human knowledge that the first human would not know, by either the first electronic device or the second human, or both; obtaining dialogue data from one or more sensors from a dialogue between the second human and the second electronic device; and classifying within the dialogue data one or more indications of the degree of emotional engagement by the first human, by the second processor and one or more of the one or more remote processors.

[0034] According to another exemplary embodiment, a method is provided for storing labeled human emotional dialogues, comprising: providing an electronic device in close proximity to a first human, including a dialogue processor, one or more dialogue output devices operably coupled to the dialogue processor, and one or more sensors operably coupled to the dialogue processor; the first processor obtaining a selected degree of emotional involvement from the electronic device, a predetermined list of degrees of emotional involvement, a first human, and a second human; instructing the first human to consider activities associated with the selected degree of emotional involvement by either or both of the electronic device and the second human; obtaining dialogue data from one or more sensors from the dialogue between the electronic device and the first human; and labeling one or more dialogues in the dialogue data with the selected degree of emotional involvement by the dialogue processor and one or more of the one or more remote processors.

[0035] According to another exemplary embodiment, the present invention provides a first electronic device comprising an electronic device in close proximity to a first human, including a first processor, one or more first output devices operably coupled to the first processor, and one or more first sensors operably coupled to the first processor; and a second electronic device comprising an electronic device in close proximity to a second human, including a second processor, one or more second output devices operably coupled to the second processor, and one or more second sensors operably coupled to the second processor; and selected by one or more of the first processor, the second processor, and one or more remote processors. A method is provided for accumulating labeled observers of human emotional dialogues, comprising: obtaining the degree of emotional engagement from one of a predetermined list of degrees of emotional engagement, a first person, and a second person; instructing the first person to consider activities associated with the selected degree of emotional engagement by one or both of the first electronic device and the second person; obtaining dialogue data from one or more first sensors from the dialogue between the electronic device and the first person; and labeling one or more labeled dialogues in the dialogue data with the selected degree of emotional engagement by one or more of the first processor and one or more remote processors.

[0036] In yet another exemplary embodiment, a system is provided for storing labeled human emotional dialogues, including an electronic device comprising a dialogue processor, one or more dialogue output devices operably coupled to the dialogue processor, and one or more sensors operably coupled to the dialogue processor, configured to be positioned in close proximity to a first human, wherein the system is configured such that the electronic device acquires a selected degree of emotional involvement from one of the dialogue processor, a predetermined list of degrees of emotional involvement, a first human, and a second human; the electronic device and or both of the second human instruct the first human to perform an activity associated with the selected degree of emotional involvement; the dialogue processor acquires dialogue data from one or more sensors from the dialogue between the electronic device and the first human; and the dialogue processor and one or more of the one or more remote processors are configured to label one or more dialogues in the dialogue data with the selected degree of emotional involvement.

[0037] Other aspects and features, including the necessity and use of the present invention, will become apparent from the following description in conjunction with the accompanying drawings.

[0038] A more complete understanding of the present invention can be derived by considering and referring to the “modes for carrying out the invention” in relation to the following exemplary drawings. In the drawings, similar reference numerals refer to similar elements or actions throughout the drawings. The exemplary embodiments presented are illustrated in the accompanying drawings. [Brief explanation of the drawing]

[0039] [Figure 1] Figure 1 illustrates the "inverted U-shaped" relationship between the degree of an individual's emotional engagement and the predicted likelihood of knowledge acquisition (including speed and complexity). [Figure 2] Figure 2 illustrates a feedback loop that includes both human-based and machine-based elements, in which an individual's emotional and cognitive states are simultaneously monitored and influenced. [Figure 3] Figure 3 shows an overview of the functional elements within an emotional and cognitive feedback loop that build up a labeled database of dialogue elements which may be used to classify an individual's emotional and cognitive states and identify topics that generate known emotional engagement and successful (and unsuccessful) teaching strategies. [Figure 4] Figure 4 shows an overview of the functional elements in deploying a feedback loop that simultaneously monitors and reinforces an individual's emotional and cognitive environment, and notifies the monitor or supporter about the individual's emotional and intellectual development. [Figure 5] Figure 5 summarizes the ability to monitor and enhance an individual's emotional and cognitive support environment using machine-based dialogue elements (represented by cartoon-like characters) within a simultaneous feedback loop. [Figure 6] Figure 6A shows a scene from a scenario in which a grandfather uses long-distance communication to instruct a young child on different shapes (i.e., geometric shapes) using virtual flashcards. Figure 6B shows a continuation of the scene shown in Figure 6A, showing the child's simultaneous instruction of substantial knowledge understanding, including the child's facial expressions and raised hands, smiles, as well as words of encouragement from a nearby parent (i.e., the “observer”) and facial expressions from the grandparents at a distance. [Figure 7] Figure 7 shows sensor and actuator components that may be included in one or more human interaction entities (HIEs) and / or associated human interaction accessories (HIAs), which may also monitor and influence emotional and cognitive states, deliver new knowledge, and help detect knowledge acquisition, along with optional access to remote computing resources. [Figure 8] Figure 8 is a flowchart summarizing the steps of the so-called "bootstrap" strategy, which involves growing a database of dialogues based on instructions or responses from learners and / or instructions or responses expressed by the learners' observers. [Figure 9]Figure 9 illustrates a method for generating “self-labeled” dialogue elements in which selected (i.e., self-identified) bodily movements and / or mental activities are performed to create a selected emotional state. [Figure 10] Figure 10 illustrates a so-called "bootstrapping" approach in which newly identified human conversations that are temporally close to known indicators of emotional state are stored in a database that is later used to determine if the conversations are frequent and / or to distinguish the indicators of emotional state. [Figure 11] Figure 11 is a flowchart illustrating the feedback control steps that adjust and / or maintain an individual's emotional state within the optimal “engagement” emotional level for learning (and well-being). [Figure 12] Figure 12 shows a timeline in which multiple knowledge presentations and / or activities are tracked simultaneously to influence the timing of re-presenting the material at times that help maximize knowledge retention in long-term memory. [Figure 13] Figure 13 is a flowchart illustrating the steps of a so-called "bootstrap" strategy, which involves growing a database of learner dialogues indicating the degree of knowledge acquisition (e.g., substantive understanding, confusion) and an additional database of instructional dialogues (i.e., "instructions") that were successful (or unsuccessful) in creating understanding. [Figure 14] Figure 14 shows a timeline of instructional dialogues that produced knowledge comprehension, as determined by at least one identified metric of learner knowledge acquisition, in order to identify additional instructions for knowledge acquisition. [Figure 15] Figure 15 shows a timeline of instructional dialogues that produce substantial knowledge acquisition, determined by multiple identified instructions for knowledge acquisition, in order to find additional instructions for knowledge acquisition that grow or "bootstrap" the labeled dialogue database. [Figure 16]Figure 16 shows the results of a reanalysis of the knowledge acquisition metrics database, in which adding a new metric for knowledge acquisition to the database reveals at least one new early indicator of knowledge acquisition. [Figure 17] Figure 17 shows the results of a reanalysis of the knowledge acquisition elements database, in which low-frequency dialogue elements are removed, early instruction in knowledge acquisition is identified, and in addition, the determination of which knowledge presentations (i.e., instruction) led to learner understanding is changed by determining at least one new metric of knowledge acquisition. [Figure 18] Figure 18 shows the results of a reanalysis of the teaching database, where the identification of new early indicators of knowledge acquisition, along with new early teachings, leads to the removal of teachings previously judged as successful and the addition of new teachings to the database of successful teachings. [Modes for carrying out the invention]

[0040] Before describing exemplary embodiments, it should be understood that the present invention may vary and is not limited to the specific embodiments described. Furthermore, since the scope of the present invention is limited only by the appended claims, it should be understood that the terminology used herein is intended to describe only specific embodiments and is not intended to limit them.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art of the field to which this invention pertains. It should be noted that, as used herein and in the appended claims, the singular forms "a," "an," and "the" include the plural form unless otherwise explicitly stated in the context. Thus, for example, a reference to "compound" includes multiple such compounds, a reference to "polymer" includes one or more polymers and their equivalents known to those skilled in the art, and so on.

[0042] In exemplary embodiments, one or more electronic devices (i.e., Human Interaction Entities (HIEs)) may 1) monitor and adjust an individual's emotional environment to maintain the individual in a state of engagement or enthusiasm, and 2) simultaneously monitor and enhance the individual's cognitive learning environment to optimize the rate and / or complexity of new knowledge acquisition. The individual's emotional environment may be influenced by one or more people, and / or adjusted directly by the HIE as a result of machine-based (i.e., HIE) prompts from one or more people. During engaging monitoring and / or adjustment, the learning environment may be structured (e.g., formal, institution-based, qualification-based) or unstructured (e.g., play, chat, entertainment).

[0043] The close link between emotion and learning calls for a synergistic approach that simultaneously addresses both an individual's emotional and cognitive states (in addition to their physical state) to promote optimal intellectual development. Enhanced emotional and intellectual developmental environments may be achieved, supported by machine-based tools that may (in real time) sense both the learner's emotional and cognitive states to extend emotional and cognitive conditions.

[0044] Combined individual emotional and cognitive states Generally, it is accepted that a learner's emotional state and their ability to acquire new knowledge are coupled. Figure 1 includes a form of the inverted U-shaped correlation 10, originally developed by Yerkes and Dodson, adapted to help elucidate the systems and methods described herein. When the degree of knowledge acquisition 12 is plotted as a function of the degree of emotional engagement 11, the dependency is approximately an inverted U-shaped relationship 10.

[0045] Within the realm of emotional engagement, three realms may be defined: 1) realm 13a, in which the individual is partially engaged, disengaged, or even bored, and overall mental performance, including the degree of knowledge acquisition, is reduced; 2) realm 13b, in which optimal engagement (represented by a filled triangle pattern between dashed lines) is present in relation to overall cognitive performance, including an elevated level of learning; and 3) realm 13c, in which there may be excessive excitement and / or a prolonged period of elevated emotional stimulation. In the latter case, performance and learning may be dramatically reduced as a result of a feeling of being overwhelmed (e.g., due to excessive and / or prolonged stress). An optimal learning and performance environment can be achieved by regulating the emotional environment within the “engaged” realm 13b in Figure 1 (i.e., avoiding both a lack of stimulation or an over-stimulation in the emotional environment).

[0046] The term “degree of emotional engagement”¹¹ is used in Figure 1, and this term means at least a near-equivalent to a number of alternative terms used in various fields to describe the spectrum of emotional states that have an impact on learning and on an individual’s overall performance and emotional well-being. Early research in this field used terms such as “pressure,” “motivation,” “alertness,” “arousal,” and “level of stress.” Other terms used to represent similar concepts include “emotional state,” “tension,” and “degree of nervousness.”

[0047] The three regions on the horizontal axis in graphs 13a, 13b, and 13c in Figure 1 may be described using a wide variety of nearly similar technical terms. For example, region 13a labeled "boredom" may also be described in terms of "not engaged," "partially engaged," "not excited," "day-dreaming," "distracted," and "disinterested." In some cases, such emotional states may vary depending on physical conditions, including, for example, being tired, being in a distracting physical environment, feeling unwell, and / or depressed. Region 13b labeled "engaged" may be described as "fully engaged," "excited," "motivated," "attentive," "occupied," and "inspired." Area 13c, labeled "overwhelming," may be described in terms of different levels of "anxiety," "strain," "elevated tension," "tension," and "stress."

[0048] The terms “degree of emotional engagement” and “degree of knowledge acquisition” are used herein, but these terms are similar to many other terms used to describe similar relationships, such as those mentioned above. Labels 12 on the vertical axis in Figure 1 may refer to a number of different metrics, including “depth of knowledge understanding,” “performance,” “number of memories formed,” “rate of memory formation” (i.e., a metric over time), “rate of knowledge acquisition,” “complexity of newly acquired knowledge,” and “(perceived) difficulty of new material.”

[0049] In this specification, there is substantial overlap between the processes of acquiring new knowledge and understanding it. However, the term “acquire” or “acquire” is used preferentially during the process of initially obtaining new knowledge, while the term “understand” or “comprehend” is used preferentially during the process of that knowledge being acquired and eventually becoming part of long-term memory. Numerous similar terms (e.g., “comprehension,” “cognition,” “grasp,” and “cognizance”) may also be used to describe such “acquisition” and “understanding” processes.

[0050] As one consequence of the combined nature of emotional and cognitive states, the rate of cognitive acquisition of knowledge can also be used as an indicator to help measure (and find other indicators of) the degree of emotional engagement. One objective of the exemplary systems and methods described herein may be to adjust the learner's emotional environment with the goal of maintaining an emotional state within a range that maximizes the depth and / or speed of acquiring new knowledge. This emotional range also tends to maximize emotional achievement, happiness, and / or feelings of well-being in most individuals.

[0051] As further elaborated in the section on “Background Technology,” these generalizations can be extended to include many different styles of knowledge, such as learning to read and write about abstract concepts, recognize smells, or learn to dance. The acquisition of new knowledge and / or abilities to perform activities may also include such styles. To emphasize the broad range of styles or types of knowledge, “understanding” may include not only the ability to “think about” concepts, but also the use of any of the human senses related to perception, training muscles to perform actions, and so on.

[0052] Combined emotional and cognitive feedback loops Figure 2 shows the elements of two feedback loops used as a framework for simultaneously monitoring and supporting the emotional and cognitive components of an individual's environment. The upper feedback loops 20, 21a, 22a, 23a consist of a human component 22a and machine-based components 21a, 23a that perceive (21a) and control (23a) the emotional support environment of individual 20. Similarly, the lower feedback loops 20, 21b, 22b, 23b consist of machine-based components 21b, 23b and a human component 22b that perceive (21b) and control (23b) the cognitive support environment of learner 20. An individual's emotional state may be assessed through direct interaction with individual 20 and / or through interaction with one or more observers (not shown) of the individual. One or more observers may or may not be involved in providing emotional support 22a (i.e., strictly providing the role of observers).

[0053] Information regarding an individual's emotional state may be communicated to one or more human supporters 22a involved in the emotional well-being of the individual 20. These emotional supporters 22a may be, for example, parents, guardians, siblings, grandparents, or social workers. The emotional state may include, for example, indications that the individual may be bored, that the individual may be feeling frustrated with a particular dialogue topic, or that a certain amount of time has elapsed since the last break from a period of focused activity. One or more human supporters 22a may take action to support the individual 20's emotional environment by directly interacting with the learner 20 (indicated by a dashed line 24a between the human supporter 22a and the individual 20), directing a machine-based dialogue 23a, or a combination of human 22a and machine-based dialogue 23a.

[0054] In parallel, in the cognitive support feedback loops 20, 21b, 22b, and 23b in the lower part of Figure 2, machine-based sensing 21b of the individual's (i.e., learner's) cognitive state, including the degree and / or speed of knowledge acquisition, may be performed in real time on a topic-by-topic basis. The learner's cognitive state may be assessed through direct interaction with the learner and / or through interaction with one or more observers (not shown) of the learner. One or more observers may or may not be involved in providing cognitive support 22b (i.e., strictly providing the role of observers).

[0055] Information regarding the learner's cognitive understanding may be communicated to one or more human instructors 22b involved in the teaching. Instructors 22b may include, for example, parents, guardians, grandparents, siblings, teachers, professors, colleagues, online assistants, tutors, etc.

[0056] Metrics of cognitive understanding may include, for example, metrics of confusion when giving new reasoning based on the presentation of knowledge or new ideas. Machine-reinforced cognitive support may also include reintroducing previously covered topics and / or synergistic material at optimal times to reinforce the formation of long-term memory. Supporting the cognitive environment of learner 20 may necessarily involve one or more instructors 22b directly interacting with learner 20 (indicated by a dashed line 24b between instructor 22b and learner 20), or generating machine-initiated dialogues 23b, machine-based “nudges” or suggestions directed to any number of people (including instructor 22b) who may be interacting with learner 20, or a combination of human-based 22b and machine-based 23b dialogues to put actions or activities into practice.

[0057] Following a structure similar to that of Figure 2, Figure 3 illustrates a key step in the accumulation of a database which may be used to train a machine-based approach to classify an individual's emotional level (upper feedback loops 30, 31a, 32a, 33a) and cognitive level (lower feedback loops 30, 31b, 32b, 33b). By monitoring one or both of the individual / learner and one or more observers 30 of the individual, dialogue elements including one or more indicators of the degree of emotional engagement (e.g., ranging from boredom to enthusiasm to overwhelming) may be identified (31a). Such dialogues may be accumulated in one or more databases 32a, which may then be analyzed to identify additional dialogue elements that are temporally consistent with the identified indicators (e.g., based on the frequency of occurrence of the dialogue elements). Alternatively or in addition, dialogue components (e.g., video, audio) may be examined by one or more human analysts and subsequently identified and / or labeled as emotional content. As will be described in more detail below, such labeling may include identifying specific emotional states (e.g., irritated, agitated, bored, distracted, absent-minded, fully engaged), and / or scoring using one or more spectrums or scales (e.g., from bored to fully engaged, overwhelmed, or stressed).

[0058] Within the emotional support feedback loops 30, 31a, 32a, 33a, the machine used to implement emotional support dialogues and strategies (e.g., HIE and / or HIA) 33a may include, and / or notify, the same additional processors involved in cognitive response classification 31a, enabling the labeling of dialogue topics and strategies according to specific states (e.g., boredom, distraction) and / or degrees (e.g., identified on a scale) of the resulting emotional responses. For example, topics (e.g., art, science, sports) and / or delivery methods (e.g., podcasts, audio / visual, text, pictures) may each be labeled and stored (33a) according to the identified emotional responses 33a. Subsequent analysis may then identify topics and / or delivery strategies that evoke positive emotional responses (e.g., based on their frequency of occurrence). Conversely, topics and / or delivery strategies that evoke less desirable emotional responses may also be tracked and avoided in subsequent dialogues. The resulting classification process may be used at the individual level, the group level of individuals (e.g., those covering specific topics of interest, subsets of ethnic behaviors), and / or applied to a large population of users.

[0059] In parallel, the cognitive support feedback loops 30, 31b, 32b, and 33b in the lower part of Figure 3 show machine-based sensing and classification 31b of the individual's cognitive state, including the degree of knowledge acquisition, performed in real time on a topic-by-topic basis. The learner's cognitive state may be assessed through direct interaction with the individual and / or through interaction with one or more observers of the learner. One or more observers may or may not be involved in providing cognitive and / or emotional support (i.e., strictly providing the role of observers).

[0060] Such dialogues may be stored in one or more databases 32b, which may then be analyzed to identify additional dialogue elements that are temporally consistent with the identified indicators (for example, based on factors including frequency of occurrence). Alternatively, or in addition, the dialogues (e.g., visual and auditory elements) may be examined by one or more human analysts and subsequently identified and / or labeled to indicators of knowledge acquisition. Such labeling may include identifying specific levels of knowledge acquisition (e.g., full understanding, partial understanding, misunderstanding) and / or being scored using a continuous scale (e.g., from full understanding to confusion).

[0061] Within the cognitive support feedback loops 30, 31b, 32b, 33b, the machine used to classify the indicators may be informed of the instruction and strategy (e.g., those produced by HIE and / or HIA) 33b used to produce the identified cognitive response 31a. This makes it possible to label the instruction and presentation strategies according to the discrete state of understanding derived (e.g., understanding, generation of confusion) and / or degree (e.g., identified on a scale). The instruction database may include, for example, topics (e.g., arts, science, sports) and / or delivery methods (e.g., podcasts, audio / visual, text, pictures) accumulated according to the labeled cognitive response 33b. Subsequent analysis may identify topics and / or delivery strategies that evoke understanding (e.g., based on their frequency of occurrence). Conversely, topics and / or delivery strategies that evoke a lack of understanding and / or misunderstanding may also be tracked and avoided in subsequent instruction.

[0062] In additional embodiments, and in concepts further developed within the parallel emotional and cognitive feedback loops shown in Figures 2 and 3, Figure 4 illustrates key steps to enable monitoring and control of the learner's emotional and cognitive state functioning, and to track and notify any appropriate (e.g., legally authorized) party or “monitor” of the learner’s current emotional state, emotional history, knowledge progress, and / or overall intellectual development. The upper feedback loops 40, 41a, 42a, 43a, 44a, and 45a outline key implementation steps to monitor, track, and enhance the emotional environment of the individual / learner 40. Within the inner feedback loops 40, 41a, 42a, and 43a, processing includes classifying the degree of emotional engagement 41a by the individual 40 in real time (e.g., utilizing accumulated training datasets as shown in Figure 3). These classifications may be based on direct interaction with the individual 40 and / or interaction with one or more observers (not shown) of the individual.

[0063] The classification may be communicated to human and / or machine-based entities, for example, to deploy personalized real-time strategies to modulate the emotional environment of learner 42a. In this case, the term “modulate” is used to describe maintaining the emotional environment within the peak range of cognitive stimuli (see Figure 1). Insufficient or excessive emotional stimuli reduce the ability to learn, as well as the ability to form long-term memories and / or maintain optimal cognitive function. Similar technical terms that may be used to describe such modulating processes include “regulate,” “enhance,” “influence,” and “control.”

[0064] Activities designed to regulate emotions (machine-based and / or deployed by one or more humans) may be passed to one or more human and / or machine-based dialogue entities 43a (i.e., HIE and / or HIA) that support the learner's emotional environment. The one or more human supporters 43a may be the same or different from those involved in deploying a particular emotional support strategy 42a or cognitive support strategy 42b. Strategies may include, for example, signaling (i.e., suggesting) activities to the human supporter to increase or decrease the level of emotional dialogue (e.g., speed, intensity), and / or animating them with cartoon-like characters. Strategies may be at least in part based on responses created by previous emotional classifications made during HIE-based dialogues (i.e., those incorporating “memory”).

[0065] The outer feedback loops 40, 41a, 44a, 45a, and 43a within the upper part of Figure 4 outline a key step in maintaining a database to track emotional states 44a, including topics that generate increased interest or indifference by an individual 40. The degree of emotional engagement 41a and the classification of the behavior 43a that led to such emotional states may be automatically labeled and archived (44a).

[0066] In addition, cognitive topics 43b associated with the classified emotions 41a may be categorized into those that generate, for example, the learner's interest, boredom, confusion, frustration, feelings of overwhelm, stress, etc. (44a). A machine-based maintenance of such topics and their emotional responses may be used to notify the individual's caregiver (45a), who may then instruct the caregiver to intervene in (or be a direct part of) a dialogue with the individual (43a). Notification of emotional responses and associated topics may be done topic by topic, periodically (e.g., according to a schedule), or as needed (e.g., immediately when an emotional depression is detected).

[0067] In parallel, the lower feedback loops 40, 41b, 42b, 43b, 44b, and 45b outline key realization steps for monitoring, tracking, and reinforcing the cognitive environment of the individual / learner 40. Within the inner feedback loops 40, 41b, 42b, and 43b, processing includes classifying the degree of cognitive acquisition 41b by the learner 40 in real time (e.g., by utilizing metrics within the accumulated training dataset, as outlined in the accompanying explanation in Figure 3). These classifications may be performed based on interactions with the learner 40 and / or interactions with one or more observers of the learner. The classifications may be passed to human and / or machine-based entities to deploy personalized real-time teaching strategies 42b.

[0068] Cognitive support strategies (machine-based and / or deployed by one or more humans) may be passed to one or more human supporters and / or machine-based (e.g., HIE and / or HIA) dialogue entities 43b to enhance the learner's cognitive environment. The one or more dialogue human supporters 43b may be the same or different from those involved in deploying instruction 42b or providing the emotional support described above.

[0069] Notification of emotional and cognitive progress to the monitor. The outer feedback loops 40, 41b, 44b, 45b, and 43b within the lower part of Figure 4 outline the steps for building and maintaining a database to track cognitive progress 44b, including topics that are fully understood, partially understood, or not understood. The classification of the degree of cognitive acquisition 41b, and the instruction 43b that led to such understanding, may be automatically labeled and archived (44b). Real-time “report cards” may be maintained, which may essentially cover all learning (i.e., by topic). Machine-generated report cards may be used to notify the learner’s supporters (45b). Such supporters and / or “monitors” may include those directly involved in the individual’s formal and extracurricular education (e.g., in-person or online teachers, coaches, professors, tutors), as well as those who may be involved in or support the individual’s overall well-being (e.g., parents, other family members, guardians, counselors). Notifications of cognitive advancements and / or topics that may require instructional reinforcement may be given on a topic-by-topic basis, periodically (e.g., according to a schedule), or as needed (e.g., when learning opportunities arise, such as field trips or demonstrations of device repair).

[0070] In addition to machine-based processes that help keep monitors up-to-date, machine-based tracking of progress (or areas of interest) may reduce the substantial time and effort required of some of the individuals being monitored, and / or reduce the emotional burden associated with formal reporting (e.g., which may be associated with traditional report cards). The distribution of such information may be restricted by legal regulations associated with individual privacy and / or by personal preferences. Furthermore, tracking and disseminating such information to monitors and educators may facilitate a number of enhanced teaching strategies, including avoiding repetition of specific topics by one or more instructors, synergistically presenting additional material related to topics recently learned, intervening when the rate of acquiring new knowledge appears to have slowed or is slower than expected, and / or providing celebrations and other positive emotional support as learning and / or emotional growth progresses.

[0071] One example of the need to monitor emotional and cognitive growth separately is that adolescence is a particularly challenging period for balancing emotional and cognitive support. This is because, at least in part, the emotional parts of the brain (various limbic system structures, including the amygdala) develop over approximately 10–12 years, while the reasoning parts of the brain (including the frontal lobe mentioned above) take up to 22–24 years to fully mature. Therefore, the relative ratio in which emotional and cognitive components can play a role in strategies that optimally support an individual's intellectual growth can vary considerably with age.

[0072] Machine-based generation, maintenance, and distribution of emotional and cognitive progress can have numerous additional benefits and applications. Despite maximum efforts to maintain frequent communication with teachers, coaches, counselors, educational administrators, and others, it is often difficult for parents or guardians of students to keep track of their progress across all the different subjects commonly covered in both formal and extracurricular education. Even learners themselves may not have a complete grasp of their progress in knowledge understanding across different topics, i.e., their range of interests.

[0073] This situation will worsen as the world becomes increasingly reliant on remote and / or hybrid learning, utilizing long-distance communication (i.e., interpersonal communication where the physical distance between participants exceeds the physiological limits of human perception without assistance). Long-distance learning has numerous potentially significant benefits, including, to name a few, the removal of geographical barriers, access to presentations by global leaders across a wide range of academic disciplines, the provision of opportunities that benefit those who are physically isolated, and the ability to learn at one's own pace. However, while online questions and formal assessments can provide some feedback, long-distance instruction often lacks the close monitoring of the knowledge acquisition and emotional environment, as well as other learning-contributing elements such as attention, motivation, and self-regulation.

[0074] In further embodiments, machine-based methods for determining and tracking knowledge acquisition may be set up to automatically (or selectively, for example, based on topic scope) notify all or any subset of monitors. Dissemination may also be set up to occur automatically (i.e., without significant learner involvement) based on learning progress, or periodically under the control of one or more instructors, supporters, or the learners themselves.

[0075] The distributed information may include any or all (as known to the HIE) of the methods and materials used for knowledge presentation, the time it took for the knowledge to be understood (or not to be acquired), a confidence score (if any) that the knowledge was understood, and / or links to preparatory and / or review materials related to the new knowledge. The accumulated database tracking individual progress may also include feedback from one or more monitors and / or any other individuals involved in the learner's development (e.g., coaches, counselors, employers). Such a centralized, machine-driven repository of cognitive progress provides resources for all involved parties that should be kept up-to-date.

[0076] Alternatively, or in addition, individual monitors may be notified of the categorized degree of emotional engagement and / or other emotional states. Such updates can be particularly valuable for monitors (e.g., parents, friends) to determine and track topics of interest (or, conversely, indifference). At a practical level, such information may help identify things (e.g., gifts), connections (e.g., particularly important friends), and / or other resources (e.g., tutors) that may improve the individual's emotional and / or cognitive environment. Recognizing such achievements can be an incentive for celebration, once a deeper understanding of a substantial topic has been achieved. If signs of boredom, feelings of overwhelm, and / or stress have been identified, such real-time tracking may help facilitate intervention when it may be appropriate.

[0077] Real-time monitoring of emotional and cognitive environments Using the same overall layout as Figures 2, 3, and 4, Figure 5 summarizes the machine learning and human-based components of a system and method for simultaneously monitoring and adjusting an individual's emotional and cognitive environments. In Figure 5, cartoon-like characters 51a, 51b, 53a, and 53b represent a machine-based process for interacting with individual 50. In practice, the machine-based interaction, implemented by the HIE and one or more HIAs, may include personal interactions with such cartoon-like characters 51a, 51b, 53a, and / or avatars representing a “personality” familiar to individual 50. Within the emotional support feedback loop 50, 51a, 52a, and 53a, machine-based sensing 51a of individual 50’s emotional state, including the degree of emotional engagement, may be performed in real time. Individual 50’s emotional state may be assessed through direct interaction with individual 50 and / or through interaction with one or more observers (not shown) of individual 50. One or more observers may or may not be involved in providing emotional support 52a (i.e., strictly provide the role of observer).

[0078] In the exemplary arrangement shown in Figure 5, two distinct individuals 52a and 52b are shown interacting with a child 50, with 52a primarily supporting the child's emotional environment and 52b primarily providing cognitive support. In this scenario, a parent may provide the majority of the emotional environment to the learner, while a professional instructor or tutor may primarily provide the cognitive environment. More generally, one or more individuals involved in the learner's support environment may interact individually with specific topics and / or components of the support system (e.g., instructional, emotional, performance monitoring), or with both the emotional and cognitive components, or with different modes of support at different times, or one individual may provide all modes of emotional and cognitive support.

[0079] Figures 6A and 6B show scenes from scenarios illustrating various forms of human-assisted emotional support and knowledge acquisition, which may be identified and subsequently utilized by machine-based techniques that facilitate emotional engagement and encourage and reinforce learning. In this scenario, the upper portion of Figure 6A illustrates a form of remote education for a little girl 60a using new knowledge displayed on a display device 62a acting as a HIE (for the little girl 60a, and in this case, also providing additional services to the girl's parents 67, as shown in Figure 6B). The knowledge involves identifying and naming geometric shapes (i.e., circles, triangles, pentagons, hearts, and plus signs, shown from left to right) that are visible within a series of virtual flashcards 63 on screen 62a. The new knowledge is compiled by remote grandparents 61a interacting with the little girl 60a via a network link and audio-visual display devices (e.g., HIE, tablet, laptop computer).

[0080] Within the scenario, new knowledge may arise from grandparents 61a, one or more additional people (not illustrated), a pre-arranged lesson plan (e.g., developed by a professional educator or motivator), information streamed using conventional coding techniques (e.g., audio recording, audio-visual display), an AI instantiated within HIE 62a, one or more AIs instantiated within a remote computing resource (not illustrated), and / or a combination of such content creators. New knowledge may be presented to the little girl 60a through audio-visual interaction with grandparents 61a, a cartoon-like character 64a on screen 62a, or a combination of such human and machine-based interaction modes.

[0081] After a short time, as shown in Figure 6B, the little girl 60b shows happiness, feeling that she has understood one or more aspects of the new knowledge that has just been presented. Machine-based identification of knowledge acquisition by the little girl 60b is: 1) The little girl is raising her hands 66a in joy, which is detected and observed by one or more cameras (not shown) in the HIE 62b; 2) The little girl 60b is displaying a broad smile and wide-open eyes 66b, which may also be detected by a camera in the HIE 62b; 3) The mother 67, who is near the girl and acting as an observer, is producing a broad smile 68a, which is detected by one or more cameras in the HIE (and / or HIA) 62b that are directed at the environment of the little girl and / or the mother; 4) The mother 67 is detecting one or more HIE 5) Grandparents 61b, as distant observers, utter the words 68b, “Yes… I did it!” as detected via a microphone (not shown) in 62b; they show a smile acknowledging understanding from the little girl 60b, as detected by a camera in a distant HIE (not shown); and 6) the grandparents also show that they believe the knowledge is understood through their own HIE (e.g., by touching a touchscreen, keyboard, or mouse); and / or 7) they simply move on to the next instructional topic, which may be determined (essentially simultaneously) as a result of several detected behaviors.

[0082] When the HIE determines that little girl 60b has grasped new information, it displays a cartoon-like character 64b on screen 62b that responds with a smile and / or an animation acknowledging and praising the little girl 60b's success in acquiring knowledge. In multiple ways, human-based and machine-based interactions may synergistically interact to provide a supportive emotional environment and monitor, encourage, evaluate, and reinforce learning.

[0083] Human Interaction Entity (HIE) HIE may comprise an interactive electronic device such as a desktop computer, portable computer, tablet, or mobile phone, which includes one or more processors, memory, user interfaces (e.g., including one or more sensors or input devices, and / or one or more actuators or output devices), communication interfaces, and / or other hardware, firmware, and software components, as better described in the accompanying description in Figure 7 below.

[0084] Some HIE (i.e., electronic device) functions may be implemented by additional operablely coupled systems (e.g., associated HIA, and / or remote servers including one or more processors, memory, and / or other hardware components and / or software modules). Optionally, the HIE may be instantiated by one or more artificial intelligence “personalities” (AIPs), “characters,” or “companions” that are friendly, socially acceptable, and enable an effective communication experience. The HIE may manage and implement the communication experience to enhance social connection and encourage learning.

[0085] Machine-based (i.e., electronic device) interactions with one or more humans carried out by an HIE may be 1) physical, 2) virtual, or 3) a combination of physical and virtual, especially within different times or different interaction environments (e.g., a physical device when the user is seated, and a corresponding virtual avatar displayed on a mobile device when the user is moving around). A physical HIE (and / or a collection of interacting and / or combined human interaction accessories (HIAs)) may include handheld interaction devices, robots (mobile or substantially stationary), robot pets, robot toys (e.g., dolls, teddy bears, baby-like figures, mechanical structures), and human-like objects (e.g., faces, heads, hands, whole bodies).

[0086] Virtual HIE features may have characteristics found in avatars, information terminals, cartoon characters, or synthesized humans or pets, and may be displayed on computer monitors, screens, laptops, mobile devices (phones, tablets, smartwatches, etc.), or other display devices including augmented reality, mixed reality, and virtual reality headwear. Virtual HIE may also be displayed as holograms, holographic-like projections, light field projections, and other techniques that make image objects appear three-dimensional.

[0087] In exemplary embodiments, the HIE (i.e., electronic device) may include environmental sensors (i.e., sensors that measure one or more elements in the HIE's environment) and human interaction sensors (i.e., sensors that measure the interaction between the HIE / HIA and the user). The environmental sensors may include one or more cameras (i.e., pointed towards the user's environment), light sensors, thermal sensors, motion sensors, accelerometers, Global Positioning System (GPS) transceivers, microphones, infrared (IR) sensors, galvanometric sensors, pressure sensors, switch sensors, magnetic sensors, proximity sensors, date and time clocks, Bluetooth transceivers, and Wi-Fi transceivers. The environmental sensors may also include devices worn by the user but pointed towards the environment, such as smart glasses (e.g., Google Glass), augmented reality headwear, earphones (e.g., with microphones), and smartwatches.

[0088] Human interaction sensors may include one or more cameras (i.e., pointed towards the user), thermal sensors, motion sensors, accelerometers, microphones, infrared (IR) sensors, galvanometric sensors, heart rate sensors, electrocardiogram sensors, electrooculogram sensors, electroencephalogram sensors, pulse oximeters, assistive devices, pressure sensors, magnetic sensors, activity monitoring devices, computer mice, joysticks, keyboards, touchscreens, and proximity sensors. Both environmental cameras and human interaction cameras may include pan and zoom capabilities.

[0089] The resulting interaction data, which may be used to classify the degree of emotional engagement and / or knowledge acquisition, may include one or more of the following: one or more audio recordings, video recordings, button presses, heart rate recordings, respiration recordings, biometric recordings, breath operation recordings, accelerometer movements, keyboard presses, computer mouse movements, joystick movements, trackball movements, touchpad touches, screen touches, pointing stick movements, eye-tracking recordings, rangefinder recordings, instrument recordings, and digital pen recordings.

[0090] If the HIE functionality includes physical objects or a collection of HIAs that interact with and / or combine, the permissibility of human interaction may be enhanced by movement, gestures, information display, pointing, sound, and other forms of HIE / HIA output or interaction. Actuators or other output components may include one or more image display devices, hologram display devices, holographic-like projectors, speakers, buzzers, propulsion systems, servos, motors, magnetic field controllers, orientation controllers, haptic controllers, laser light and other forms of pointing devices, switch controllers, actuators for accessory control, and controllable haptic surfaces.

[0091] In additional exemplary embodiments, the HIE output mode may vary depending on the time and circumstances. For example, the full range of HIE output may be used in a stationary environment where only video communication is performed in a mobile setting, and / or only audio output may be produced while driving. Generally, greater mobility is available using unconnected HIEs and / or HIAs (i.e., not connected to any fixed wired power source or telecommunications source). Connected HIEs may enable continuous interaction (with less concern about power consumption, battery life, etc.) and / or more advanced interaction devices (e.g., holographic displays, projected images, etc.). In further exemplary embodiments, a combined approach may be employed in which radio telecommunications is used for most daytime interactions, with battery recharging and / or database updates performed additionally when connected at night (e.g., while the user is sleeping).

[0092] The memories and experiences of a HIE may include those acquired as a result of interactions with the user via human interaction sensors. Such interactions are often multi-mode in nature, involving input from numerous sensors (sound, image, touch, bio, etc.), perceived across a wide range of scales (e.g., fine eye movements, larger head movements, whole gestures observed when the whole body is viewed, camera sensors detecting objects recognized in a landscape), and across a wide range of time scales (from milliseconds to months). The HIE may be updated in part or in whole based on the availability of new information and / or computing resources, as well as other forms of guidance provided by other parties, such as engineers, psychologists, counselors, teachers, advisors, supervisors, other humans, and / or other HIEs.

[0093] HIE (i.e., electronic device) software and / or firmware may be implemented by a combination of conventional programming and various forms of machine learning, including convolutional neural networks and / or deep learning techniques. The neural network may encompass large components of HIE behavior, such as the representation of a wide range of behaviors, as well as the control of actuators in the HIE and / or HIA. Alternatively, a combination of neural network and / or non-neural network-based programming may be segmented into subcomponents, each subsystem performing a specific task, such as identifying the degree of knowledge acquisition, recognizing emotions, retrieving strategies for acquiring new information, speech recognition, cursive writing recognition, word translation, speech formation, facial feature recognition, gesture recognition, natural language processing, animation display, joint movement control, or holographic display.

[0094] Figure 7 shows exemplary sensor and actuator components that may be included in one or more Human Interaction Entities (HIEs) and / or one or more Human Interaction Accessories (HIAs) communicating with the HIEs. One or more HIAs associated with an HIE may be considered to provide one or more subcomponents of the HIE, often unconnected, enabling improved mobility and portability (e.g., physically small, lightweight, inexpensive, low power consumption) during user interaction. In mobile situations, the HIE and / or HIA may include one or more disposable or rechargeable batteries 73b and one or more processing elements 73a with varying levels of computing power, ranging from field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs) to multiprocessors and hardware acceleration (e.g., for neural network processing). The HIE components 72a, 72b, 72c, 72d, 72e, 72f, 72g, 73a, and 73b may be interconnected via a printed circuit, which generally includes multiple paths and / or bus structures 73, for rapid inter-component communication.

[0095] The HIE (and optionally, one or more associated HIAs) 70 may include a range of components designed to direct information to the user, including one or more light sources 72b ranging from individual light-emitting diodes (LEDs) to multispectral display panels, one or more sound generating devices 72g ranging from piezoelectric buzzers to high-performance earphones or speakers, and other output (i.e., output from the perspective of the HIE) devices (e.g., haptic systems, projection systems, augmented reality displays).

[0096] The HIE (and one or more associated HIAs) 70 may also include a range of components designed to sense user responses, including, for example, a simple push button 72a fixed to the surface 71 of the HIE or HIA, one or more cameras 72d, 72e which may be directed towards an individual user 74a, another observer 74b, and / or the HIE environment, one or more microphones 72f which may sense sounds within the environment of the HIE or HIA, one-dimensional or multi-dimensional motion detectors (e.g., accelerometers) 72c which may sense motion such as shaking the HIE or HIA and / or tilting relative to Earth's gravity, and other input (i.e., input from the perspective of the HIE) devices (e.g., touch screen, computer mouse, trackball, keyboard, joystick, smart pen).

[0097] The HIE (and associated HIA) 70 may interact exclusively with a single human user 74a (75a). In this scenario, if other users are present, they may have their own individual HIEs or other processor-based interaction devices (e.g., tablets, laptops). Alternatively, the HIE may interact with multiple users (e.g., 74a, 74b) at any given time or at different times (75a, 75b). For example, the HIE may be used to present new knowledge to a learner (e.g., 74a). The HIE may use a push button 72a, a motion detector 72s, a camera 72d, and / or a microphone 72f to detect the learner's response. The same HIE may also use the same camera 72d or a different camera 72e and microphone 72f to monitor the learner's observer response. In other words, a single HIE may perform the combined functions of two or more HIEs.

[0098] In further exemplary embodiments, substantial processing may be required to put HIE-human interaction into action, including monitoring emotional engagement, delivering new knowledge, and subsequently monitoring knowledge comprehension. Different HIEs (and associated HIAs) may possess different levels of computing resources. If the computing resources available within the HIE are insufficient to deliver the interaction in a timely manner, sequencing and computation may be offloaded to one or more remote processors 77a, 77b, 77c (indicated by the dashed box 78 in Figure 7). This may be put into action using so-called cloud computing, which may include telecommunications protocols known in the art (e.g., Wi-Fi, TCP / IP) 76, as well as large processor arrays and / or hardware acceleration (in particular for neural network training and / or processing).

[0099] Optionally, HIE / HIA may be instantiated using an AI "personality" (i.e., embedded in software, firmware, and / or hardware). For the purposes of this application, AIP (i.e., artificial intelligence personality) is a construct of understanding that may generate machine-based interactions with one or more humans in a human, cartoon, or pet-like manner, substantially in real time. The AI ​​or AIP may include a background or database of 1) information, 2) memories, and / or 3) experiences. Including memories and knowledge in the form of a familiar character and / or a person-specific dialogue methodology may help improve the emotional and learning environment by avoiding potentially distracting novelty in the dialogue process and creating a more receptive and relaxed environment.

[0100] The AI's "personality" component may always be present, present only when selected, or completely unavailable. Examples of situations where effectiveness may be enhanced by having a "personality" present or at least available include on-screen cartoon-like character representations, including HIE in a child's favorite toy, and being always responsive to child prompts. Examples of situations where it may be preferable to conduct knowledge comprehension assessments and tracking without a "personality" present include during more formal examinations (e.g., for licensing, certification, or promotion), and / or in situations where the personality's memories and / or experiences are not deployed (e.g., questionnaires in a doctor's, legal counsel's, or insurance company's office).

[0101] In exemplary embodiments, HIE-based representations of knowledge or information may be 1) embedded in machines (e.g., software, firmware, and / or neural networks), and / or 2) incorporated through the ability to retrieve information when needed using telecommunications, such as internet and / or so-called “cloud” searches. Some information, such as birthdays and anniversaries of all users in a human network, as well as, for example, the user’s relatives and acquaintances, may be specifically taught to the HIE. Other information may be more generally known and available and accessed through internet search strategies known in the field. The retrieval and selection of information, along with the ability to synthesize new information from such multiple sources, greatly extends the “intelligence” component of the HIE. Further aspects of systems and methods for establishing cumulatively acquired AIPs of time division and / or time shifts are described in U.S. Patent Application No. 10,762,414 filed on 19 April 2019, U.S. Patent No. 10,915,814 filed on 15 June 2020, and U.S. Patent Application No. 17 / 081,806 filed on 27 October 2020, issued as U.S. Patent No. 10,963,816, all of which are expressly incorporated herein by reference.

[0102] Monitoring the degree of emotional involvement In exemplary embodiments, a wide range of modalities of the interaction component (e.g., audio, video, touch screen) may be monitored to determine the degree of emotional engagement by an individual. The interaction monitoring the degree of emotional engagement may be in response to actions or activities sourced from HIE and / or one or more people, or it may not be the result of any specific identifiable activity (i.e., the indication of the degree of emotional engagement may arise spontaneously).

[0103] As mentioned above, any number of degrees of emotional involvement may be determined, but the three degrees (i.e., boredom, engagement, and overwhelm) may be of specific values ​​when monitoring and subsequently supporting and / or adjusting an individual's emotional environment (see Figure 11). Starting with "engagement," common indications (and examples in parentheses) from many people include the following: • Words (for example, "interesting," "exciting") • Phrases (for example, "I like this," "That's it!") • Interjections (e.g., "OK", "Wow") • Vocalization (for example, "I did it," "Keep going!") • Gestures (for example, movements of large body parts that are specifically designed to attract attention) ·smile • Hand movements (for example, waving your hands, especially towards yourself and / or to get attention) • Finger configuration (e.g., raising the thumb, pointing) • Written symbols (e.g., bold text, large amounts of text) • Head movements (e.g., especially nodding) • Facial muscle movements (e.g., those associated with concentration) • Eye movements (for example, tracking an individual's activities within their environment in real time) • Press the HIE / HIA button and touch the screen (e.g., quickly and forcefully) • Responses related to engagement (e.g., indicating a desire for further engagement) • Questions related to the dialogue (for example, questions in particular about what any newly presented topic means) • One or more changes in tone of voice (in which case the details generally depend on the language) • The volume of the voice (for example, whether others can easily hear it), and • Immediate response to dialogue initiated by HIE or human.

[0104] If an emotional level of “boredom” (or similar terminology) is detected, the HIE may respond by taking direct action or prompting another person to initiate an activity that attempts to raise the individual’s level of interest and re-engage with either or both the HIE and the other person. Common indications of boredom (and examples in parentheses) include: • Words (for example, "It's boring!", "I'm tired") • Phrases (for example, "I've heard that before," "Aren't you done yet?") • Interjections (e.g., "What?", "That's annoying") • Vocalizations (for example, "I'm bored," "Ahhh") • Gestures (for example, swatting away) • Nodding (for example, a horizontal nod that indicates a lack of interest) • Hand movements (for example, waving goodbye) • Leg movements (for example, shaking) • Finger configuration (for example, lowering the thumb) • Written symbols (for example, lightly written symbols, scribbles) • Head movements (e.g., tilting to one side, remaining still) • Facial muscles (e.g., relaxed, inactive) • Eye movements (for example, gaze not directed at any of the individuals attempting to engage) • The position of the eyelids (e.g., partially or completely closed) • Pressing the HIE / HIA button and touching the screen (e.g., slow or unresponsive) • The movement of one or more objects detected by HIE / HIA (e.g., fiddling with or playing with an object that is not related to the interaction) No questions • Do not attempt to make inferences about the dialogue. • The volume of the voice (for example, barely audible or completely inaudible), • Long delay in responding to a conversation initiated by a human or HIE

[0105] If an emotional level of “overwhelmed” or “stressed” (or similar terminology) is detected, the HIE may directly intervene or encourage another person to initiate activities that reduce the individual’s stress level. This may involve, for example, slowing down the pace of the conversation (by the HIE or the other person they are talking to), suggesting physical activity, and / or changing the topic of conversation. Common indications (and examples) of being overwhelmed by many people include: • Words (for example, "Stop!", "What!") • Phrases (for example, "I'm tired," "Stop") • Interjections (e.g., "Wow," "No way") • Vocalizations (for example, "yes," "ouch") • Gestures (for example, showing the palm of the hand to indicate "stop") • Tighten the muscles around the mouth, or make you look sad. • Hand movements (for example, to reject someone trying to get involved) • Finger configuration (e.g., aggressive gestures) • Written symbols (for example, those written with force) • Head movements (e.g., especially horizontal movements) • Movements of the arms, legs, and other muscles (e.g., shivering) • Facial muscles (for example, overall tightening) • Forehead muscles (for example, when frowning) • Eye movements (e.g., stern, responsive) • Changes in pupil size (e.g., stress-related pupil dilation) • Blinking (noticeable in some individuals) • Press the HIE / HIA button and touch the screen (e.g., quickly and / or repeatedly) • Shake the HIE / HIA (for example, with excessive force) • The movement of one or more objects detected by HIE / HIA (e.g., an object being thrown or discarded) • Reactions related to attempts at engagement (e.g., rejection) • Responses to attempts to ask questions (e.g., responses unrelated to the question) • Responses to attempts at engagement (e.g., responses unrelated to the topic of engagement) • One or more changes in the tone of voice (e.g., a rise in pitch) • One or more changes in volume (e.g., loud, explosive), and • A short time between an attempt to engage and a (possibly irrelevant) response

[0106] Mechanical activities perceived by HIE / HIA (e.g., pointing, shaking, pressing buttons) can be particularly useful (in combination with audio and video data) in assessing the degree of emotional engagement. For example, performing activities requiring a degree of precision (e.g., drawing, typing, driving a virtual car) may indicate a high level of engagement. High-level activities (e.g., pressing multiple buttons, continuous movement, rapid response) may also indicate engagement. On the other hand, prolonged inactivity in HIE / HIA (especially during the main part of the activity) and / or delayed responses may indicate a state of absentmindedness (e.g., boredom, distraction).

[0107] Many of the same dialogue modes (e.g., words, interjections, gestures) may be used by the observer to indicate an individual's emotional state (to the HIE). In fact, the observer may indicate their level of satisfaction when determining that the individual is in a state of “engrossment.” However, the classification scheme must distinguish features in the observer’s dialogue from features used to directly detect an individual’s emotional state. For example, an observer may indicate complete engrossment when communicating an observation to the HIE, but such an observation may indicate that the observed individual is, for example, in a state of “boredom” or “overwhelmed” (i.e., different from the observer).

[0108] The following distinctions can be made between the observer's dialogue and the observed individual's dialogue: 1) The observer generally uses second-person (e.g., "You did a great job!") and / or third-person (e.g., "He's excited!") language during dialogue referring to the observed individual. 2) HIE / HIA dialogue by the observer may be filtered to consider only that which occurs when it is detected that the observer is looking in the direction of the monitored individual (visibility may be conducted remotely via one or more cameras) and / or listening to the monitored individual. 3) The observer may convey the individual's emotional state by mimicking the observed individual's responses (e.g., hand gestures). 4) The observer may respond to HIE interrogations (e.g., pressing virtual or real buttons). 5) Some observers (e.g., parents observing a child) may have an emotional attachment. In such cases, the observer's emotional responses may be used as a contribution to help classify the observed individual's emotional state. For example, a parent might express frustration when observing a child who is spaced out, or happiness when the child is engrossed in something.

[0109] Monitoring the degree of knowledge acquisition As described in the "Summary of the Invention" above, a key aspect of both teaching and learning is the ability to recognize when and to what extent knowledge acquisition is occurring. The ability to recognize when some degree or level of knowledge understanding (e.g., from superficial to thorough) has been achieved is a valuable attribute of a skilled instructor (e.g., a professional teacher, colleague, or friend). As a result, such an instructor may be able to "pace" the presentation of new, synergistic, and / or complex material to the learner at a speed that does not overwhelm the learner while maintaining a high level of interest (and other emotions). Optimally, the identification of when learning has occurred may be performed in real time as each new knowledge element is presented.

[0110] As approaches to both teaching and learning become more complex, complete knowledge acquisition may not be a single event. Knowledge acquisition may be staged, including an initial understanding that may then be reinforced and / or strengthened one or more times (e.g., by reintroduction and / or mental repetition of the material). In a similar trajectory, a broad spectrum of understanding frequently occurs, ranging from superficial learning, including simple memorization, to deep understanding, which may be used to generate new reasoning and support complex decision-making.

[0111] In exemplary embodiments, a wide range of modalities of the dialogue component (e.g., audio, video, touch screen) may be monitored to determine the presence (in contrast to absence) of learner knowledge comprehension. Even within each mode, the dialogue may contain multiple cues for comprehension. For example, one or more cameras may detect both a nod and a smile as cues for comprehension, and a microphone may simultaneously detect expressions such as "I understand." Common cues (and examples) for knowledge comprehension include: • Words (for example, "OK," "Understood," "I agree") • Phrases (for example, "I understand," "I see," "Amazing") • Interjections (e.g., "OK," "Yes," "Oh") • Vocalization (for example, "yes," "hmm") • Gestures (e.g., waving, swiping) • Nodding (for example, especially vertical nodding to indicate understanding) ·smile • Hand movements (e.g., waving hands, tracing shapes such as checkmarks, typing on a keyboard) • Finger configuration (e.g., raising the thumb, pointing) • Written symbols (e.g., checkmarks, stars, etc., indicating the correct answer from a selection of possible answers) • Head movements (e.g., tilting in a specific direction, especially vertical movements) • Movement of facial muscles (for example, creating patterns of wrinkles, especially around the eyes) • Movement of the muscles in the forehead (for example, relaxing a frown) • Eye movements (e.g., looking up, focusing on review topics) • Changes in pupil size (for example, relaxation of stress-related pupil dilation) • Blinking (noticeable in some individuals) • Pressing the HIE / HIA button (e.g., a physical or virtual button for quick selection) • Touch the HIE / HIA screen (for example, generate a drawing, select the correct answer) • Shake the HIE / HIA (for example, in the sense that it may be pre-assigned depending on the number of shakes and / or direction of the shake) • The movement of one or more objects detected by HIE / HIA (e.g., the movement of a pen or pointer) • Reactions related to knowledge and understanding (e.g., joy, disappointment) • Questions related to knowledge comprehension (for example, questions specifically about what new knowledge means) • Instructions related to knowledge and understanding (e.g., solving related problems) • Reasoning related to knowledge and understanding • One or more changes in tone of voice (in which case the details generally depend on the language) • One or more changes in volume (loud voice, assertion), and • Time between the expression of new knowledge and one or more responses The latter criterion (i.e., the time between the representation of knowledge and one or more responses) may be used synergistically (i.e., within a multifactorial process), for example, as one element to help measure the confidence level of some of the other responses, in which case shorter time is generally associated with a firmer grasp of the new knowledge or the possibility that the learner may have encountered the knowledge previously.

[0112] In a similar manner, a wide range of modalities within the dialogue component may be monitored to determine whether the learner lacks or is missing knowledge (which may be referred to herein as “confusion,” but may include other forms or the degree of knowledge loss). Within each mode, the learner’s actions may include multiple indicators of confusion, which may contribute, as further described below, to a score or criterion of confidence that the learner does not understand the new material, as well as a criterion of the degree to which the material is not understood. Common indicators (and examples) of knowledge loss and / or confusion include: • Words (for example, "repeat," "hate") • Phrases (for example, "I don't understand," "I don't get the meaning") • Interjections (e.g., "What?", "No") • Vocalization (for example, "Eh?") • Gestures (e.g., waving, shrugging) Shaking your head (for example, horizontally, especially to indicate that you don't understand) • Downcast expression • Shrug your shoulders • Hand movements (for example, waving your hands, tracing the shape of an "X") • Finger configuration (e.g., thumb pointing downwards, pointing in a different direction) • Written symbols (for example, an "X" to indicate an incorrect answer from a list of possible answers, or a strikethrough on existing text or images) • Head movements (for example, especially horizontal movements, tilting in a specific direction) • Movements of facial muscles (for example, patterns of wrinkles, especially around the eyes, squinting) • Movements of the forehead muscles (for example, especially those involving lifting the upper eyelids) • Grimacing (including, for example, contraction of the muscles around the forehead, eyes, and mouth) • Eye movements (for example, looking up, concentrating intensely) • Changes in pupil size (e.g., stress-related pupil dilation) • Blinking (for example, frequent blinking, especially by several individuals) • Press the HIE / HIA button (for example, a delay with one or more corrections) • Touching the HIE / HIA screen (e.g., generating a drawing for clarity, selecting an incorrect answer) • Shake the HIE / HIA (for example, in the sense that it may be pre-assigned depending on the number of shakes and / or direction of the shake) • The movement of one or more objects detected by HIE / HIA (e.g., the movement of a pen or pointer) • Reactions related to knowledge understanding (e.g., frustration, anger) • Questions related to knowledge comprehension (for example, asking about new knowledge being expressed in different ways) • Instructions related to knowledge (e.g., inability to solve problems related to new knowledge) • Incorrect reasoning related to understanding new knowledge • One or more changes in tone of voice (in which case the details generally depend on the language) • One or more changes in volume (quiet, doubtful), and • Time between the expression of new knowledge and one or more responses The latter criterion (i.e., the time between the expression of knowledge and the response) may be used to help measure confidence in a simultaneous dialogue, in which case longer times may be associated with a lack of grasp of new knowledge. However, in this case, the pause or delay may also suggest that the learner is taking time to "think about" the consequences based on the new information, and perhaps even discovering a new inference. Therefore, the pause or delay may be considered a criterion of low confidence in a lack of grasp of information, unless combined with other criteria as described above.

[0113] In further exemplary embodiments, as an alternative to or in addition to machine-based direct monitoring of learners, indications of knowledge comprehension and / or confusion may be determined based on responses by one or more observers of the learner. The observers may simultaneously, and often without much thought, monitor any range of responses that indicate a lack of knowledge comprehension or any of the comprehensions listed above. The observers may also have a better understanding (compared to machine-based analysis) of the individual learner's prior background, including factors such as emotional maturity, typical timing of responses when grasping concepts of varying degrees of complexity, development of interests in a range of topics, learning habits, and familiarity with the learner's expressions, including those expressed verbally, facially, with hand gestures, culturally, linguistically, etc.

[0114] Similar to the strategies described above for monitoring emotional states, separation is necessary when using observer dialogue to identify and develop classification schemes. For example, a sense of confusion expressed by a learner is likely to be related to a lack of understanding of some newly presented knowledge, while confusion expressed by the learner's observer may be related to not knowing whether or not the learner understands elements of the newly presented knowledge. In other words, confusion by the observer may be unrelated to whether or not the learner understands the newly presented knowledge.

[0115] In further embodiments, machine-based analysis of observer dialogue and learner dialogue may be distinguished, and observer dialogue may be isolated to those that are particularly relevant to the learner. For example, the identification of utterances (e.g., "You got it!"), interjections, comments, and other indications of knowledge comprehension (or lack thereof) may be isolated from those that appear to be directed at the learner. Observer dialogue may take into account, for example, whether the observer is looking at the learner and / or within audible distance from the learner (including via remote access). Observer dialogue that appears to be directed at other individuals or objects in the observer's environment may be excluded from the database accumulation for evaluation of learner performance and / or observer indications of the learner's knowledge acquisition level.

[0116] In summary, machine-based assessments of learner comprehension based on learner interactions with observers may involve any or all of the exemplary forms of the interactions listed above that indicate comprehension and / or lack of comprehension. However, the accumulation and maintenance of observer databases may take into account observer intent and / or whether the observer's interactions are relevant to the learner. Databases used to train or deploy observer classification schemes to measure knowledge acquisition may maintain such distinctions. Machine-based analysis of observer interactions may take into account observer intent (and, if desired, criteria for the reliability of that intent) in addition to utilizing interaction information to assess the degree of knowledge acquisition by the learner.

[0117] In further additional exemplary embodiments, responses by the learner and / or one or more observers of the learner may be accumulated by 1) viewing knowledge understanding as a continuum from deep understanding at one end of the spectrum to a complete lack of understanding or confusion at the other end of the same spectrum, or 2) treating levels of understanding or confusion as any number of distinctly identifiable degrees of knowledge acquisition. Scales that are essentially continuous or have multiple stages may include identifiable characteristics, for example (roughly including understanding from high to low): • A substantial understanding sufficient to make inferences. • An initial understanding of a complex topic that is likely to require at least several repetitions or synergistic instruction for a more complete understanding. ·Superficial understanding • Responses that appear to have been memorized (i.e., correct answers, but not fully understood) Some aspects may be correct, but others are incorrect; a partial understanding. • Uncertainty regarding understanding • Acceptance of not understanding, and • One or more responses that indicate an incorrect understanding (e.g., leading to incorrect reasoning)

[0118] Within either approach (i.e., a continuous spectrum or any number of discrete levels of understanding), identified dialogue indicators, as well as criteria for the intensity and reliability of (optionally) pre-assigned specific references, can help determine both the degree of knowledge understanding and the confidence level in its classification. The presence and / or absence of different indicators (including conflicting indicators), the timing of those indicators, the synergistic relationships between indicators, pre-assigned confidence levels for some or all indicators, and references by either or both the learner and / or observer may be included in the rating of such confidence levels.

[0119] To illustrate the distinction between scoring the degree of knowledge acquisition and the reliability of that scoring, the phrase "I understand" uttered by a learner is a highly reliable indicator of the learner's potential for a high level of understanding. On the other hand, a learner's smile may indicate understanding, or it may be a reaction to an unrelated event in the learner's environment. Therefore, a smile itself may be a low-reliability indicator of knowledge understanding, but when combined with one or more indicators (e.g., raising an arm as shown in Figure 6B), a synergistic (i.e., multi-factor) combination of indicators may result in a high level of reliability. Identifying synergistic combinations of indicators of knowledge understanding is an example of using a multivariate approach within a classification scheme, which will be discussed further later.

[0120] New knowledge that produces a faster and / or complete understanding may be given a greater weight in the database of knowledge (i.e., teachings) that produce a greater understanding. Such weights may be used, for example, in the calculation of confidence levels and / or effectiveness in classification schemes (e.g., neural networks, SVMs, hyperplanes) to determine enhanced teaching strategies. Where only partial understanding or confusion by the learner is identified, the knowledge database may also include unsuccessful teaching strategies that may be avoided, for example, between future teaching sessions and / or teaching strategies directed to other individuals.

[0121] Figure 8 is a flowchart showing steps that may be used to iteratively develop a dialogue database, including 1) responses by learner 85 to knowledge presentations, and / or 2) responses by one or more observers 88 of the learner to knowledge presentations. Responses by learners 80a, 80b may be perceived by the learner's HIE 82a and / or include input from devices such as keyboards, mice, push buttons, etc. (not shown). Responses by one or more observers 81a, 81b, which may be perceived by the observer's HIE 82c and / or the learner's HIE 82a, may include not only those based on visual “observing” learner 80a, but may also be based (partially or entirely) on, for example, auditory confirmation, recognition of one or more choices by the learner using a keyboard or other data input device, and / or one or more responses to questions to learner 81a.

[0122] In additional exemplary embodiments, knowledge may be presented to individual 80a by one or more human beings 81a and / or HIEs 82a, or both. During and after the presentation of new knowledge, dialogue data from learner 80b may be acquired by the learner's HIE 82b. This dialogue data is processed (locally or using one or more remote processors not shown) to identify one or more dialogue components, including learner responses 83. Timestamps may be generated (84), and the identified dialogue components may be added to a database of appropriate responses (e.g., knowledge understanding, partial understanding, confusion, and / or continuum indication) 85.

[0123] Dialogue data may also be collected to determine one or more responses of one or more observers (if any) 81b of the learner 80a. One or more observers 81b may or may not be involved in the presentation of knowledge 81a to the learner 80a. During or after the knowledge presentation, observer dialogues may be acquired by one or more observer HIEs 82c, or by an HIE associated with the learner (i.e., a single HIE may interact with two or more people). Dialogue data may be processed (locally or using one or more remote processors) to identify one or more dialogue components, including identified observer responses 86. A timestamp may then be generated (87), and the labeled dialogues may be added to the appropriate observer database 88.

[0124] Dialogue labeling used to train classification schemes As particularly detailed in the accompanying explanation in Figure 3, a collection of training databases of dialogues “labeled” on either a continuous spectrum or discrete levels of emotional engagement and / or knowledge comprehension is a fundamental aspect of developing the classification scheme. Labeling may be performed in real time as the dialogue is being conducted. Alternatively or in addition (e.g., particularly when labeling may require correction), labeling may be performed at some point after machine and / or human review, for example, as a result of formal written / online assessments and / or further individual questioning, where the degree of knowledge comprehension may be more evident.

[0125] In knowledge acquisition, the identification of levels of understanding may be spontaneous by the learner. For example, the learner may verbally indicate "I understand" or "I don't understand" during the dialogue. The dialogue may also be elicited in response to HIE and / or human questions. Such prompting responses represent an exemplary way of developing a labeled database of cognitive response elements in an overt form.

[0126] In more "confidential" forms (i.e., where interruptions generated during interaction with an individual are less noticeable), using the organization of the feedback loop described above, dialogue labeling may also be intentionally induced by the HIE (or a human collaborating with the HIE) by presenting the learner with cognitive or emotional content that results in predictable (i.e., known to the machine) dialogue elements. For example, the learner may be given a request to solve a mathematical equation that is beyond their current developmental level, the text may be presented in a foreign language, or the learner may be asked a question whose answer is clearly outside of their experience. Dialogue elements resulting from such tasks may be labeled as indicators of a lack of understanding (i.e., without requiring any other form of labeling input).

[0127] Similarly, the HIE (or a human collaborating with the HIE) may reintroduce material previously identified as understood by the learner. Such presentations may be intended, at least in part, to elicit one or more responses from the learner indicating a degree of knowledge comprehension (e.g., well understood, boring, partially understood). The learner's dialogue and / or indicators by the learner's observer may then be labeled according to such (i.e., machine-determined) degree of understanding.

[0128] In exemplary embodiments, more generally, a topic of known understanding may be presented to the learner, in which case at least one purpose of the presentation is to elicit a dialogue that may later be labeled according to the degree of understanding known to the machine. By controlling the cognitive environment, such machine-based methods may be used to train one or more machine-based classification schemes for a particular individual (e.g., in real time), or more generally, to classify the responses of any individual.

[0129] A similar approach may be put into practice to evoke a specific emotional response, in which case dialogue elements or indicators within the review dialogue may be labeled according to a given or expected emotional response. This may be done in an overt form, for example, the learner may simply be asked, “Are you bored?” or “Are you overwhelmed?” The emotional content presented before such a question, as well as one or more responses (e.g., “yes,” “no”), may then be used to categorize the identified response elements and the previous content that produced the response.

[0130] Another overt mode of evoking specific (i.e., machine-known) emotional response elements involves requiring learners to “think,” consider, ponder, and / or communicate about topics that produce known (i.e., self-identified) emotional responses. For example, an individual might be asked to think about or talk about their favorite pet. If the individual describes their favorite pet (e.g., a cat, a dog), such a description would typically include several indicators of engagement and increased excitement. The dialogue may further be categorized as content capable of reliably producing known responses (i.e., interest, happiness). Even if a response to the topic range is not explicitly communicated, the emotional dialogue elements brought about by simply thinking about a favorite pet may be recorded and labeled as indicating interest or engagement.

[0131] Any number of topic ranges may be taken up for the purpose of evoking such emotional responses, including, at least in part, asking about one's closest friends, favorite songs, favorite foods, cherished toys, best video games, etc. A similar process for additional classification may be performed if learners are asked to think about something boring or something that evokes feelings of overwhelming or stress (e.g., losing a friend).

[0132] Topics for which responses are known may also be introduced periodically to determine whether they evoke emotional responses that fall into substantially the same categories as those in previous dialogues covering the same topic. If, as a result of addressing such a topic, emotional responses fall into unexpected categories, this may be an indication that the learner is not in an attentive state (e.g., the learner is overwhelmed, unwell, distracted, or tired).

[0133] Topics with predetermined or nearly universal classifications may be introduced (by humans or HIEs). Introducing an upcoming holiday, award notification, or favorable weather forecast most often results in an increase in engagement response. On the other hand, informing an individual of multiple nearby dangers is more likely to produce a sense of overwhelm and / or an increase in stress levels. These programming strategies as a whole are repeatedly applied to grow and enhance the diversity of instructions in the emotional engagement database and are sometimes called “Application-Driven Human Labeling” (ADHL).

[0134] Figure 9 outlines such public and private methods for creating a database 97 of labeled dialogue elements. Labelled dialogues selected to produce known emotional responses 90 (to one or more machines) are put into action 91. The degree of selected emotional engagement may result from a predetermined set of selections of humans, any local or remote computing device (in particular, those instantiated with AI), and / or, for example, a broad spectrum of emotional responses (e.g., by cognitive scientists).

[0135] The dialogue may be assigned an emotional label in 90 based on 1) responses produced by most humans (e.g., stress responses to images of tragic scenes), 2) previous responses by the individual 94a, or 3) suggesting or directly stating the target's emotional state (e.g., "What's your favorite sport?", "Think about your favorite vacation"). Actions directed at the individual 94a may be carried out through a machine-based dialogue, one or more human supporters 92, or both. The resulting responses 94b may be collected by a machine-based entity 93b, labeled in 96 according to selected emotional states, and added in 97 to one or more databases. Separate databases may be assigned to individuals in, for example, different cultures, age groups, combinations of human supporters, and / or HIE dialogue capabilities.

[0136] In most respects, the strategies outlined in Figure 9 may also be applied to elicit responses to known cognitive presentations. In this case, cognitive problems may be presented that are assumed to be either well within the learner's capabilities or significantly beyond them. The elicited response elements may then be labeled according to the pre-assigned difficulty level of the presented problem. Similar to the segmentation of databases described above with respect to the affective dialogue database, the database resulting from the classified learning dialogue elements may be stored in a database applicable to a broad (e.g., global) population, or separate databases may be maintained for different age groups, learning environments (e.g., formal classroom vs. play at home), ranges of learning topics (e.g., science, art, music), HIE dialogue abilities, etc.

[0137] Figure 10 illustrates several strategies for feeding labeled user (or observer) dialogues representing an individual's emotional state into one or more databases. In this example, three databases 100a, 100b, and 100c are stored, representing, for example, the three levels of emotional engagement shown in Figures 1 and 11. The leftmost database 100a may represent an emotionally “bored” or absent-minded state. The indicators in the middle database 100b represent an “engaged” emotional state, and the indicators stored in the rightmost database 100c may represent an overwhelmed and / or stressed emotional state. At least one indicator occupies each database, with the indicator represented by a solid 101a existing in database 100a for the bored emotional state, the indicator represented by a checkered pattern 101b existing in database 100b for the engrossed state, and the dotted indicator 101c occupying database 100c for the overwhelmed state.

[0138] The central portion of Figure 10 includes four timelines 102, 103, 104, and 105, each representing the occurrence of HIE-sensing interactions with different fill patterns. Within each timeline, two representative interaction "modes" or types are shown (e.g., modes 106a and 106b in timeline 102). These representative modes may be sensed, for example, by the HIE's video component (e.g., 106a) and audio component (e.g., 106b) (see Figure 7). During such interaction sessions, additional modes (not shown) may be sensed simultaneously (e.g., pressing a push button, mouse click, heart rate monitor).

[0139] One of the elements employed in Figure 10 to accumulate labelled emotional state indicators is the observation that multiple indicators of a particular emotional state are often generated by an individual (or an observer of the individual) in temporal proximity. Therefore, if one dialogue indicator of an emotional state is identified, other dialogues within the same time frame (i.e., before and after the identified dialogue) may indicate a similar emotional state. Temporarily adjacent indicators may be recorded in response to an event or activity (as shown in the top two timelines 102, 103) or may be spontaneously generated and identified as a result of continuous monitoring of the individual (as shown in the bottom two timelines 104, 105).

[0140] In the top timeline 102 of Figure 10, monitoring revealed that dialogue 102a was identified as indicating boredom (i.e., indicated by a filled pattern, previously identified and stored in the “boredom” database 100a (101a)). Within this exemplary scenario, the presence of a prize (indicated by a “winner” banner 105a) then alerted the individual, thereby rapidly highlighting their level of interest, as detected by a dialogue pattern represented as a checkered pattern 102c (i.e., element 101b in database 100b) previously identified as indicating an “engaged” state. Other dialogues, both dialogues 102b prior to the identified “engaged” dialogue 102c and dialogues 102d, 102e, and 102f following it, may also be added to the database of potential indicators of engagement as a result. Alternatively, or in addition, if an action in a dialogue (e.g., winning prize 105a) has a predictable outcome (e.g., excitement and enthusiasm), all dialogue immediately following that action may be labeled and archived according to the predicted outcome (e.g., "enthusiasm" as the declared result by the winner).

[0141] In this exemplary scenario, two potential indicators 102d and 102f, indicated by upward-sloping diagonal lines, are associated with a mode (e.g., video) linked to timeline 106a, and two potential indicators 102b and 102e, indicated by downward-sloping diagonal lines, are associated with a second mode (e.g., audio) within timeline 106b. Such labeled dialogues (and / or templates representing essential elements of dialogues) may be accompanied by a database of labeled emotional state dialogues. For example, dialogues represented by upward-sloping diagonal lines 102d and 102f and downward-sloping diagonal lines 102b and 102e may be added to dialogues represented by a checkered pattern 109b in an updated “Intoxication” template database 108a (shown in the lower part of Figure 10). In timeline 102, an additional dialogue represented by X pattern 102g was identified, but it occurred considerably after the emotional trigger event 105a and / or was not determined to reliably indicate “enthusiasm” or another emotional state.

[0142] The initial emotional state within timeline 103 was one of the interactions, as indicated by the early dialogue represented by the filled-in check pattern 103a. The individual was then startled three times (e.g., by a loud noise) at 105b, 105c, and 105d. Between the second surprise 105c and the third surprise 105d, the individual remained engrossed, as indicated by the downward-sloping diagonal line pattern, but after the third surprise 105d, indications of feeling overwhelmed or stressed were observed, as represented by the dotted pattern 103d. This was temporally correlated with two responses having two dialogue responses represented by the double-circle patterns 103c and 103e. In this scenario, dialogues represented by the double-circle pattern were consistently associated with previously identified indicators of being overwhelmed (see 105a, 105b, and 105c), leading to the inclusion of dialogue indicators represented by the double-circle pattern, in addition to those represented by the dot pattern 109c, in an updated database 108c of dialogues labeled as being in an overwhelmed emotional state. Subsequent additional dialogues represented by the X pattern 103f were not reliably correlated with any of the emotional states.

[0143] Individuals may be monitored indefinitely, with or without actions that may trigger specific emotional responses, to help accumulate any number of indices for a labeled emotional state (symbolically represented by vertical ellipsis 107 in Figure 10). In timeline 104, two indications for the emotional state of "boredom," represented by fill patterns 104a and 104c, coincided with another indication 104b, represented by a mesh pattern. If a dialogue represented by a mesh pattern is determined to consistently indicate such a bored state, the dialogue (or a template representing the dialogue) may be added to the fill pattern index 109a in the "boredom" database 108a.

[0144] As described above, dialogues may occur that do not match other known indicators of the identified emotional states (e.g., 104d in timeline 104, and 105d and 105e in timeline 105, represented by the X pattern). Such dialogues may be discarded or await further analysis after the dialogue database has been further developed (e.g., with more and / or more reliable indicators).

[0145] Even without an HIE initiation action, the confidence in identifying and / or identifying additional indicators of known emotional states may be enhanced by using indicators of such emotional states. For example, in Figure 10, the indicator of "intense" represented by the checkered patterns 104f and 105g appear consistent with the indicators represented by the upward-sloping and downward-sloping diagonal patterns (104e in timeline 104, as well as 105f, 105h, and 105i in timeline 104), supporting the view that templates for these dialogues may be included in a database representing the "intense" emotional state. In a similar flow, the dialogues represented by the double circles 105b and 105c coincide with the dialogues identified by the dot pattern representing the "overwhelmed" state, supporting the action of placing the double-circle dialogue patterns into the "overwhelmed" database 108c.

[0146] Such databases may be infinitely refined. Separate databases may be deployed for individuals (for example, initially using a more generalized database as a starting point and / or incorporating “memories” associated with the individual), or for different groups of individuals, including those of different ages, cultures, backgrounds, languages, dialogue settings, etc.

[0147] Adjusting the level of emotional involvement Figure 11 is a flowchart outlining an embodiment in which the emotional state of an individual 110a is monitored on a machine basis, and based on the interaction, actions or activities designed to maintain the individual's emotional state in an engrossed state 115 are implemented (by machine and / or through one or more people). Furthermore, if the emotional state is determined to be "engrossed," new material and / or other dialogue activities may be introduced (116b), in which case the timing of the presentation 117a may be selected to help maintain the engrossed state (117b).

[0148] In the scenario shown in Figure 11, individual 110a interacts with one or more machine-based characters 112a and one or more human beings 111 in 113. Spontaneous, dialogue-based emotional responses by individual 110b are monitored by machine-based 112b in 114. The emotional responses are then categorized in 115 into one of three categories: 1) boredom, 2) enthusiasm, or 3) overwhelming.

[0149] If individuals 110a and 110b are rated as bored or indifferent, actions 116a may be selected that optionally include “signaling” to a human participant 111 to increase individual 110a’s level of excitement and / or interest. These may include, for example, a wide range of activities designed to increase the level of engagement (see below), a gesture to regain attention, and / or an increase in the speed and / or complexity of introducing a new topic and / or activity. On the other hand, if individuals 110a and 110b are rated as overwhelmed or stressed, actions 116c may be selected that reduce the stress level. These may include, for example, taking a break from studying, incorporating a joke, and / or slowing down the speed and / or complexity of the topic or other activity being worked on.

[0150] In further exemplary embodiments, if individuals 110a, 110b are assessed to be emotionally engaged and interested, a determination may be made in 116b to assess whether the pace of the individual-machine-based and human interaction is within a range that maintains the level of interest (for example, based on the interest expressed with respect to the topic range and / or the pace at which new material is presented during the previous interaction). If necessary (i.e., to maintain an optimal pace), new actions may be delayed in 117b, and / or the human participant 111 may be “signaled” by machine-based recommendations to slow down (or speed up) before putting into action a new teaching topic range in 113 and / or before returning to other activities.

[0151] Regardless of the specific activity, the timing of such actions can be strongly influenced, including 1) the speed at which the subject matter is introduced (taking into account the complexity of the new subject matter), and 2) the timing of the reintroduction of subject matter related to and / or synergistic to the introduced topic. The degree of emotional engagement in “boredom” generally signals the need to increase the speed at which new subject matter is introduced (unless simply overall indifference). In such situations, the increased speed may be applied progressively until an emotional state of “enthusiasm” is achieved. Conversely, sensing a degree of emotional engagement in “overwhelmance” generally signals the need to decrease the speed at which subject matter is introduced. The speed may be gradually reduced until indications of feelings of overwhelmance are no longer detected.

[0152] An archive (i.e., memory) of the rate at which a specific individual maintains an emotional state of "engagement" may be used in future monitoring and activity adjustments to target the optimal rate of topic presentation (i.e., the individual's cognitive ability). Rates may also be accumulated and analyzed at the level of a large population or a subset of the population, in which case the initial rate of knowledge introduction that is likely to achieve an emotional state of engagement may be determined according to factors such as age and / or educational background. During deployment, the initial rate of presenting and / or repeating the material may be adjusted based on such factors, if known.

[0153] Repetition can be an important mode of learning, helping to transfer knowledge elements or skills from conscious to unconscious. Among all different learning styles (as previously discussed, for example, in the section on "Background Skills"), repetition may work synergistically to reinforce understanding and retention. However, unnecessary repetition can quickly become counterproductive, as is the timing of repetition of key components of optimal learning (e.g., between the initial concept introduction and reinforcement steps).

[0154] The risks of unnecessary and / or inappropriately timed repetition, and the resulting boredom, are exacerbated by the increasing prevalence of both distance learning and distance communication. In virtual classrooms, teachers and other instructors may have little to no coordination or understanding of the unique blend and / or sequence of topics pursued by a particular student (child or adult). In a similar vein, as the world becomes increasingly reliant on telecommunications for both educational and social interaction with learners and loved ones, it may become increasingly difficult to assess emotional states (e.g., over distance) and / or whether genuine learning is taking place. Distance communication can easily circumvent subtle guidance by learners, facilitators, and / or instructors regarding either knowledge understanding or confusion.

[0155] The phrase "repetition breeds boredom" succinctly summarizes numerous adages and observations on the potential negative effects of repetition. Countless studies have focused on the impact of boredom on motivation, learning strategies, cognitive resources, self-regulation, and academic growth. Common metrics used to assess these effects include classroom attention and school dropout rates. Such surveys repeatedly show that only a small number of students reported not experiencing boredom during their high school education. When topic acquisition is tracked in real time, there is an opportunity to reintroduce each topic at optimally spaced intervals designed to reinforce each topic, optimizing the consolidation of long-term memory.

[0156] Early research in the late 19th century on spaced repetition to enhance long-term memory was reported in Hermann Ebbinghaus's groundbreaking 1885 monograph [Ebbinghaus H (1885); trans Ruger HA, Bussenius CE (1913) [Memory: A Contribution to Experimental Psychology] (Columbia University Teachers College, New York)]. Numerous (traditionally flashcard-based) systems propose schedules to enhance the memorization of new knowledge according to a set schedule. There are also numerous variations of such learning schedules and the basis for applying them. For example, within several schedules based on a projected "forgetting curve," learning is enhanced when the probability of recall falls into the 80%–90% range. This reinforces reminders approximately one day, one week, one month, and two months later.

[0157] In exemplary embodiments, machine-based monitoring and subsequent tracking may be used to select the timing of reintroducing a topic (e.g., in 117a of Figure 11) to adapt to such “spaced repetitions” or other timing strategies. Targets for engagement timing may also be adjusted based on the individual’s abilities, preferences (e.g., those of a parent or mentor), and / or circumstances (e.g., the individual’s mood). For example, temporary targets may be based on the child’s age, whether the individual is part of a structured learning environment or a larger group, the deadline for completing the entire task, the time of day (e.g., the level of engagement decreases before bedtime), etc.

[0158] Figure 12 shows illustrative timelines 120a, 120b of machine-based tracking of spaced repetitions following the presentation of new knowledge 121a, 121b, 121c, 121d. Within the upper timeline 120a, two different topics 121a, 121b were introduced to the learner (by the HIE and / or one or more people (not shown)) at two different times 123a, 124a. The first topic 121a was re-presented after its initial introduction 123a at three optimally spaced times 123b, 123c, 123d (indicated by arrows pointing in the direction of the timeline). The second topic 121b was introduced slightly later than the first topic 124a. Therefore, the three optimally spaced iterations 124b, 124c, and 124d of the second topic 121b each followed the spaced iterations of the first topic 121a at approximately the same time intervals.

[0159] The timing of machine-based tracking and topic reintroduction may involve any number of topics and may occur between any number of other ongoing activities by individuals (represented as filled circles 122 within timelines 120a and 120b). Knowledge initially presented by a human may involve "signaling" or suggesting to the human when the subject matter should be reintroduced at the optimal time. Otherwise, another human may be signaled, and / or appropriately timed HIE dialogues may be generated.

[0160] The lower timeline 120b in Figure 12 illustrates an exemplary scenario in which intervening repetitions of new knowledge 121c may be considered interrupted and / or unnecessary. Following the initial presentation 125a and single repetition 125b of knowledge 121c, shown above the lower timeline 120b, the learner (correctly) remeasured the new knowledge, represented by an arrow pointing away from the timeline 127. As a result, further repetitions of the newly introduced knowledge may be considered unnecessary, represented by a dashed line 129a with no arrow indicating the time of the repetition.

[0161] Below the lower timeline 120b, new knowledge associated with a time deadline (represented by a calendar event 121d) is shown, introduced with a single reminder 126b, representing 126a. For example, the new knowledge might include instructions on how to operate the tools needed to perform a friend's hobby, for which a gift-giving anniversary is approaching. Once the anniversary time 128 has passed, no further reminders or re-presentations of the knowledge 121d may be necessary, as indicated by the dashed line 129b (without an arrow pointing towards the timeline) spanning the entire remainder of timeline 120b.

[0162] Numerous additional circumstances may result in delays or termination of the tracking and repetition of knowledge presentation. For example, an individual may simply indicate a lack of interest in the introduced topic. If new knowledge is accompanied by instructional information associated with an activity (e.g., instructions on how to repair a device), and there is instruction that the activity has been performed (including by someone else), further repetition is generally unnecessary. The lack of availability of objects (e.g., tools) or services (e.g., telecommunications) necessary to carry out a repeated presentation may delay the target timing. Information presented may be superseded by the acquisition of updated material (e.g., second language practice may be terminated as a result of a canceled trip, or practice of a musical piece may be repeated if that piece is included in an updated musical piece).

[0163] Accumulation and removal of labeled dialogue databases When deploying various databases, bootstrap programming steps may be applied to grow the database of interactions to identify the degree of emotional engagement and / or emotional state, interests and / or activities influencing the emotional state, the degree of knowledge acquisition, and successful (and conversely, unsuccessful) teaching strategies. Such databases may be generalized by including interactions involving a wide range of users (e.g., different ages, learning styles, and interests), a wide range of environments (e.g., classroom, home office, individual tutoring, play), combinations of interactions (e.g., group learning settings, parent-child, grandparent-grandchild, siblings, newly acquired acquaintances), and / or a wide range of learning topics (e.g., science, languages, arts).

[0164] As the database grows, the most common interactions may be identified and maintained, while less frequent interactions may be dropped or "removed." Such a bootstrapping approach may be iterated indefinitely to improve accuracy, particularly to address a wider range of users and to progressively enhance both teaching and learning methodologies. Recent advances in the field of machine learning suggest that improvements can scale with the size of such a database, which is simply limited by the availability of computational resources.

[0165] An additional strategy that may be employed when accumulating and then removing various databases involves considering the uniqueness and / or “statistical power” of particular responses. As mentioned above, a given response may fall within a repertoire of responses in both the emotional and cognitive environments. For example, a frown may be a response that expresses both emotional irritation and a lack of understanding. The frequency of a dialogue and / or its presence in multiple databases not only reduces the statistical power to classify or distinguish responses represented by different databases, but the usefulness of observing a response in a particular database may be reduced or distorted by the ubiquitous nature of the response. To counteract this, responses appearing in multiple databases may be given greater weight to be removed (i.e., excluded) unless there are distinctive features found within such responses (e.g., deeper brow folds in a frown indicating irritation compared to a response indicating a lack of understanding). In other words, removing labeled dialogues from a database may involve factors beyond mere frequency of dialogue occurrence, such as statistical power, consideration of multivariates (discussed later), and ubiquity.

[0166] Alternatively, or in addition, if known to one or more processors performing the classification, knowledge of an individual's specific background (such as cultural grouping, dialect, customs, or age range) may assist in classifying the dialogue. For example, a “thumbs up” gesture or standing up, when considered in isolation without considering cultural and / or linguistic standards, may have the opposite meaning to lowering one's tone of voice as one approaches the end of a sentence. Thus, including “memories” (e.g., cultural, age, interest, and linguistic grouping classifications) based on past dialogues with an individual may be included as input in various classification schemes and taken into consideration when building corresponding databases.

[0167] Widespread advances at both the algorithmic and hardware acceleration levels are now being deployed using neural network-based approaches. Convolutional neural networks (CNNs) are most commonly applied to classify image data. Deep neural networks are currently at the forefront of classifying speech data. Other classification approaches include logistic regression, hyperplane analysis which may take into account nonlinear relationships between different instructions for knowledge acquisition, and support vector machines (including kernel machines which take into account nonlinear relationships, and Bayesian approaches which take into account prior probabilities), which may be used to interface with neural networks.

[0168] In exemplary embodiments, combinations of such approaches may also employ probabilistic approaches that include, in particular, synergistic relationships between modes (e.g., the utterance of "no" and the observation of a frown) when considering multiple dialogue modes simultaneously, as well as pre-assigned assumptions particularly related to the confidence of classification (e.g., the utterance of "I understand" as an indicator of high confidence in knowledge understanding compared to the detection of a smile, which may arise from a wide range of dialogue conditions). As an example, a neural network-based approach may be used to classify audio and / or video streams to identify dialogue components that may indicate the degree of knowledge acquisition. The neural network output may then be used as multivariate input to a support vector machine (SVM), a hyperplane, and / or another classifier.

[0169] In further exemplary embodiments, as a result of identifying indicators of knowledge acquisition or confusion, additional or removed instructions for knowledge acquisition may shift the previously measured time of knowledge acquisition, and consequently shift the determination of certain knowledge presentations that were initially considered to lead to knowledge comprehension. In this case, previously determined knowledge presentations that are no longer associated with learner comprehension indicators may be removed (i.e., "removed") from the database of successful knowledge presentations and / or updated knowledge presentations (i.e., instructions) that occurred before the newly determined time of knowledge acquisition was added. Steps that may lead to the insertion and / or removal of instructional elements are further illustrated in Figure 18.

[0170] The iterative expansion of the learning instruction database may result in a repeated process of reanalyzing the databases of new knowledge and instruction to keep track of updated or newly determined metrics of knowledge acquisition within each of those databases. Such reanalysis may be performed periodically, after a pre-selected number of corrections to the learning instruction have been accumulated, when substantial computational resources are available (e.g., to retrain the network), or for a period of time selected by, for example, the technology developers.

[0171] The ability to remove and add to the database during reanalysis may depend, in part, on the availability of dialogue data. If complete teaching and dialogue data (i.e., the new knowledge presented and all subsequent dialogue components) is available, the complete dialogue may simply be reprocessed against the new classification provided by the updated dialogue database. If the original dialogue data is no longer available, for example, it may only be possible to simply remove the dialogue associated with the removed classification from the dialogue database.

[0172] Figure 13 is a flowchart illustrating, with examples, several key steps in a strategy for growing a database of dialogues that include instructions for knowledge comprehension. While the flowchart in Figure 13 shows the accumulation of learner 130a's responses, a similar overall strategy may be used to evaluate learner 130a's responses by observer 131 and accumulate observer 131's responses that indicate knowledge comprehension by the learner.

[0173] In Figure 13, new knowledge may be presented to learner 130a by one or more human instructors 131, a pre-established “lesson plan” deployed using conventional coding strategies, an HIE 130a, or a combination of one or more humans and one or more machine-based sources. The learner’s response 130b to the knowledge presentation may be detected by one or more sensors within the HIE 132b. The responses of one or more observers of learner 130a (including one or more instructors 131) may also be detected by the HIE (or other HIE (not shown)) and analyzed with respect to observer indicators of the degree of knowledge acquisition.

[0174] In Figure 13, the interaction detected by one or more HIEs 132b is represented by a multimode timeline 134. For example, the upper row 134a of timeline 134 may represent video data (e.g., images of the learner's face, arms, and / or hands) collected by one or more HIE cameras, while the lower row 134b simultaneously represents audio data (e.g., phrases, words, utterances) collected using an HIE microphone. Similar timelines may be simultaneously available based on interaction data from other HIE sensors (e.g., screen touches, object movements, button presses), for example, as shown in the description of Figure 7 above. Within the overall learner response to knowledge presentation, various response or reaction components may be identified (represented as patterned or filled segments 136a, 135, 136b, 136c along timeline 134).

[0175] Based on dialogue characteristics or “templates” calculated from previous learner dialogues that demonstrate knowledge comprehension, responses (represented by filled-in segments 135) were classified in 137 as indicating knowledge comprehension by the learner. As a result, the entire learner dialogue 134 and the timestamps 138 of the dialogue components identified as indicating knowledge comprehension 135 are stored in the learner dialogue database in 139a, which may then be used to determine additional characteristics and / or templates of knowledge acquisition. In practice, when further analyzing the learner dialogue database, one or more dialogue components contained within the entire dialogue data 134 (represented by dotted segments 136a, 136b, and 136c) may be determined to be one or more additional reliable indicators of knowledge acquisition, thereby iteratively improving the indicator database 139a.

[0176] In parallel with the above sequence, if instead an indicator of knowledge confusion 135 is identified within the entire dialogue 134, the entire dialogue 134 may be stored in a dialogue database indicating the degree or level of confusion, along with the timestamp 138 of the confusion indicator and additional dialogue components 136a, 136b, 136c that may indicate confusion (139a). Alternatively or in addition, the dialogue (including the dialogue component 135 and its occurrence time 138) may be stored in the entire database containing the dialogue spectrum, and the identified dialogue 134 may be flagged as leading to a degree of knowledge acquisition ranging from substantial learner understanding to a lack of understanding.

[0177] Upon identifying understanding 135, the knowledge that generated understanding 133 (including optionally any teaching techniques and / or related material used to present the knowledge) may be stored in 139b in a database of knowledge leading to successful knowledge acquisition. In parallel, if dialogue 134 indicates a lack of understanding, one or more additional databases may store unsuccessful teaching presentations. As described in various places (see, for example, Figures 17 and 18), the database may be re-evaluated at any time, in which case dialogues may be added to or removed from the teaching database based on additions and re-evaluations regarding the classification of reliable indicators of knowledge acquisition.

[0178] More generally, in “learning about learning,” such databases may also provide primary sources of material to develop enhanced educational strategies (i.e., “teaching about teaching”) and investigate the rates of academic and personal growth at the classroom, school, state, socioeconomic category, age group, cultural population, and other levels.

[0179] Figure 14 shows a timeline of a learner interaction that includes at least one instruction for knowledge acquisition, which helps to identify or bootstrap additional metrics of knowledge acquisition. In this example, the initial database 140a of the instruction for knowledge acquisition 141 contains characteristics of instructions for successful knowledge acquisition by the learner (e.g., neural network classification, template matching), which are represented as a filled segment 141a. In the timelines 143a, 143b, 143c, and 143d shown in Figure 14 (similar to the timelines shown in Figure 13), the upper row of the timeline (e.g., 144a in timeline 143a) represents one mode of interaction, such as video recorded by an HIE camera, and the lower row of the timeline (e.g., 144b in timeline 143a) represents another mode of interaction from one or more HIE components, such as audio recorded by an HIE microphone, button presses using a mouse, keyboard typing, touchscreen input, or eye tracking directed to a selected location.

[0180] In Figure 14, each timeline 143a, 143b, 143c, and 143d is aligned for display (i.e., horizontally along the time axis) as new knowledge 142a was presented at the time indicated by the vertical dashed line 142b. These presentations of knowledge 142a may be accompanied by one or more presentation modes (e.g., audio, text, video) carried out by one or more people and / or one or more machine-based devices. The displayed timelines 143a, 143b, 143c, and 143d include at least one known metric of learner knowledge acquisition (i.e., 141a of the learner metric database 141), indicated by the filled patterns 145b, 145g, 145j, and 145o and pointed to by the filled downward arrows.

[0181] Each timeline includes additional dialogue components that may be determined as additional reliable indicators of learner knowledge acquisition, including dialogues 145a, 145c, 145d, and 145e in timeline 143a, 145f, 145h, and 145i in timeline 143b, 145k, 145l, and 145m in timeline 143c, and 145n, 145p, and 145q in timeline 143d. Dialogues may be used to identify dialogue components and / or form one or more dialogue templates within a classification scheme that identifies important and / or essential characteristics of the degree of knowledge acquisition (e.g., substantial understanding, partial understanding, lack of understanding). In Figures 14, 15, 16, 17, and 18, similar interaction characteristics are represented as similar fill patterns (e.g., dots, circles, stripes, checks) within the interaction components 145a, 145b, 145c, 145d, 145e, 145f, 145g, 145h, 145i, 145j, 145k, 145l, 145m, 145n, 145o, 145p, and 145q.

[0182] As indicated by the vertical ellipsis 146, some of the dialogues 147 shown in Figure 14 represent a subset of a much larger number of dialogues that may be stored in the dialogue database 140b. By periodically analyzing the characteristics of the stored dialogue database, which frequently occur during knowledge acquisition by the learner, characteristics may be identified and isolated to identify further dialogues that indicate the degree of knowledge acquisition. For example, in Figure 14, the dialogue segments represented by the checkered patterns 145c, 145i, 145l, and 145n are present in all the displayed dialogues 143a, 143b, 143c, and 143d. Thus, the characteristics of the checkered dialogue segment 148b are added to the previously identified characteristics represented by the solid pattern 148a in the updated database 140b of dialogues 148 used to determine the degree of knowledge acquisition.

[0183] In addition to a database of substantive knowledge comprehension indicators, the same sequence and entire bootstrapping strategy shown in Figure 14 may be used to develop a database of 1) observer indications of learner knowledge comprehension, 2) indications of learner comprehension gaps after knowledge presentation, 3) observer indications of a certain level of understanding or confusion by the learner after knowledge presentation, and / or 4) treatment of knowledge acquisition as a spectrum from understanding to confusion within each data acquisition modality. The classification scheme may be trained to output numerical and / or any number of discrete levels indicating the degree of knowledge acquisition. Starting with a small number of indicators using one or more sensors, additional (i.e., temporally adjacent) indicators may be identified and cataloged using such a bootstrapping approach.

[0184] As mentioned above, similar database deployment strategies may be applied to the accumulation of indicators of the degree of emotional engagement. In this case, the indicators of emotional engagement may be bootstrapped in response to specific topics or activities (similar to the presentation of knowledge 142a in Figure 14) and / or while monitoring individuals for spontaneous (e.g., clustered) indicators of varying degrees of emotional engagement.

[0185] Figure 15 advances the database development and classification strategy shown in Figure 14 by identifying additional dialogue components in the same temporal domain for any number of dialogue components previously identified as indicating the degree of knowledge acquisition 151a, 151b. As in Figure 14, new dialogues 153a, 153b, 153c, 53d resulting from the presentation of new knowledge 152a in time indicated by the vertical dashed line 152b (i.e., the position where the timeline is horizontally positioned so that it aligns with this criterion) may be interrogated to add to the database 150a of dialogues 151 previously identified as indicating knowledge comprehension 151a, 151b, so as to generate an updated database 150b of dialogue 157, which includes not only the previously identified indicators 158a, 158b but also the newly identified indicators of knowledge comprehension 158c, 158d.

[0186] More specifically, a database 150a containing multiple indicators of knowledge comprehension 151a, 151b may be used to identify additional indicators of knowledge comprehension, as shown in the two preceding dialogues 153a, 153b (i.e., dialogue components 155b and 155c in dialogue 153a, and dialogue components 155g and 155i in dialogue 153b) following a knowledge presentation 152b, where classifications indicating the degree of knowledge acquisition are indicated by filled-in downward arrows. The occurrence of multiple indicators of knowledge comprehension within an individual's dialogue may optionally be used to weight or score nearby (i.e., temporally coincidental) dialogue components (i.e., 155a, 155d, and 155e in dialogue 153a, and 155f and 155h in dialogue 153b) to indicate a higher confidence that knowledge comprehension is occurring.

[0187] In the timelines 153c and 153d of the two lower dialogues in Figure 15 (selected from a number of dialogues, as indicated by the vertical ellipsis 157), dialogue components identified as indicating knowledge comprehension, represented by filled-in downward arrows, are based on recently identified indicators (shown in Figure 14) represented by checkered patterns (i.e., 155k in dialogue 153c and 155n in dialogue 153d). The presence of these new dialogue components 155k and 155n may occur frequently enough after knowledge-presenting dialogues to become reliable indicators of the degree of knowledge acquisition (i.e., 155f and 155h in dialogue 153c, and 155o, 155p, and 155q in dialogue 153d).

[0188] As a result of their frequency of occurrence, dialogue components represented by downward-sloping lines (e.g., 155d, 155h, 155l, 155o) and double circles (e.g., 155a, 155f, 155j, 155p) are added to the updated database 150b of dialogue 157 (i.e., 157c and 157d are added). When reanalyzing the dialogue shown in Figure 15, the addition of the dialogue component indicating knowledge acquisition, represented by the double circle 157d, determines earlier knowledge acquisition in the three displayed dialogues (i.e., 155a in dialogue 153a, 155f in dialogue 153b, and 155j in dialogue 153c). As will be explained in more detail in the accompanying explanations in Figures 16, 17, and 18, earlier (i.e., more accurate) assessments of when knowledge acquisition occurred generate numerous consequences and build up a more accurately labeled database.

[0189] Figure 16 advances such iterative reanalysis, designed to improve the reliability and speed of identifying indicators of the degree of knowledge acquisition (in this case, a similar strategy may also be applied to a database of the degree of emotional engagement). In this scenario, the initial database 160a of knowledge acquisition indicators 161 includes two indicators represented by a filled pattern 161a and a checkered pattern 161b. During a teaching session, new knowledge 162a is presented during a timeline 163a of the resulting dialogue, shown in the upper part of Figure 16. As previously mentioned, this timeline 163a may be multi-mode, for example, with video responses represented in the upper line 164a of the timeline and audio responses in the lower line 164b. Within this timeline 163a, two dialogue components 165b, 165c are identified as indicating knowledge acquisition by the learner (indicated by filled downward arrows). Additional dialogue components 165a, 165d, 165e are not initially recognized as indicating knowledge acquisition.

[0190] Some time later (indicated by the vertical ellipsis 166), the database 160b of dialogue 167 was updated to include not only the initial indicators of knowledge acquisition represented by the fill pattern 167a and the checkered pattern 167b, but also the additional indicator represented by the mesh pattern 167c. The timeline 163b in the lower part of Figure 16 shows the result of reanalyzing the dialogue shown in the upper part of the figure 163a in light of the updated database 160b of dialogue 167. After the presentation of knowledge 162b, the indications of knowledge acquisition 168b and 168c based on indicators 167a and 167b in the indicator database are re-evaluated. However, due to the addition of the new indicator 167c of knowledge acquisition to the updated database 160b, an additional indicator 168a of knowledge acquisition is found. The remaining dialogue components 168d and 168e are still not recognized as indicating knowledge comprehension. The newly recognized instruction 168a for knowledge comprehension indicates that knowledge is understood by learners at an earlier time than determined during the original analysis.

[0191] Figure 17 shows further consequences that may be brought about by different scenarios in which the knowledge acquisition dialogue database is updated. In this scenario, the database 170a of dialogue 171 includes at least two indicators of knowledge comprehension, indicated by a filled pattern 171a and a checkered pattern 171b. The upper part of Figure 17 shows a timeline 173 of dialogues 175a, 175b, 175c, 175d, and 175e, following the initial presentation of new knowledge 172a. The timeline 173 may be multi-mode; for example, the upper row 173a of the timeline shows dialogues determined from video data, and the lower row 173b of the timeline shows dialogues determined from audio data.

[0192] In this exemplary scenario, a single dialogue 175a (double circle pattern) was detected after the first presentation of knowledge 172a, but this dialogue was not initially classified by any dialogue 171 in the dialogue database 170a correlated with understanding. Thus, a second presentation of knowledge 172b was made (e.g., by a human and / or HIE). This second presentation 172b was responded to by two dialogues that were initially determined to indicate understanding (i.e., represented by the checkered pattern 175b and the filled pattern 175c). Two additional dialogue components 175d and 175e that occurred later in the timeline 173 were not correlated with any instruction 171 in the knowledge understanding database 170a.

[0193] At some point in time (indicated by the vertical ellipsis 176), the database 170b of knowledge understanding indicators 177 was updated such that 1) dialogues represented by the solid fill pattern 177a remained in the database, 2) dialogue templates represented by the checkered pattern 177b were removed from the database, for example, as other indicators that occurred more frequently were found as the database accumulated, and 3) two indicators 177c and 177d were added to the database, for example as a result of increased detection frequency in dialogues that produce understanding.

[0194] As a result of updating the indicator database 170b, a reanalysis of the dialogue timeline 174, shown in the lower part of Figure 17, yields a positive indication of knowledge comprehension 179a immediately following the initial presentation of knowledge 178a. In addition, the dialogue pattern represented by the checkered pattern 177b is removed from the dialogue database 170b, thereby removing dialogue 179b, which was previously determined to indicate knowledge comprehension (indicated by a downward-pointing white arrow marked with an X).

[0195] In this scenario, where knowledge comprehension was not initially detected during the first presentation of new knowledge 178a, the second (e.g., possibly repeated) presentation of knowledge 178b (even if presented in a different format) may indeed result in dialogue represented by a checkered pattern, which is one of the learner's boredom and / or even frustration as a result of content repetition. The indication for knowledge comprehension, represented by a filled-in pattern 179c, continues to indicate knowledge comprehension for either the first presentation of new knowledge 178a or the second presentation 178b (which presentation it is, though uncertain, it can probably be recognized based on the elapsed time between presentations). The two final dialogue components 179d, 179e in the dialogue timeline 74 remain unassociated with any indication of knowledge comprehension based on the indication database 170b.

[0196] Following the exemplary embodiment shown in Figure 17, Figure 18 illustrates the potential consequences in updating instructions in a knowledge acquisition database for the accumulation and use of a database containing successful teachings used to enhance teaching content and techniques. The upper portion of Figure 18 shows a multi-mode 183a, 183b timeline 183 (similar to Figure 17) in which a first presentation of new knowledge 182a produced a response 185a that was not (mistakenly) recognized as indicating knowledge comprehension. As a result, a second presentation was made at 182b (e.g., using a different teaching style) resulting in two classifications (indicated by a checkered pattern 185b and a filled pattern 185c) that were initially determined to indicate knowledge comprehension. Two additional interactions 185d, 185e were not found to coincide with knowledge comprehension. As a result of these initial analyses, the first presentation of new knowledge 182a was not included in the database of successful teachings 180a. Instead, the second presentation of knowledge 182b, which was initially flagged as generating knowledge understanding 185b, 185c within timeline 183, was added to the teaching database 81b, along with other teachings 181a, 181c that had previously been determined to produce successful knowledge understanding.

[0197] As shown in the lower part of Figure 18, after some time (represented by the vertical ellipsis 186), and after the instructions in the database of understanding (as described in Figure 17) have been updated, the accumulation of successful teachings has been significantly altered by a reanalysis of the same dialogue 184. Specifically, based on the updated identification of knowledge understanding, represented by the double-circle pattern 189a, the first presentation of knowledge 188a is flagged as generating understanding. Thus, the content and method of the presentation associated with the first knowledge presentation 188a are added to the database of successful teachings 180b in 187c. In addition, the time it took for the learner to understand the new knowledge may be updated from the second knowledge presentation 188b to after the first presentation of the new knowledge 188a (presumably indicating a deeper understanding).

[0198] On the other hand, after reanalysis, the dialogue previously identified as indicating understanding represented by the checkered pattern 189b was no longer considered a reliable indicator of understanding (indicated by the white-filled arrow marked with an X). The subsequent instruction of understanding represented by the solid-fill pattern 189c remained identified, but assuming that the first presentation 188a had already generated understanding 189a, it was unclear whether this was a response to knowledge presentation 188a or 188b. The second presentation of knowledge 188b was removed from the database of successful teachings 180b (indicated by the X mark 187b). The other teaching 187a remained in the database 180b, and the other dialogues 189d and 189e were identified but not recognized as generating understanding during this particular timeline 184.

[0199] Adjusting emotional connections Skilled educators, coaches, employers, parents, and others often consciously or unconsciously manipulate the physical and emotional environment to maintain engagement and facilitate learning within the classroom, playground, workplace, home, and other settings. Numerous strategies may be employed by such influencing individuals to maintain emotional engagement. For example, leveraging the fact that most human brains are highly empathetic (beginning around age 5-6), successful instruction may relate to experiences encountered by others, as well as to the learner's own experiences. Oralization is an effective strategy for evoking such emotions and associated learning. Dialogue-based oralization is a particularly powerful technique due to its inclusion of active participation by the listener. The introduction of tasks (especially during instruction), including the possibility of failure, fosters a sense of accomplishment that leads to great satisfaction when such tasks are overcome.

[0200] Skilled presenters may include visual and / or auditory “surprises” (e.g., an unexpected image of a sunset or a sound louder than usual) and other emotional cues (e.g., music) to keep the audience engaged. A variation of this strategy, cognitive imbalance, involves presenting material that does not fit into the normal experience and understanding of the world (e.g., an image of a cow quacking like a duck). To maintain an emotional connection, presenters may directly involve the audience or even simply include the possibility of audience participation. Including praise, happiness, and other forms of emotional support during learning is a very powerful technique that not only reinforces the concept at hand but also enhances the overall interest and / or excitement in the learning activity or area.

[0201] In further embodiments, machine-based methods for interacting with and maintaining engagement with an individual may employ similar strategies, either directly through the HIE or by "signaling" or suggesting a way for a human to interact. Direct interaction and / or signaling via the HIE may take into account the timing judgments described above (e.g., not overwhelming). The machine-aware interaction may also involve one or more humans interacting with one or more virtual characters to put such strategies into practice and / or entertain.

[0202] As a further consideration, especially when interacting with children, the fact that "play" is almost impossible to completely suppress suggests that it plays a significant role in the behavioral and neurological development of many animal species. Play is observed in mammals, birds, other vertebrates, and even invertebrates. Play can continue into adulthood, and hobbies and interests often reflect the environment and interests of childhood. Play is generally defined based on three criteria: 1) it resembles serious behavior (e.g., hunting, escape), but may be exaggerated or modified; 2) it does not involve an imminent survival purpose and is found to be enjoyable; and 3) it is undertaken spontaneously in a stress-free environment. In some cases, play is seen as practice-based learning of "real-life" situations. In many cultures, play is considered a major contribution to the formation of social connections. In other cultures where social independence may be more strongly emphasized, play generally relies heavily on objects (e.g., dolls, building blocks, and other toys).

[0203] In exemplary embodiments, real-time, machine-based monitoring of emotional states during play may enable strategies, monitor enhanced emotional and cognitive environments, and / or provide numerous benefits beyond play without being put into practice. These strategies and benefits include: 1. Children may be allowed to play alone or in a given environment (e.g., with siblings and friends) undisturbed until there is an indication (via machine-based detection) of one or more predetermined emotional states, including stress, frustration, or boredom. 2. Monitors or caregivers (e.g., parents, teachers, guardians) may avoid repeatedly asking questions and / or interrupting play activities to assess emotional state. 3. Machine-based monitoring may allow one or more caregivers to perform other tasks simultaneously (for example, during playtime with one or more children). Such “time-sharing” is further described in U.S. Patent No. 10,915,814 filed June 15, 2020, which may indicate a lack of adequate caregiver attention or no indication that such time-sharing is taking place. 4. Machine-based monitoring, combined with a predetermined action, may supervise and / or guide future play activities in a manner that maintains a high level of interest. This may include deploying so-called “conditional actions,” as further described in U.S. Patent No. 10,915,814 filed October 13, 2020. Such guidance may be restrictive (e.g., denying access to activities that may be age-inappropriate) or inclusive (e.g., machine-based suggestion of the involvement of others in certain play activities). 5. The ideal time for notification and subsequent involvement of additional supporters and / or other playmates may be based on machine-based monitoring of play activities and / or emotional states (e.g., approaching completion of a prescribed activity). 6. Play activities may be categorized and cataloged to enable machine-based “cues” for future activities of interest, as well as to make individuals aware of a range of topics that may not yet be explored and / or may be avoided. 7. Any need or “desire” for a physical object may be cataloged and communicated to supporters, others involved in the play activity, and / or others involved in the individual’s well-being (for example, particularly as a suggestion during the time when gifts are exchanged). 8. The amount of time and interest spent in specific play activities (e.g., online games, use of specific toys) or with specific playmates may be cataloged, monitored, and (if necessary) controlled. 9. If the gameplay involves a level of challenge that may be adjusted (e.g., playing a game, building a virtual object), the difficulty may be modified to allow a predetermined range of success and / or failure in completing those challenges.

[0204] In further exemplary embodiments, during an activity shared between an individual and one or more supporters, one or more supporters may be supported during the activity by exclusively providing machine-derived information, prompts, and / or HIE behaviors that help facilitate emotional and / or cognitive engagement. The learner or other individual may be completely unaware or only partially aware that such machine prompts are being provided. To such one or more individuals, the human supporter may appear to possess a “mighty power” if the knowledge is augmented or, in a pleasing way.

[0205] Examples of shared activities in which such enhanced dialogue may be employed include reading books, listening to stories, building or restoring objects, visiting world facilities (e.g., museums, historical sites) that may include art, architecture, endemic plants, or rare animals, or virtually viewing a region. During such shared activities, supporters may provide (e.g., through HIE) expanding material in the form of images, audio clips, historical perspectives, cultural relevance, contextual relationships, trivia, scientific foundations, or mathematical perspectives. Such exclusive prompts from one or more supporters of an individual may also be accompanied by tracking the timing of the activity, such as being notified of deadlines for completing tasks or the predicted times of events (e.g., sunset, television programs). Implementing such enhanced or expanded dialogue can rapidly increase engagement. Over time, expanded dialogue can help develop predictions of increased levels of interest and / or relevance.

[0206] In addition to controlling the pace and complexity of presenting new material, the HIE may also control the timing of breaks in the dialogue, the introduction of which may take into account the material being presented and / or other elements of the dialogue (e.g., the end of topic presentation, detection of confusion or frustration). In additional exemplary embodiments, both the start and duration of such breaks may be controlled (e.g., directly by the HIE and / or via cues from one or more people). The duration may be influenced by the perceived type of activity being carried out (e.g., preparing drinks, eating, discussion, exercise).

[0207] As explained more thoroughly in the "Background Technology" section, the same neurotransmitters and hormones involved in regulating important emotional states are produced and regulated during physical activity, including exercise. Performing physical activity of sufficient intensity and / or duration to significantly impact the levels of these neurotransmitters and hormones produces measurable effects on blood flow and the physiological function of excitatory cells (i.e., nerves, muscles). During secondary activities (e.g., meetings, screen viewing), both cognitive and emotional states may benefit from taking short "increased cognition breaks" (ICBs). ICBs may include, for example, a one-minute break every 30 minutes, and / or physical activity in the middle (e.g., push-ups, sit-ups, jumping jacks, squats).

[0208] ICBs may be yet another way to enhance engagement, cognitive performance, and / or creative productivity as a whole. As one component that helps expand the entire emotional and cognitive environment, HIEs may facilitate, initiate, and / or track ICBs. In addition to the elapsed time since the previous ICB, the initiation of an ICB may take into account the emotional state of one or more individuals, natural breaks within cognitive activity (e.g., topic changes, switching presenters), the expected duration of secondary activities, and the time of day. Targets of physical activities that may be monitored by HIEs include the duration of the activity, the physical characteristics of those activities (e.g., number of steps, push-ups, sit-ups, jumping jacks, squats), and / or physiological targets (e.g., increase in heart rate or respiratory rate). Scheduled breaks may take into account individual or group preferences and / or the individual or group's physiological ability to safely perform the activity (e.g., due to age or physical limitations).

[0209] The above-mentioned disclosure of exemplary embodiments has been presented for illustrative and explanatory purposes. The description is not intended to be exclusive or to limit the invention to the exact form disclosed. Numerous variations and modifications of the embodiments described herein will be obvious to those skilled in the art in light of the above-mentioned disclosure. It will be understood that various components and features described using specific embodiments may be added, deleted, and / or replaced with those of other embodiments, depending on the intended use of the embodiment.

[0210] Furthermore, in the description of representative embodiments, this specification may present methods and / or processes as a specific sequence of steps. However, to the extent that the method or process does not depend on the specific order of steps described herein, the method or process should not be limited to the specific sequence of steps described herein. As those skilled in the art will recognize, other sequences of steps may be possible. Therefore, the specific order of steps described herein should not be construed as a limitation on the claims.

[0211] While the present invention is permissible in various forms and alternative configurations, specific examples are shown in the drawings and described in detail herein. The present invention should not be limited to any particular form or method disclosed; rather, it should be understood that the present invention encompasses all variations, equivalents, and alternatives within the scope of the appended claims.

Claims

1. A system for an observer for an individual to achieve a target degree of emotional engagement by the individual, comprising: an observer electronic device configured to be placed near a human observer, the observer electronic device including an observer processor and one or more observer output devices operably coupled to the observer processor; a personal electronic device configured to be placed near an individual, the personal electronic device including a personal processor, one or more personal output devices operably connected to the personal processor, and one or more sensors operably connected to the personal processor; said system comprising: using the one or more personal output devices, the personal electronic device and / or an observer presenting knowledge to the individual that the individual may not know; the one or more sensors acquiring interaction data related to the presented knowledge; one or more of the individual processor, the observer processor, and the one or more remote processors classify within the interaction data one or more indications of the individual's degree of emotional engagement with the knowledge being presented, the individual's degree of emotional engagement being one or more of the individual being apathetic, disinterested, distracted, bored, annoyed, and overwhelmed; the individual processor, the observer processor, and one or more of the one or more remote processors select activities intended to direct the individual's level of emotional engagement toward a target level of emotional engagement, the target level of emotional engagement being one or more of the individual being enthusiastic, interested, motivated, enthusiastic, excited, or stimulated; the one or more observer output devices represent activities to an observer and prompt the observer to initiate actions with respect to the individual to move the individual's level of emotional engagement towards a target level of emotional engagement; The system is configured as follows:

2. A system as described in claim 1, wherein artificial intelligence is instantiated within one or both of the personal processor and the observer processor.

3. The system of claim 2, wherein the artificial intelligence includes an artificial intelligence personality that is familiar to the individual and / or an observer.

4. The system of claim 1, wherein classifying the one or more indications of the individual's degree of emotional engagement includes identifying within the interaction data one or more of the following: one or more words, phrases, interjections, vocalizations, gestures, smiles, hand movements, finger configurations, written symbols, head movements, facial muscle movements, forehead muscle movements, eye movements, changes in pupil size, blinking, pressing an electronic device button, touching an electronic device screen, shaking the individual's electronic device, movement of one or more objects sensed by the electronic device, gaze direction, changes in tone of voice, changes in voice volume, changes in speaking rate, changes in reaction time, lack of response to a comment, lack of response to a question, and reactions unrelated to a question.

5. The system of claim 1, wherein one or more additional interactions labeled with the degree of emotional engagement of the individual within the interaction data are added to a database of emotional interactions by one or more of the individual processor and the one or more remote processors.

6. The system of claim 5, wherein one or more degree of emotional engagement human dialogue classifier templates are generated by identifying one or more of the labeled emotional dialogues and combinations of labeled emotional dialogues that occur with a high frequency within the database of labeled emotional dialogues.

7. The system of claim 5, wherein the database of labeled emotional interactions is used to train one or more neural networks configured to classify the degree of emotional engagement of human interactions.

8. The system of claim 1, further configured to transmit one or more of the knowledge, the time the knowledge was expressed, and the degree of emotional engagement of the individual from one or more of the individual processor, the observer processor, and the one or more remote processors to one or more monitoring processors configured to communicate with the observer and one or more monitoring humans other than the individual.

9. The system described in claim 1, wherein the one or more observer output devices include an observer display, and activity is represented on the observer display by characters presented on the observer display.

10. The system of claim 1, wherein the activity is represented as one or more of text, video, and audio on one or more observer output devices.

11. A method for an observer for an individual to achieve a target degree of emotional involvement by the individual, comprising: providing an observer electronic device in proximity to a human observer, the observer electronic device including an observer processor and one or more observer output devices operably coupled to the observer processor; providing a personal electronic device proximate to the individual, the personal electronic device including a personal processor, one or more personal output devices operably coupled to said personal processor, and one or more sensors operably coupled to said personal processor; presenting to the individual, by said personal electronic device and / or an observer, knowledge that the individual may not know; obtaining interaction data from one or more sensors related to the knowledge being presented from an interaction between the personal electronic device and the individual; classifying, by one or more of the personal processor and the one or more remote processors, within the interaction data, one or more indications of an individual's degree of engagement with the presented knowledge, wherein the individual's degree of emotional engagement is one or more of the individual being apathetic, disinterested, distracted, bored, annoyed, and overwhelmed; selecting, by one or more of the individual processor, the observer processor, and the one or more remote processors, an activity intended to direct an individual's level of emotional engagement toward a target level of emotional engagement, the target level of emotional engagement being one or more of an individual being enthusiastic, interested, motivated, enthusiastic, excited, and stimulated; displaying activities to an observer via the one or more observer output devices to prompt the observer to initiate actions with respect to the individual to move the individual's emotional engagement level toward a target emotional engagement level; A method comprising:

12. The method of claim 11, wherein artificial intelligence is instantiated within one or both of the individual processor and the observer processor.

13. The method of claim 12, wherein the artificial intelligence includes an artificial intelligence personality that is familiar to the individual and / or an observer.

14. The method of claim 11, wherein classifying the one or more indications of the individual's degree of emotional engagement comprises identifying within the interaction data one or more of the following: one or more words, phrases, interjections, vocalizations, gestures, smiles, hand movements, finger configurations, written symbols, head movements, facial muscle movements, forehead muscle movements, eye movements, changes in pupil size, blinking, pressing an electronic device button, touching an electronic device screen, shaking the individual's electronic device, movement of one or more objects sensed by the electronic device, direction of gaze, changes in tone of voice, changes in voice volume, changes in speaking rate, changes in reaction time, lack of response to a comment, lack of response to a question, and reactions unrelated to a question.

15. The method of claim 11, wherein one or more additional interactions labeled with a degree of emotional engagement of an individual within the interaction data are added to a labeled emotional interaction database by one or more of the individual processor and the one or more remote processors.

16. The method of claim 15, wherein one or more degree of emotional engagement human dialogue classifier templates are generated by identifying one or more of the labeled emotional dialogues and combinations of labeled emotional dialogues that occur with a high frequency within the database of labeled emotional dialogues.

17. The method of claim 15, wherein the labeled emotional dialogue database is used to train one or more neural networks configured to classify the degree of emotional engagement in human dialogue.

18. The method of claim 11, further comprising transmitting one or more of the knowledge, the time the knowledge was expressed, and the degree of emotional engagement of the individual from one or more of the individual processor, the observer processor, and one or more remote processors to one or more monitoring processors configured to communicate with the observer and one or more monitors other than the individual.

19. The method of claim 11, wherein the observer interacts with the individual via a video communications link.

20. The method of claim 11, wherein one or more observer output devices represent to an observer the time to perform one of the activities and the repetition of the activity.

21. The method of claim 11, wherein one or more observer output devices include an observer display, and activity is represented on the observer display by characters presented on the observer display.

22. The method of claim 21, wherein the characters are presented only on an observer display and not on one or more personal output devices.

23. The method of claim 11, wherein the activity is represented as one or more of text, video, and audio on one or more observer output devices.

24. The method described in claim 23, wherein one or more of text, video, and audio are rendered only on the observer output device and not on the one or more personal output devices.

25. A method for an individual observer to achieve a target degree of emotional involvement by the individual, comprising: providing an observer electronic device in proximity to a human observer, the observer electronic device including an observer processor, one or more observer output devices operatively coupled to the observer processor, and one or more sensors operatively coupled to the observer processor; providing a personal electronic device proximate to the individual, the personal electronic device including a personal processor and one or more personal output devices operably coupled to the personal processor; presenting knowledge to the individual that the individual may not know by one or more personal output devices and / or an observer; obtaining observer interaction data from one or more sensors of the observer electronic device from an interaction between an observer and the observer electronic device; classifying, by one or more of the observer processor and one or more remote processors, within the observer interaction data, one or more indications indicative of a degree of emotional engagement by an individual, the degree of emotional engagement by an individual being one or more of the following: the individual is disinterested, apathetic, distracted, bored, annoyed, and overwhelmed; selecting, by one or more of the observer processor and one or more remote processors, an activity intended to direct an individual's level of emotional engagement toward a target level of emotional engagement, the target level of emotional engagement being one or more of the individual being enthusiastic, interested, motivated, enthusiastic, excited, and stimulated; displaying activities to an observer via the one or more observer output devices to prompt the observer to initiate actions with respect to the individual to move the individual's emotional engagement level toward a target emotional engagement level; A method comprising:

26. A method for a human observer to achieve a target degree of emotional involvement by an individual, comprising: providing a personal electronic device proximate to the individual, the personal electronic device including a personal processor, one or more personal output devices operably coupled to the personal processor, and one or more personal sensors operably coupled to the personal processor; providing an observer electronic device in proximity to a human observer, the observer electronic device including an observer processor and one or more observer output devices operably coupled to the observer processor; presenting to the individual, at said one or more personal output devices, knowledge that the individual may not know; obtaining, from the one or more personal sensors, interaction data from an interaction between an individual and the personal electronic device related to presented knowledge; presenting interaction data on the one or more observer output devices; indicating, by a human observer using the one or more observer sensors, a degree of observed emotional engagement by an individual; selecting, by one or more of the personal processor, the observer processor, and the one or more remote processors, an activity intended to direct the observed degree of emotional engagement toward a target degree of emotional engagement; presenting a human interaction entity to the one or more personal output devices to initiate an activity with the individual; A method comprising:

27. A method for a human observer for an individual to achieve a target degree of emotional involvement by the individual, comprising: providing a personal electronic device proximate to the individual, the personal electronic device including a personal processor, one or more personal output devices operably coupled to said personal processor, and one or more personal sensors operably coupled to said personal processor; providing an observer electronic device in proximity to a human observer, the observer electronic device including an observer processor, one or more observer output devices operatively connected to said observer processor, and one or more observer sensors operatively connected to said observer processor; presenting to the individual, at said one or more personal output devices, knowledge that the individual may not know; obtaining, from the one or more personal sensors, interaction data from an interaction between an individual and the personal electronic device related to presented knowledge; presenting interaction data on the one or more observer output devices; indicating, by a human observer using the one or more observer sensors, an activity intended to orient an observed degree of emotional engagement by an individual toward a target degree of emotional engagement; presenting a human interaction entity to the one or more personal output devices to initiate an activity with the individual; A method comprising:

28. A method for a human observer for an individual to achieve a target degree of emotional involvement by the individual, comprising: providing a personal electronic device proximate to the individual, the personal electronic device including a personal processor, one or more personal output devices operably coupled to said personal processor, and one or more personal sensors operably coupled to said personal processor; providing an observer electronic device in proximity to a human observer, the observer electronic device including an observer processor, one or more observer output devices operatively connected to said observer processor, and one or more observer sensors operatively connected to said observer processor; presenting to the individual, at said one or more personal output devices, knowledge that the individual may not know; obtaining, from the one or more personal sensors, interaction data from an interaction between an individual and the personal electronic device related to presented knowledge; classifying within the interaction data, by one or more of the individual processor, the observer processor, and one or more remote processors, a degree of emotional engagement by the individual with the presented knowledge; presenting a degree of individual emotional engagement on the one or more observer output devices; indicating, by a human observer using the one or more observer sensors, an activity intended to orient an individual's emotional engagement level toward a target emotional engagement level; presenting a human interaction entity to the one or more personal output devices to initiate an activity with the individual; A method comprising:

29. 1. A method for monitoring human learning using artificial intelligence, comprising: providing an electronic device in proximity to a first human, the electronic device including a dialogue processor instantiated with the artificial intelligence, one or more dialogue output devices operably coupled to the dialogue processor, and one or more sensors operably coupled to the dialogue processor; expressing to the first human, by one or both of the electronic device and a second human, knowledge that the first human would not know; acquiring interaction data from the one or more sensors from an interaction between the electronic device and the first human; classifying within the interaction data, by one or more of the interaction processor and one or more remote processors, one or more indications of a degree of emotional engagement by the first human; classifying within the interaction data, by one or more of the interaction processor and the one or more remote processors, one or more indications of a degree of knowledge acquisition by the first human; A method comprising:

30. 1. A system for monitoring the emotional state of a human being during learning, comprising: an electronic device configured to be placed in proximity to a first human, the electronic device comprising a dialogue processor, one or more dialogue output devices operably coupled to the dialogue processor, and one or more sensors operably coupled to the dialogue processor; one or both of the electronic device and a second human express, using the one or more interactive output devices, to the first human knowledge that the first human would not know; the one or more sensors acquiring interaction data; one or more of the interaction processor and one or more remote processors classifying, within the interaction data, one or more indications of a degree of emotional engagement by the first human; and One or more of the dialogue processor and one or more remote processors classify within the dialogue data one or more indications of a degree of knowledge acquisition by the first human. A system configured as follows.

31. 1. A method for monitoring an emotional state of a human being during learning, comprising: providing an electronic device in proximity to a first human, the electronic device including a dialogue processor, one or more dialogue output devices operably coupled to the dialogue processor, and one or more sensors operably coupled to the dialogue processor; expressing to the first human, by one or both of the one or more interactive output devices and a second human, knowledge that the first human would not know; acquiring interaction data from the one or more sensors; classifying within the interaction data, by one or more of the interaction processor and one or more remote processors, one or more indications of a degree of emotional engagement by the first human; classifying within the interaction data, by one or more of the interaction processor and the one or more remote processors, one or more indications of a degree of knowledge acquisition by the first human; A method comprising: