System and method for adjusting and optimizing human-computer interaction using multimodal user feedback

Abstract

A system and method for human-computer interaction. An adaptive behavior control system of a human-computer interaction system controls an interaction subsystem to perform a plurality of actions for a first action type according to a computer behavior strategy, each action being a different alternative action for the action type. The adaptive behavior control system detects a human response of an interaction participant to the performance of each action of the first action type based on data received from a human response detection subsystem. The adaptive behavior control system stores information indicating each detected human response in association with information identifying an associated action. In the event that the stored information indicating the detected human response for the first action type satisfies an update condition, the adaptive behavior control system updates the computer behavior strategy for the first action type.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 62 / 715,665, filed on August 7, 2018, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to human-computer interaction systems, and more particularly to novel and useful systems and methods for using user feedback in human-computer interaction systems. Background Art

[0004] Typical human-computer interaction systems receive user feedback via input devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figures 1A to 1D is a schematic representation of a system according to various embodiments;

[0006] Figure 2 is a simplified diagram depicting the system architecture of a system according to various embodiments;

[0007] FIG. 3A to FIG. 3E is a representation of exemplary strategies according to various embodiments;

[0008] Figures 4 to 7 is a representation of a method according to various embodiments; and

[0009] Figure 8 is a representation of a multi-party interaction according to various embodiments. DETAILED DESCRIPTION

[0010] The following description of the embodiments is not intended to limit the disclosure to these embodiments, but rather to enable any person skilled in the art to make and use the embodiments disclosed herein.

[0011] Overview

[0012] Embodiments herein include systems and methods using an adaptive behavior control system (of a human-computer interaction system) that uses a computer-readable computer behavior policy to control at least one of an audio output system, a display device, and a robot. The adaptive behavior control system is configured to update the computer behavior policy in response to information indicating a detected human reaction.

[0013] In some embodiments, the computer behavior policy specifies multiple actions for a specific action type and a weight value for each action. For example, for the "greeting" action type, the policy may include three different greeting action options, each with a corresponding weight value. In some embodiments, each weight value corresponds to a frequency of use. In some embodiments, each weight value is a percentage corresponding to the frequency at which a specific action should be performed, and the weights for all actions of the action type total 100%. In some embodiments, each weight value is used to generate a weighted random sampling, and the action of a specific action type is selected based on the weighted random sampling.

[0014] An initial computer behavior strategy is generated, and an action for an action type is executed, and the adaptive behavior control system stores information indicating a human reaction detected in response to the execution of the action in association with information identifying the action. In this way, the adaptive behavior control system creates a log recording all reactions to an action.

[0015] In the case where the stored information indicating the detected human response to an action satisfies the update condition, the adaptive behavior control system updates the weight values ​​of the action and other actions of the same action type in the computer behavior strategy. In some embodiments, in the case where the stored information indicating the detected human response to a first action indicates that the positive user feedback is below a threshold, the adaptive behavior control system reduces the weight value of the first action. In some embodiments, in the case where the stored information indicating the detected human response to the first action indicates that the positive user feedback is above a threshold, the adaptive behavior control system increases the weight value of the first action. In this way, the adaptive behavior control system updates the weight values ​​of the actions in the computer behavior strategy in response to the control of the human-computer interaction system according to the computer behavior strategy to cause positive user feedback from the human-computer interaction participants.

[0016] In some embodiments, the human-machine interaction system is a robot that includes a mechanical body component (e.g., Figure 1D 104d), movable mechanical facial components (e.g., Figure 1D 103d) and at least one movable mechanical attachment (e.g., Figure 1D105d), and the adaptive behavior control system is configured to control the facial component according to the computer behavior strategy. In some embodiments, the control of the facial component according to the computer behavior strategy includes: controlling the facial component to orient the facial component in the direction of the detected face of the interactive participant by using at least one of the audio data captured by the audio capture system and the image data captured by the video capture system. In some embodiments, the action of a certain action type in the computer behavior strategy includes controlling the facial component to orient the facial component in the direction of the detected face of the interactive participant. In some embodiments, the action of a certain action type includes a conversation action. In some embodiments, the action of a certain action type in the computer behavior strategy includes controlling the robot to perform a conversation action. In some embodiments, the action of a certain action type in the computer behavior strategy includes controlling the robot to move. In some embodiments, the action of a certain action type in the computer behavior strategy includes controlling the robot to perform a movement corresponding to a posture.

[0017] In some embodiments, the adaptive behavior control system is configured to detect the presence and corresponding positions of interaction participants and generate a three-dimensional world model of the detected positions of the detected interaction participants. In some embodiments, during the initiation of an interaction with a previously detected interaction participant, the adaptive behavior control system controls the facial component to orient the facial component in the direction of the detected interaction participant by using the previously detected positions included in the world model. In some embodiments, the adaptive behavior control system is configured to generate a three-dimensional world model of all detected faces, and wherein, during the initiation of an interaction with a participant associated with a previously detected face, the adaptive behavior control system controls the facial component to orient the facial component in the direction of the detected face by using the previously detected directions included in the world model.

[0018] In some embodiments, the weight values ​​in the computer behavior policy are updated based on information received from at least one external human-computer interaction system.

[0019] system

[0020] Figure 1A

[0021] In some embodiments, the human-computer interaction system 100a includes an adaptive behavior control system 110, an interaction subsystem 112, and a human reaction detection subsystem 111. Figure 1A In some embodiments, the adaptive behavior control system 110 is a computing system such as Figure 2 In some embodiments, the adaptive behavior control system 110 is one of an FPGA, an ASIC, and a system on a chip (SoC).

[0022] Interaction Subsystem

[0023] In some embodiments, the interaction subsystem 112 includes an audio output subsystem (e.g., Figure 1B 130 Figure 1D In some embodiments, the interactive subsystem 112 includes a video display subsystem (e.g., Figure 1C 102c). In some embodiments, the interaction subsystem 112 includes a mechanical robotic subsystem (e.g., head assembly 103d, attachment 105d of FIG. 1). In some embodiments, the interaction subsystem 112 includes a lighting subsystem. In some embodiments, the interaction subsystem 112 includes an LED (light emitting diode) ring.

[0024] In some embodiments, the audio output subsystem includes at least one speaker (e.g., Figure 1B 130 Figure 1D In some embodiments, the audio output subsystem includes at least one audio output device. In some embodiments, the video display subsystem includes at least one display device (e.g., Figure 1C In some embodiments, the mechanical robot subsystem includes at least one robot body component (e.g., Figure 1D In some embodiments, the mechanical robot subsystem includes at least one robot body head component (e.g., Figure 1D In some embodiments, the mechanical robotic subsystem includes at least one robotic attachment (e.g., Figure 1D 105d). In some embodiments, the mechanical robotic subsystem includes at least one robotic body head assembly mechanically attached to the robotic body assembly. In some embodiments, the facial assembly includes a motorized assembly configured to rotate the head assembly relative to the body assembly with at least one degree of freedom about a first axis. In some embodiments, the facial assembly includes a motorized assembly configured to rotate the head assembly relative to the body assembly with at least two degrees of freedom about a first axis and about a second axis.

[0025] In some embodiments, the interaction subsystem 112 includes a lighting subsystem that includes an array of lights, and the lighting subsystem is configured to individually control each light of the lighting subsystem. In some embodiments, the interaction subsystem 112 includes an LED ring that includes an array of lights, and the LED ring is configured to individually control each light of the LED ring.

[0026] Detection subsystem

[0027] In some embodiments, the human response detection subsystem 111 includes an audio capture subsystem (e.g., Figure 1B120 Figure 1D In some embodiments, the human response detection subsystem 111 includes a video capture subsystem (e.g., Figure 1C 101c, Figure 1D In some embodiments, the human reaction detection subsystem 111 includes a thermal detection subsystem.

[0028] In some embodiments, the audio capture subsystem includes at least one audio capture device. In some embodiments, the audio capture subsystem includes at least one microphone. In some embodiments, the audio capture subsystem includes at least one microphone array (e.g., Figure 2 B's 120).

[0029] In some embodiments, the video capture subsystem includes at least one video capture device. In some embodiments, the video capture subsystem includes at least one camera (e.g., Figure 1C 101c, Figure 1D 101d). In some embodiments, the video capture subsystem includes at least one camera array.

[0030] In some embodiments, the human reaction detection subsystem 111 includes a thermal detection subsystem configured to detect body heat of a human. In some embodiments, the human reaction detection subsystem 111 includes a thermal detection subsystem configured to detect body temperature of a human.

[0031] In some embodiments, human reaction detection subsystem 111 includes a touch sensor.

[0032] In some embodiments, the human reaction detection subsystem 111 includes a piezoelectric pressure sensor.

[0033] In some embodiments, human reaction detection subsystem 111 includes a capacitive touch sensor.

[0034] In some embodiments, the human reaction detection subsystem 111 includes a resistive touch sensor.

[0035] In some embodiments, the human response detection subsystem 111 includes a blood pressure sensor.

[0036] In some embodiments, human reaction detection subsystem 111 includes a heart rate sensor.

[0037] In some embodiments, the human response detector 111 includes a biometric sensor.

[0038] In some embodiments, the human reaction detection subsystem 111 includes an interface (e.g., Bluetooth, NFC, WiFi, etc.) configured to receive data from a remote sensor (e.g., a wearable device such as, for example, a FitBit and an Apple Watch).

[0039] Audio System

[0040] Figure 1B 1 is a schematic representation of an audio-based human-computer interaction system according to various embodiments. In some embodiments, the human-computer interaction system 100b includes an adaptive behavior control system 110, an interaction subsystem including an audio output controller 131 and a speaker 130, and a human reaction detection subsystem including an audio input processor 121 and a microphone 120, such as Figure 1B In some embodiments, the human-computer interaction system 100b is a smart speaker. In some embodiments, the human-computer interaction system 100b is an Amazon Echo smart speaker including an adaptive behavior control system according to an embodiment disclosed herein.

[0041] In some embodiments, the audio input processor 121 is configured to process the audio data generated by each microphone 120 and provide the processed audio data to the adaptive behavior control system 110. In some embodiments, the adaptive behavior control system 110 includes the audio input processor 121. In some embodiments, the audio input processor 121 is configured to process the audio data generated by each microphone 120 by combining the audio data of each microphone 120 into a combined audio data stream. In some embodiments, the audio input processor 121 is configured to process the audio data generated by each microphone 120 by selecting at least one audio data stream of the microphone 120, combining each selected data stream, and the combined audio data stream of the selected data stream is sent to the adaptive behavior control system 110.

[0042] In some embodiments, direction information is associated with each microphone 120, and the audio input processor 121 is configured to determine the direction of the captured audio based on the direction information associated with the microphone 120 with the highest captured audio volume. In this way, the adaptive behavior control system 110 can use the audio input processor 121 in conjunction with the microphones 120 to determine the location of the interactive participant based on the captured audio.

[0043] In some embodiments, the audio output controller 131 is configured to control the speaker 130 according to data provided to the audio output controller 131 by the adaptive behavior control system 110. In some embodiments, the audio output controller 131 is configured to select one of the plurality of speakers 130 based on information provided by the adaptive behavior control system 110, and control the selected speaker 130 to emit audio according to the data provided to the audio output controller 131 by the adaptive behavior control system 110.

[0044] In some embodiments, direction information is associated with each speaker, and the adaptive behavior control system 110 controls the audio output controller 131 to emit audio by using the speaker 130 that matches the direction information provided by the adaptive behavior control system. In this way, the adaptive behavior control system 110 can use the audio output controller 131 in conjunction with the speaker 130 to emit audio in the direction of a specific interactive participant.

[0045] In some embodiments, the adaptive behavior control system 110 includes an audio input processor 121. In some embodiments, the adaptive behavior control system 110 includes an audio output controller 131.

[0046] Audio and video systems

[0047] Figure 1C 1 is a schematic representation of an audio and video based human-computer interaction system according to various embodiments. In some embodiments, the human-computer interaction system 100c includes an adaptive behavior control system (e.g., 110), an interaction subsystem (not shown) including a display 102c and a speaker, and a human reaction detection subsystem including a camera 101c and a microphone (not shown). In some embodiments, the human-computer interaction system 100c is a smart phone. In some embodiments, the human-computer interaction system 100c is a computer. In some embodiments, the human-computer interaction system 100c is a mobile device.

[0048] robot

[0049] Figure 1D is a schematic representation of a robot human-machine interaction system according to various embodiments.

[0050] In some embodiments, the human-computer interaction system 100d includes an adaptive behavior control system (e.g., 110), an interaction subsystem including a head assembly 103d and at least one accessory 105d, and a human reaction detection subsystem including a camera 101d of the head assembly 103d and a microphone 106d of the head assembly 103d. In some embodiments, the head assembly 103d is mechanically attached to the body assembly 104d. In some embodiments, the head assembly 103d is mechanically attached to the body assembly 104d. In some embodiments, the human-computer interaction system 100d includes a motor assembly 163 configured to rotate the head assembly 103d around a vertical axis 160d. In some embodiments, the human-computer interaction system 100d includes a motor assembly 162 configured to move the head assembly 103d around a horizontal axis 161d.

[0051] In some embodiments, the head assembly 103d includes a speaker 102d.

[0052] In some embodiments, head assembly 103d includes an adaptive behavior control system (eg, 110). In some embodiments, body assembly 104d includes an adaptive behavior control system (eg, 110).

[0053] In some embodiments, the robotic interaction system 100d includes an audio input processor 121, which is communicatively coupled to an adaptive behavior control system of the system 100d, and the adaptive behavior control system of the system 100d uses the audio input processor 121 to identify the direction of the interaction participant, as disclosed herein.

[0054] In some embodiments, the robotic interaction system 100d includes an audio input processor 121, which is communicatively coupled to an adaptive behavior control system of the system 100d, the audio input processor 121 is communicatively coupled to a microphone 106d, and the adaptive behavior control system of the system 100d uses the audio input processor 121 to identify the direction of the interactive participant based on the volume of the audio captured by the microphone 106d and the current position of the motor 163.

[0055] In some embodiments, the robot interaction system 100d includes an audio input processor 121, the audio input processor 121 is communicatively coupled to the adaptive behavior control system of the system 100d, the audio input processor 121 is communicatively coupled to a microphone array including a plurality of microphones arranged in different directions, and the adaptive behavior control system of the system 100d uses the audio input processor 121 to identify the direction of the interactive participant based on the volume of the audio captured by the microphones in the microphone array. In some embodiments, the body component 104d includes a microphone array. In some embodiments, the head component 103d includes a microphone array, the microphone array rotates with the head component 103d through a motor 163, and the adaptive behavior control system of the system 100d uses the rotational position information of the motor 163 to identify the direction of the interactive participant. In some embodiments, the head component 103d includes a microphone array, the microphone array rotates with the head component 103d through a motor 162, and the adaptive behavior control system of the system 100d uses the rotational position information of the motor 162 to identify the direction of the interactive participant. In some embodiments, the robot system 100d includes a GPS receiver, and the adaptive behavior control system of the system 100d uses the information of the GPS receiver to identify the direction of the interactive participant. In some embodiments, the robot system 100d includes an inertial measurement unit (IMU), and the adaptive behavior control system of the system 100d uses information from the IMU to identify the direction of the interactive participant. In some embodiments, the robot system 100d includes an accelerometer, and the adaptive behavior control system of the system 100d uses information from the accelerometer to identify the direction of the interactive participant.

[0056] Computer Behavior Policy

[0057] In some embodiments, an adaptive behavior control system (eg, 110) stores computer readable data of a computer behavior policy.

[0058] In some embodiments, an adaptive behavior control system (eg, 110) stores computer-readable data of a computer behavior policy for each interacting participant detected by the adaptive behavior control system.

[0059] In some embodiments, the adaptive behavior control system (e.g., 110) stores computer readable data of a computer behavior strategy for each of a plurality of interaction participants. In some embodiments, the adaptive behavior control system (e.g., 110) stores the same computer behavior strategy for each interaction participant. In some embodiments, the adaptive behavior control system (e.g., 110) independently updates the computer behavior strategy for each interaction participant.

[0060] In some embodiments, the adaptive behavior control system (e.g., 110) updates the computer behavior policy based on human reactions detected for participants associated with the computer behavior policy. In some embodiments, the adaptive behavior control system (e.g., 110) updates the computer behavior policy based on human reactions detected from participants associated with the computer behavior policy. In some embodiments, the computer behavior policy specifies the same action for the first action type. In some embodiments, the computer behavior policy specifies different actions for the first action type. In some embodiments, the computer behavior policy of each participant is initialized with the same weight value. In some embodiments, the computer behavior policy of each participant is initialized with the same weight sequence. In some embodiments, the independent update of the computer behavior policy of each participant results in at least two computer behavior policies with different weight sequences. In some embodiments, the independent update of the computer behavior policy of each participant results in at least two computer behavior policies with different weight values.

[0061] In some embodiments, the data of the computer behavior policy includes instructions for performing an action of a certain action type. In some embodiments, the data of the computer behavior policy includes an identifier of an action of a specific action type (a certain action type) and a corresponding weight value for determining when to perform the action. In some embodiments, the data of the computer behavior policy includes an identifier of an action of a specific action type (a certain action type) and a corresponding weight value for determining when to give priority to the action. In some embodiments, the data of the computer behavior policy includes an identifier of an action of a certain action type and a corresponding weighted sequence of actions of the action type, which is used to determine when to perform the action. In some embodiments, the data of the computer behavior policy includes a computer executable instruction of an action of a certain action type and a corresponding weighted sequence of actions of the action type, which is used to determine when to perform the action. In some embodiments, the data of the computer behavior policy includes a computer executable instruction of an action of a certain action type and a corresponding weighted sequence of actions of the action type, which is used to determine when to perform the action. In some embodiments, the data of the computer behavior policy includes a link to a computer executable instruction of an action of a certain action type and a corresponding weighted value for determining when to perform the action. In some embodiments, the data of the computer behavior policy includes a link to a computer executable instruction for an action of an action type and a corresponding weighted sequence of actions of the action type, the weighted sequence being used to determine when to perform the action.

[0062] In some embodiments, the data of the computer behavior strategy includes data of detected human reactions of interactive participants corresponding to the execution of an action of the first action type. In some embodiments, the data of detected human reactions are data indicating detected emotions of interactive participants. In some embodiments, the data of detected human reactions are data indicating classification of detected emotions of interactive participants. In some embodiments, the classification of detected emotions of interactive participants is one of "positive" and "negative". In some embodiments, the classification of detected emotions of interactive participants is one of "positive", "negative" and "neutral". In some embodiments, the classification of detected emotions of interactive participants is one of multiple reaction values. In some embodiments, the reaction value is an integer with a range (e.g., 0 to 100) or a floating point value with a range (e.g., 0.0 to 1.0), wherein the reaction values ​​are sorted from negative to positive. In some embodiments, the reaction value is a binary value indicating a "positive" or "negative" reaction.

[0063] FIG. 3A to FIG. 3E is a representation of an exemplary computer behavior policy.

[0064] Figure 3A An exemplary computer behavior policy for a first action type ("action A") is depicted. In some embodiments, the first action type is an action type that is performed during execution of a human-computer interaction application that specifies the performance of multiple actions according to an interactive session. In some embodiments, the human-computer interaction application specifies the performance of several action instances of the first action type. In some embodiments, the human-computer interaction application specifies the performance of actions of several action types. Figure 3A , three alternative actions are defined for action type A, namely action A, action B and action C. Figure 3A An initial strategy is depicted where no human response is detected. In some embodiments, Figure 3A During the generation of the strategy 301 of , a weight value for each action of action type A is specified, and these weight values ​​are used to generate a weighted sequence for selecting an action (action A, action B, or action C) of action type A (first action type). In some embodiments, Figure 3A The weighted sequence of specifies that action A is to be performed seven times, followed by action B twice, then action C once, and then the sequence is repeated.

[0065] Figure 3B An update strategy 302 is depicted, which is Figure 3A The updated version of the strategy 301 for each action of the first action type (action type A) has detected ten reactions. Figure 3B As shown, action A has received two positive reactions, action B has received two positive reactions, and action C has received six positive reactions.

[0066] Figure 3C An update strategy 303 is depicted after the human response recorded for action type A has satisfied the update condition. Figure 3C As shown, the weight values ​​have been updated and the corresponding weighted sequence has been updated to reflect the distribution of positive reactions recorded for actions A, B, and C.

[0067] Figure 3D An update strategy 304 is depicted, which is Figure 3C An updated version of the strategy 303 after twenty reactions have been detected for each action of the first action type (action type A). Figure 3D As shown, action A has received ten positive reactions, action B has received three positive reactions, and action C has received seven positive reactions.

[0068] Figure 3E Describes the Figure 3D The update strategy 305 for the positive response depicted in FIG. 305 after the human response recorded for action type A has satisfied the update condition. Figure 3E As shown, the weight values ​​have been updated and the corresponding weighted sequence has been updated to reflect the distribution of positive reactions recorded for actions A, B, and C.

[0069] method

[0070] Method 400

[0071] Figure 4 is a representation of a method according to various embodiments. In some embodiments, the human-computer interaction system performs method 400. In some embodiments, Figure 1A The human-computer interaction system performs method 400. In some embodiments, Figure 1B The human-computer interaction system performs method 400. In some embodiments, Figure 1C The human-computer interaction system performs method 400. In some embodiments, Figure 1D The human-computer interaction system executes method 400.

[0072] In some embodiments, the method 400 includes: (the human-computer interaction system) an adaptive behavior control system controls the interaction subsystem (eg, Figure 1A 112) to be based on the computer behavior policy (e.g. Figure 8 802 and 803) perform multiple actions for a first action type, each action is a different alternative action for the action type (process S401); the adaptive behavior control system is based on the human reaction detection subsystem (e.g., Figure 1A 111) received data to detect (interaction participants, for example, Figure 8 810, 820 in the method 400) to detect a human reaction to the execution of each action of the first action type (process S402); and the adaptive behavior control system stores information indicating each detected human reaction in association with information identifying the associated action (process S403). In some embodiments, method 400 includes: when the stored information indicating the detected human reaction to the first action type satisfies the update condition, the adaptive behavior control system updates the computer behavior strategy for the first action type (process S404).

[0073] In some embodiments, method 400 includes: in response to user input received from an operator device communicatively coupled to the adaptive behavior control system via a network interface (e.g., 211), the adaptive behavior control system updates the computer behavior policy for the first action type based on the received user input.

[0074] External human-computer interaction system

[0075] In some embodiments, method 400 includes: the adaptive behavior control system storing information indicating a human reaction detected by the external human-computer interaction system to the performance of an action of the first action type by the external human-computer interaction system, and the adaptive behavior control system storing the information detected by the external behavior control system in association with the information identifying the associated action. In this way, the adaptive behavior control system updates the strategy used by the adaptive behavior control system based on the human reaction detected by the external human-computer interaction system and the human-computer interaction system of the adaptive behavior control system.

[0076] Weighted Sequence

[0077] In some embodiments, the policy specifies a weighted sequence of multiple actions for a first action type, and the adaptive behavior control system controls the interaction subsystem to perform multiple actions for the first action type according to the weighted sequence for the first action type. As a first example, the following is an exemplary equally weighted sequence of actions A, B, and C: {A, B, C, A, B, C, ...}. As a second example, the following is an exemplary weighted sequence of actions A, B, and C, where the weight values ​​of A, B, and C are 70%, 20%, and 10%, respectively: {A, A, A, A, A, A, A, B, B, C, A, A, A, A, A, A, B, B, C, ...}. As a third example, in some embodiments, the following is an exemplary weighted sequence of actions A, B, and C, where the weight values ​​of A, B, and C are 70%, 20%, and 10%, respectively, where each action is sampled from the following distribution: {C, A, A, A, C, B, A, A, A, A, B, C}.

[0078] Strategy Selection

[0079] In some embodiments, method 400 includes: the adaptive behavior control system selects a computer behavior strategy based on at least one of the identified interaction participants, the determined current interaction context, the action type, the current treatment goal, and the determined current therapy type. In some embodiments, method 400 includes: the adaptive behavior control system selects a computer behavior strategy based on the current treatment goal. In some embodiments, method 400 includes: the adaptive behavior control system determines the current therapy type being used during interaction with at least one participant, and selects a computer behavior strategy based on the determined current therapy type. In some embodiments, the current therapy type includes at least one of ABA (applied behavior analysis), OT (occupational therapy), and MI (motivational interviewing) therapy. In some embodiments, method 400 includes: the adaptive behavior control system identifies at least a first interaction participant, and selects a computer behavior strategy based on the identified first interaction participant. In some embodiments, method 400 includes: the adaptive behavior control system determines the current interaction context, and selects a computer behavior strategy based on the determined current interaction context. In some embodiments, the current interaction context includes a date, a location, an occasion (e.g., a birthday), an event (e.g., a funeral), a detected overall human emotion of at least one interaction participant (e.g., the participant seems happy or sad today), or one of a detected human mood of at least one interaction participant.

[0080] In some embodiments, the adaptive behavior control system includes a plurality of computer behavior strategies. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts for at least one interactive participant. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types for at least one interactive participant. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types for each of a plurality of interactive participants.

[0081] In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of treatment goals for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of treatment goals for at least one interactive participant.

[0082] In some embodiments, the adaptive behavior control system stores information indicating each detected human reaction in association with a selected computer behavior strategy, and the adaptive behavior control system provides information indicating each detected human reaction for the selected strategy to an external system in response to a request provided by the external system via an API of the adaptive behavior control system.

[0083] Execute multiple actions for the first action type according to the computer behavior strategy (process S401)

[0084] In some embodiments, controlling the interaction subsystem to perform multiple actions for a first action type according to a computer behavior strategy includes: controlling the interaction subsystem (e.g., 112, 131, 130, 102c, 102d, 103d, 105d, 106d) to perform a first action for the first action type based on the weight values ​​of the multiple actions for the first action type included in the strategy. In some embodiments, for a first instance of performing an action for the first action type, the adaptive behavior control system (e.g., 110) selects an action with the highest weight value (as specified in the strategy) and performs the selected action. In some embodiments, for a first instance of performing an action for the first action type, the adaptive behavior control system (e.g., 110) randomly selects an action and performs the selected action.

[0085] In some embodiments, controlling an interaction subsystem (e.g., 112) to perform multiple actions for a first action type according to a computer behavior policy includes: controlling the interaction subsystem to perform a first action for the first action type based on weight values ​​of the multiple actions for the first action type included in the policy and log information indicating actions previously performed by the human-computer interaction system for the first action type.

[0086] In some embodiments, controlling the interaction subsystem to perform multiple actions for a first action type according to the computer behavior policy includes: controlling the interaction subsystem to select an action for the first action type based on a weighted sequence of multiple actions for the first action type. For example, the following is an exemplary equally weighted sequence of actions A, B, and C: {A, B, C, A, B, C, ...}. For example, the following is an exemplary weighted sequence of actions A, B, and C, where the weight values ​​of A, B, and C are 70%, 20%, and 10%, respectively: {A, A, A, A, A, A, A, B, B, C, A, A, A, A, A, A, B, B, C, ...}.

[0087] Actions and action types

[0088] In some embodiments, the action type is a type of human interaction, such as, for example, a greeting, a thank you, a gesture, a question, a statement, a reaction, an expression, etc. In some embodiments, for at least one action type (e.g., a greeting action type), the computer behavior policy specifies multiple alternative actions. In some embodiments, the multiple alternative actions include different actions for the action type (e.g., a greeting action). In some embodiments, the multiple alternative actions include different audible actions for the action type (e.g., a greeting action). In some embodiments, the audible action includes an action performed by controlling a speaker (e.g., 130, 102d). In some embodiments, the multiple alternative actions include different body movement actions for the action type (e.g., a greeting action). In some embodiments, the body movement action includes controlling a motor (e.g., Figure 1D In some embodiments, the body movement action includes controlling a robotic accessory (e.g., Figure 1D In some embodiments, the body movement action includes controlling the robot head component (e.g., Figure 1D In some embodiments, the body movement action includes controlling the robot body components (e.g., Figure 1D In some embodiments, the plurality of alternative actions include different displayable actions for the action type (e.g., a greeting action) (presented by a display device). In some embodiments, the displayable action includes controlling a display device (e.g., Figure 1C 1052c) to perform the action.

[0089] In some embodiments, the plurality of alternative actions includes at least two actions for emitting audible sounds at different volumes by controlling the audio output subsystem (eg, 130, 102d).

[0090] In some embodiments, the plurality of alternative actions includes at least two actions for emitting audible sounds at different playback rates by controlling the audio output subsystem.

[0091] In some embodiments, the plurality of alternative actions includes at least two actions for uttering spoken word sounds at different speeds by controlling the audio output subsystem.

[0092] In some embodiments, the plurality of alternative actions includes at least two actions for emitting different sounds by controlling the audio output subsystem.

[0093] In some embodiments, the plurality of alternative actions includes at least two actions for emitting audible output using different voices by controlling the audio output subsystem.

[0094] In some embodiments, the plurality of alternative actions includes at least two actions for emitting audible output corresponding to the spoken words using different tones by controlling the audio output subsystem.

[0095] In some embodiments, the plurality of alternative actions includes at least two actions for emitting audible output corresponding to spoken words corresponding to different verbal instructions by controlling the audio output subsystem.

[0096] In some embodiments, the plurality of alternative actions include at least two actions for displaying different images by controlling the video display subsystem (eg, 102c).

[0097] In some embodiments, the plurality of alternative actions includes at least two actions for performing different body postures by controlling the mechanical robotic subsystems (eg, 103d, 104d, 105d, 162, 163).

[0098] In some embodiments, the plurality of alternative actions includes at least two actions for performing different body postures by controlling a mechanical robotic subsystem including at least one robotic appendage (eg, 105d).

[0099] In some embodiments, the plurality of alternative actions include at least two actions for performing different body postures by controlling a mechanical robotic subsystem comprising at least one robotic appendage (e.g., 105d) and a robotic head assembly (e.g., 103d) mechanically attached to a robotic body assembly (e.g., 104d), wherein the head assembly includes a motorized assembly (e.g., motors 162, 163) configured to rotate the head assembly relative to the body assembly with at least one degree of freedom.

[0100] In some embodiments, the plurality of alternative actions includes at least one action performed by controlling a lighting subsystem comprising an array of lights, wherein the lighting subsystem is configured to individually control each light of the lighting subsystem.

[0101] In some embodiments, the plurality of alternative actions includes at least one action performed by controlling an LED ring including an array of lights, wherein the LED ring is configured to individually control each light of the LED ring.

[0102] In some embodiments, the plurality of alternative actions include at least one action performed by controlling an audio output subsystem. In some embodiments, the plurality of alternative actions include at least one action performed by controlling an audio output subsystem and a video output subsystem. In some embodiments, the plurality of alternative actions include at least one action performed by controlling an audio output subsystem. In some embodiments, the plurality of alternative actions include at least one action performed by controlling an audio output subsystem, a video output subsystem, and a mechanical robotics subsystem.

[0103] In some embodiments, the plurality of alternative actions includes at least one action performed by controlling at least one of an audio output subsystem, a video output subsystem, and a mechanical robotics subsystem.

[0104] In some embodiments, the plurality of alternative actions includes at least one action performed by controlling at least one of an audio output subsystem, a video output subsystem, a mechanical robotic subsystem, a light emitting subsystem, and an LED ring.

[0105] In some embodiments, the plurality of alternative actions include at least one action for performing a facial expression by controlling a robotic head assembly (e.g., 103d). In some embodiments, the head assembly includes at least one mechanical facial feature, and the robotic head assembly is configured to control the movement of the at least one mechanical facial feature to perform at least one facial expression. In some embodiments, the head assembly includes at least one display device, and the robotic head assembly is configured to control the display of the display device to perform at least one facial expression. In some embodiments, the head assembly includes at least one light emitting subsystem, and the robotic head assembly is configured to control the light emission of the light emitting display system to perform at least one facial expression. In some embodiments, the head assembly includes at least one LED ring, and the robotic head assembly is configured to control at least one LED in the LED ring to perform at least one facial expression.

[0106] In some embodiments, the plurality of alternative actions include at least one action of causing the robot head assembly (e.g., 103d) to output audio in a specific voice by controlling the audio output subsystem of the robot head assembly. In some embodiments, the plurality of alternative actions include at least one action of causing the robot head assembly (e.g., 103d) to output audible speech by controlling the audio output subsystem of the robot head assembly.

[0107] Detecting human response (process S402)

[0108] Reaction detection

[0109] In some embodiments, the adaptive behavior control system detects human responses (process S402) including detecting human responses based on responses from a human response detection subsystem (e.g., Figure 1A 111 Figure 1B 120 and 121, Figure 1C 101c, Figure 1D 101d and 106d) receive data to detect the interaction participants (e.g., Figure 8 810, 820) of a human response to the execution of the first action.

[0110] In some embodiments, the adaptive behavior control system is configured to detect human reactions using a gesture detection system that recognizes human gestures based on video data of interacting participants provided by the human reaction detection subsystem. In some embodiments, the adaptive behavior control system includes a data structure that associates a human gesture with one of a positive human reaction and a negative human reaction.

[0111] In some embodiments, the adaptive behavior control system is configured to detect human reactions by using a smile detection system that recognizes human gestures based on video data of interacting participants provided by the human reaction detection subsystem.

[0112] Emotion Detection

[0113] In some embodiments, the adaptive behavior control system detects human reactions (process S402) including using an emotion detection system (e.g., Figure 2 279) based on the human response detection subsystem (e.g., Figure 1A 111 Figure 1B 120 and 121, Figure 1C 101c, Figure 1D 101d and 106d) receive data to detect the interaction participants (e.g., Figure 8In some embodiments, the emotion detection system is included in the human-computer interaction system. In some embodiments, the emotion detection system is included in the adaptive behavior control system. In some embodiments, the emotion detection system is included in the human response detection subsystem. In some embodiments, the emotion detection system is an external system communicatively coupled to the human-computer interaction system via a network interface of the human-computer interaction system. In some embodiments, the emotion detection system is a cloud-based system accessible via an application programming interface (API). In some embodiments, the emotion detection system is Emotion Detection System. In some embodiments, the emotion detection system is Emotion Detection System. In some embodiments, the emotion detection system is Emotion Detection System. In some embodiments, the emotion detection system is Emotion Detection System. In some embodiments, the emotion detection system is Emotion Detection System. In some embodiments, the emotion detection system is Emotion detection system. In some embodiments, the emotion detection system is configured to detect emotions based on captured image data. In some embodiments, the emotion detection system is configured to detect emotions based on captured audio data. In some embodiments, the emotion detection system is configured to detect emotions based on captured image data and captured audio data. In some embodiments, the emotion detection system includes an audible emotion detection subsystem, which is configured to detect emotions based on captured audio data. In some embodiments, the emotion detection system includes a visible emotion detection subsystem, which is configured to detect emotions based on captured image data. In some embodiments, emotions detectable by the emotion detection system include anger, contempt, disgust, fear, happiness, neutrality, sadness, and surprise. In some embodiments, emotions detectable by the emotion detection system include happiness, sadness, anger, confusion, disgust, surprise, calmness, and unknown. In some embodiments, the emotion detection system is configured to classify the detected emotions as positive, negative, or neutral. In some embodiments, the adaptive behavior control system obtains a determined emotion classification (eg, positive, neutral, negative) using an emotion detection system after performing an action of the first action type and stores the determined emotion classification in association with the performed action.

[0114] Storing information indicating each detected human response (process S403)

[0115] In some embodiments, the adaptive behavior control system stores information indicating each detected human reaction (process S403) including storing data indicating each detected emotion. In some embodiments, the adaptive behavior control system stores information indicating each detected human reaction (process S403) including: determining a classification of each detected emotion, and storing data indicating the classification of each detected emotion. In some embodiments, the classification of the detected emotion of the interaction participant is one of "positive" and "negative". In some embodiments, the classification of the detected emotion of the interaction participant is one of "positive", "negative" and "neutral".

[0116] In some embodiments, the adaptive behavior control system stores information indicating each detected human response (process S403) including: for each detected human response, determining whether the response is related to at least one therapeutic goal, and storing data indicating each therapeutic goal related to the detected human response.

[0117] In some embodiments, at least one therapeutic goal is a short-term interaction goal (e.g., smile back at the robot). In some embodiments, at least one therapeutic goal is a long-term goal (e.g., demonstrate a long-term behavioral skill).

[0118] In some embodiments, the adaptive behavior control system stores information indicating each detected human reaction (process S403) including: for each detected human reaction, determining whether the reaction is related to at least one interaction target, and storing data indicating each interaction target related to the detected human reaction.

[0119] In some embodiments, at least one interaction goal is a short-term interaction goal (e.g., smiling back at the robot). In some embodiments, at least one interaction goal is a long-term goal (e.g., demonstrating a long-term behavioral skill).

[0120] Update the computer behavior policy for the first action type (process S404)

[0121] In some embodiments, the strategy (e.g., 301 to 305) specifies a weighted sequence of multiple actions for a first action type, and the adaptive behavior control system controls the interaction subsystem to perform the multiple actions for the first action type according to the weighted sequence for the first action type. In some embodiments, the adaptive behavior control system stores information indicating each detected human reaction (process S403) including: determining a classification of each detected emotion, and storing data indicating the classification of each detected emotion (e.g., "positive", "negative", and "neutral").

[0122] In some embodiments, updating the computer behavior policy for the first action type (process S404) includes: determining, based on the result of the distribution comparison, whether the difference between the distribution of detected positive human emotions stored for the first action type and the distribution of actions in the weighted sequence of actions for the first action is higher than a difference threshold; and updating the computer behavior policy for the first action type in response to determining that the difference between the distribution of detected positive human emotions stored for the first action type and the distribution of actions in the weighted sequence of actions for the first action is higher than the difference threshold.

[0123] In some embodiments, updating the computer behavior policy for the first action type (process S404) includes: determining, based on the result of the distribution comparison, whether the difference between the distribution of detected negative human emotions stored for the actions of the first action type and the distribution of actions in the weighted sequence of actions for the first action is higher than a difference threshold; and updating the computer behavior policy for the first action type in response to determining that the difference between the distribution of detected negative human emotions stored for the actions of the first action type and the distribution of actions in the weighted sequence of actions for the first action is higher than the difference threshold.

[0124] In some embodiments, updating the computer behavior policy for a first action type (process S404) includes: determining, based on a result of the distribution comparison, whether a difference between the distribution of detected human reactions related to the treatment target in the stored human reaction information for the actions of the first action type and the distribution of actions in the weighted sequence of actions for the first action is higher than a difference threshold; and updating the computer behavior policy for the first action type in response to determining that a difference between the distribution of reactions related to the treatment target for the actions of the first action type and the distribution of actions in the weighted sequence of actions for the first action is higher than the difference threshold.

[0125] In some embodiments, updating the computer behavior policy for a first action type (process S404) includes: determining, based on a result of the distribution comparison, whether a difference between the distribution of detected human reactions related to any one of a plurality of therapeutic targets in the stored human reaction information for the action of the first action type and the distribution of actions in a weighted sequence of actions for the first action is higher than a difference threshold; and updating the computer behavior policy for the first action type in response to determining that a difference between the distribution of reactions related to any one of a plurality of therapeutic targets for the action of the first action type and the distribution of actions in a weighted sequence of actions for the first action is higher than a difference threshold.

[0126] In some embodiments, the distribution comparison is performed after a predetermined number of iterations are performed on the sequence. In some embodiments, the distribution comparison is performed periodically at regular time intervals. In some embodiments, the distribution comparison is performed after a predetermined number of positive human emotions have been recorded. In some embodiments, the distribution comparison is performed after a predetermined number of positive human emotions have been recorded for a specific action of the first action type.

[0127] In some embodiments, the distribution comparison is performed after a predetermined number of negative human emotions have been recorded. In some embodiments, the distribution comparison is performed after a predetermined number of negative human emotions have been recorded for a particular action of the first action type.

[0128] In some embodiments, the distribution comparison is performed after a predetermined number of human responses have been recorded for at least one treatment target. In some embodiments, the distribution comparison is performed after a predetermined number of human responses have been recorded for at least one treatment target of the first action type.

[0129] In some embodiments, the distribution comparison is performed after a predetermined number of human reactions have been recorded for at least one interaction target. In some embodiments, the distribution comparison is performed after a predetermined number of human reactions have been recorded for at least one interaction target of the first action type.

[0130] For example, in a weighted sequence having a distribution of action A (70%), action B (20%), action C (10%), where a positive reaction is recorded as action A (10%), action B (20%), action C (70%), the computer behavior policy will be updated for the first action type. In some embodiments, the weighted sequence is updated so that the distribution of actions within the sequence matches the distribution of positive reactions recorded for actions of the first action type. In some embodiments, the weighted sequence is updated so that the distribution of actions within the sequence is incrementally adjusted based on the distribution of positive reactions recorded for actions of the first action type.

[0131] In some embodiments, updating the computer behavior policy for a first action type (process S404) includes: determining whether a predetermined number of consecutive positive human emotions have been detected for at least one action of the first action type; and updating the computer behavior policy for the first action type in response to determining that a predetermined number of consecutive positive human emotions have been detected for at least one action of the first action type.

[0132] In some embodiments, updating the computer behavior policy for a first action type (process S404) includes: determining whether a predetermined number of consecutive positive human reactions have been detected for at least one action of the first action type; and updating the computer behavior policy for the first action type in response to determining that a predetermined number of consecutive positive human reactions have been detected for at least one action of the first action type.

[0133] Method 500

[0134] Figure 5 is a representation of a method according to various embodiments. In some embodiments, the human-computer interaction system performs method 500. In some embodiments, Figure 1A The human-computer interaction system performs method 500. In some embodiments, Figure 1B The human-computer interaction system performs method 500. In some embodiments, Figure 1C The human-computer interaction system performs method 500. In some embodiments, Figure 1D The human-computer interaction system executes method 500.

[0135] In some embodiments, the method 500 includes: (of a human-computer interaction system, such as 100a, 100b, 100c, 100d) an adaptive behavior control system (e.g., Figure 1A 110) based on computer behavior strategies (e.g. FIG. 3A to FIG. 3E 301 to 305) in the above) to select the first action among the multiple actions according to the weight values ​​of the multiple actions specified by the computer behavior strategy for the first action type (process S501); the adaptive behavior control system controls the interaction subsystem of the human-computer interaction system (for example, Figure 1A 112) to perform the selected first action (process S502); the adaptive behavior control system is based on the human response detection subsystem (e.g., Figure 1A 111) receives data to detect interaction participants (e.g., Figure 8810, 820) to the human reaction of the execution of the first action (process S503); and the adaptive behavior control system stores the information indicating the detected human reaction in association with the information identifying the first action (process S504). In some embodiments, method 500 includes: when the stored information indicating the detected human reaction to the first action satisfies the update condition, the adaptive behavior control system updates the weight value of the first action in the computer behavior policy (process S505). In some embodiments, method 500 includes: in response to user input received from an operator device communicatively coupled to the adaptive behavior control system via a network interface (e.g., 211), the adaptive behavior control system updates the weight value of the first action in the computer behavior policy based on the received user input.

[0136] In some embodiments, process S502 is similar to Figure 4 In some embodiments, process S503 is similar to Figure 4 In some embodiments, process S504 is similar to Figure 4 In some embodiments, process S505 is similar to Figure 4 Process S404.

[0137] Strategy Selection

[0138] In some embodiments, method 500 includes the adaptive behavior control system selecting a computer behavior strategy based on at least one of the identified interacting participants, the determined current interaction context, the action type, the current treatment goal, and the determined current therapy type.

[0139] In some embodiments, method 500 includes the adaptive behavior control system selecting a computer behavior strategy based on a current treatment goal.

[0140] In some embodiments, method 500 includes: the adaptive behavior control system determines a current therapy type being used during an interaction with at least one participant, and selects a computer behavior strategy based on the determined current therapy type. In some embodiments, the current therapy type includes at least one of ABA (applied behavior analysis), OT (occupational therapy), and MI (motivational interviewing) therapy. In some embodiments, method 500 includes: the adaptive behavior control system identifies at least a first interaction participant, and selects a computer behavior strategy based on the identified first interaction participant. In some embodiments, method 500 includes: the adaptive behavior control system determines a current interaction context, and selects a computer behavior strategy based on the determined current interaction context. In some embodiments, the current interaction context includes one of a date, an occasion (e.g., a birthday), an event (e.g., a funeral), and a detected overall human emotion of at least one interaction participant (e.g., the participant seems happy or sad today).

[0141] In some embodiments, the adaptive behavior control system includes a plurality of computer behavior strategies. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts for at least one interactive participant. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types for at least one interactive participant. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types for each of a plurality of interactive participants.

[0142] In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of treatment goals for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of treatment goals for at least one interactive participant.

[0143] In some embodiments, the adaptive behavior control system stores information indicating each detected human reaction in association with a selected computer behavior strategy, and the adaptive behavior control system provides information indicating each detected human reaction for the selected strategy to an external system in response to a request provided by the external system via an API of the adaptive behavior control system.

[0144] Method 600

[0145] Figure 6 is a representation of a method according to various embodiments. In some embodiments, the human-computer interaction system performs method 600. In some embodiments, Figure 1A The human-computer interaction system performs method 600. In some embodiments, Figure 1B The human-computer interaction system performs method 600. In some embodiments, Figure 1C The human-computer interaction system performs method 600. In some embodiments, Figure 1D The human-computer interaction system executes method 600.

[0146] In some embodiments, the method 600 includes: an adaptive behavior control system of a human-computer interaction system detects a plurality of interaction participants (e.g., Figure 8 810, 820) (process S601); the adaptive behavior control system stores at least one computer behavior strategy for each detected interactive participant (e.g., Figure 8 802, 803) (S602); the adaptive behavior control system controls the interaction subsystem (e.g., 112, 131, 130, 102c, 102d, 103d, 105d, 106d) to perform an action of the first action type for the interaction participant according to the computer behavior strategy selected for each detected interaction participant (S603); for each detected interaction participant, the adaptive behavior control system performs an action of the first action type according to the computer behavior strategy selected from the human reaction detection subsystem (e.g., Figure 1A 111 Figure 1B 120 and 121, Figure 1C 101c, Figure 1D101d and 106d) to detect the human response of the interactive participant to the execution of each action of the first action type (S604); and for each detected human interactive participant, the adaptive behavior control system stores information indicating each detected human response in association with information identifying the associated action (S605). In some embodiments, method 600 includes: for each detected human interactive participant, if the stored information indicating the detected human response to the first action type satisfies an update condition, the adaptive behavior control system updates the selected computer behavior strategy of the interactive participant for the first action type (S606).

[0147] In some embodiments, method 600 includes, in response to user input received from an operator device communicatively coupled to the adaptive behavior control system via a network interface (eg, 211), the adaptive behavior control system updating at least one computer behavior policy based on the received user input.

[0148] The computer behavior policy for each detected interacting participant is updated independently based on the detected human responses of the associated interacting participant.

[0149] In some embodiments, the adaptive behavior control system stores at least a first computer behavior strategy for a first interactive participant and a second computer behavior strategy for a second interactive participant, the first strategy and the second strategy each specifying the same multiple actions for a first action type, each action being a different alternative action for the action type.

[0150] In some embodiments, process S603 is similar to Figure 4 In some embodiments, process S604 is similar to Figure 4 In some embodiments, process S605 is similar to Figure 4 In some embodiments, process S606 is similar to Figure 4 Process S404.

[0151] Strategy Selection

[0152] In some embodiments, method 600 includes: the adaptive behavior control system selects a computer behavior strategy for each detected interactive participant based on at least one of the following: the identity of the interactive participant, the current interaction context determined for the interactive participant, the action type, the current treatment goal, and the current therapy type determined for the interactive participant.

[0153] In some embodiments, method 600 includes the adaptive behavior control system selecting a computer behavior strategy for at least one detected interacting participant based on a current treatment goal.

[0154] In some embodiments, method 600 includes: for at least one detected interaction participant, the adaptive behavior control system determines the current therapy type being used during the interaction with the participant, and selects a computer behavior strategy based on the determined current therapy type. In some embodiments, the current therapy type includes at least one of ABA (applied behavior analysis), OT (occupational therapy), and MI (motivational interviewing) therapy. In some embodiments, method 600 includes: for at least one detected participant, the adaptive behavior control system selects a computer behavior strategy based on the identification of the interaction participant. In some embodiments, method 600 includes: for at least one detected participant, the adaptive behavior control system determines the current interaction context of the participant, and selects a computer behavior strategy based on the determined current interaction context. In some embodiments, the current interaction context includes one of a date, an occasion (e.g., a birthday), an event (e.g., a funeral), and a detected overall human emotion of at least one interaction participant (e.g., the participant seems happy or sad today).

[0155] In some embodiments, the adaptive behavior control system includes a plurality of computer behavior strategies. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts of at least one interactive participant. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types of at least one interactive participant. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types for each of a plurality of interactive participants.

[0156] In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of treatment goals for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of treatment goals for at least one interactive participant.

[0157] In some embodiments, the adaptive behavior control system stores information indicating each detected human reaction in association with a corresponding computer behavior strategy, and in response to a request for human reaction information for a specified strategy provided by an external system via an API of the adaptive behavior control system, the adaptive behavior control system provides information indicating each detected human reaction for the specified strategy to the external system.

[0158] Method 700

[0159] Figure 7 is a representation of a method according to various embodiments. In some embodiments, Figure 1D The human-computer interaction system executes method 700.

[0160] In some embodiments, the human-computer interaction system that executes method 700 includes: a body component (e.g., 104d); a head component (e.g., 103d), which is mechanically attached to the body component via a motor component (e.g., 162, 163); the motor component, which is configured to rotate the head component around a vertical axis (e.g., 160d) and a horizontal axis (e.g., 161d); at least a first motorized attachment (e.g., 105d), which is configured to perform movements according to at least one human interaction gesture (e.g., arm gesture, hand gesture, waving, sign language, etc.); an adaptive behavior control system (e.g., 110), which is communicatively coupled to at least one of the motor component, the head component, and the attachment; a human reaction detection subsystem (e.g., 111, 101d, 106d), which is communicatively coupled to the adaptive behavior control system; and an interaction subsystem (e.g., 102d), which is communicatively coupled to the adaptive behavior control system.

[0161] In some embodiments, the method 700 includes: a human-computer interaction system (eg, Figure 1D The adaptive behavior control system (e.g., 110) of 100d) detects multiple interactive participants (e.g., Figure 8 810, 820) (process S701); the adaptive behavior control system stores at least one computer behavior strategy for each detected interactive participant (e.g., Figure 8802, 803) (S702); the adaptive behavior control system controls at least one of the interactive subsystem (e.g., 102d), the motor assembly, the accessory (e.g., 105d), the body assembly (104d), and the head assembly (103d) to perform an action of a first action type for the interactive participant according to a computer behavior strategy selected for each detected interactive participant (S703); for each detected interactive participant, the adaptive behavior control system detects a human response of the interactive participant to the execution of each action of the first action type according to data received from the human response detection subsystem (S704); and for each detected human interactive participant, the adaptive behavior control system stores information indicating each detected human response in association with information identifying an associated action (S705). In some embodiments, the method 700 includes: for each detected human interactive participant, when the stored information indicating the detected human response to the first action type satisfies an update condition, the adaptive behavior control system updates the computer behavior strategy of the interactive participant for the first action type (S706).

[0162] In some embodiments, method 700 includes, in response to user input received from an operator device communicatively coupled to the adaptive behavior control system via a network interface (eg, 211), the adaptive behavior control system updating at least one computer behavior policy based on the received user input.

[0163] The computer behavior policy for each detected interacting participant is updated independently based on the detected human responses of the associated interacting participant.

[0164] In some embodiments, detecting a plurality of interaction participants includes detecting a position of each detected interaction participant, and generating a detected interaction participant (eg, Figure 8 810, 820) of the detected position of the three-dimensional world model (e.g. Figure 8 of 278).

[0165] In some embodiments, during initiation of an interaction with a previously detected interacting participant (e.g., 810, 820), the adaptive behavior control system controls the motor assembly (e.g., 162, 163) to orient the head assembly (e.g., 103d) in the direction of the previously detected interacting participant by using the previously detected position included in the world model (e.g., 278).

[0166] In some embodiments, during processing of a previously detected interactive participant's response to an action performed, the adaptive behavior control system controls the motor assembly (e.g., 162, 163) to orient the head assembly (e.g., 103d) in the direction of the previously detected interactive participant by using the previously detected position included in the world model (e.g., 278).

[0167] In some embodiments, detecting multiple interaction participants includes detecting the face of each interaction participant. In some embodiments, the orientation of each detected face is recorded. In some embodiments, the three-dimensional world model (e.g., 278) indicates the orientation of each detected face, and during the initiation of an interaction with a previously detected interaction participant, the adaptive behavior control system controls the motor assembly to orient the head assembly in the direction of the previously detected face of the interaction participant by using the previously detected position and facial orientation included in the world model.

[0168] In some embodiments, detecting multiple interactive participants includes detecting a face of each interactive participant, and the three-dimensional world model (e.g., 278) includes a detected orientation of the detected faces, and during processing a previously detected interactive participant's response to the performed action, the adaptive behavior control system controls the motor assembly to orient the head assembly in the direction of the previously detected interactive participant's face by using the previously detected position and facial orientation included in the world model.

[0169] In some embodiments, the adaptive behavior control system stores at least a first computer behavior strategy for a first interactive participant and a second computer behavior strategy for a second interactive participant, the first strategy and the second strategy each specifying the same multiple actions for a first action type, each action being a different alternative action for the action type.

[0170] In some embodiments, process S703 is similar to Figure 4 In some embodiments, process S704 is similar to Figure 4 In some embodiments, process S705 is similar to Figure 4 In some embodiments, process S706 is similar to Figure 4 Process S404.

[0171] Strategy Selection

[0172] In some embodiments, method 700 includes: the adaptive behavior control system selects a computer behavior strategy for each detected interactive participant based on at least one of the following: the identity of the interactive participant, the current interaction context determined for the interactive participant, the action type, the current treatment goal, and the current therapy type determined for the interactive participant.

[0173] In some embodiments, method 600 includes the adaptive behavior control system selecting a computer behavior strategy for at least one detected interacting participant based on a current treatment goal.

[0174] In some embodiments, the method 700 includes: for at least one detected interaction participant, the adaptive behavior control system determines the current therapy type being used during the interaction with the participant, and selects a computer behavior strategy based on the determined current therapy type. In some embodiments, the current therapy type includes at least one of ABA (applied behavior analysis), OT (occupational therapy), and MI (motivational interviewing) therapy. In some embodiments, the method 700 includes: for at least one detected participant, the adaptive behavior control system selects a computer behavior strategy based on the identification of the interaction participant. In some embodiments, the method 700 includes: for at least one detected participant, the adaptive behavior control system determines the current interaction context of the participant, and selects a computer behavior strategy based on the determined current interaction context. In some embodiments, the current interaction context includes one of a date, an occasion (e.g., a birthday), an event (e.g., a funeral), and a detected overall human emotion of at least one interaction participant (e.g., the participant seems happy or sad today).

[0175] In some embodiments, the adaptive behavior control system includes a plurality of computer behavior strategies. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts for at least one interactive participant. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types for at least one interactive participant. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of interactive contexts for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types for each of a plurality of interactive participants.

[0176] In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of treatment goals for each of a plurality of interactive participants. In some embodiments, the adaptive behavior control system includes a computer behavior strategy for each of a plurality of treatment goals for at least one interactive participant.

[0177] In some embodiments, the adaptive behavior control system stores information indicating each detected human reaction in association with a corresponding computer behavior strategy, and in response to a request for human reaction information for a specified strategy provided by an external system via an API of the adaptive behavior control system, the adaptive behavior control system provides information indicating each detected human reaction for the specified strategy to the external system.

[0178] Multi-party and multi-plane interaction

[0179] In some embodiments, the first action type of action is used to control the robot body head component of the human-machine interaction system (e.g., Figure 1D In some embodiments, the action of the first action type is an action for controlling the robot body head component of the human-machine interaction system to orient the head component in the direction of the detected face of the detected interaction participant.

[0180] In some embodiments, during initiation of an interaction with a previously detected interaction participant, the human-computer interaction system (e.g., Figure 1D 100d) is configured to orient the head assembly in the direction of a previously detected interactive participant (e.g., 810, 820) by using a previously detected position of the participant specified by a world model (e.g., 278) stored by the human-computer interaction system. In some embodiments, during initiation of an interaction with a previously detected interactive participant, the human-computer interaction system (e.g., Figure 1D 100d) is configured to orient the head assembly in the direction of a previously detected face of a previously detected interacting participant by using a previously detected position and a previously detected facial orientation of the participant specified by a world model (e.g., 278) stored by the human-computer interaction system.

[0181] In some embodiments, during processing of a previously detected response from an interactive participant to an action performed, the human-computer interaction system (e.g., Figure 1D100d) is configured to orient the head assembly in the direction of a previously detected interactive participant (e.g., 810, 820) by using a previously detected position of the participant specified by a world model (e.g., 278) stored by the human-computer interaction system. In some embodiments, during processing of a response of a previously detected interactive participant to the performed action, the human-computer interaction system (e.g., Figure 1D 100d) is configured to orient the head assembly in the direction of a previously detected face of a previously detected interacting participant by using a previously detected position and a previously detected facial orientation of the participant specified by a world model (e.g., 278) stored by the human-computer interaction system.

[0182] In some embodiments, the head assembly (e.g., 103d) includes a motorized assembly (e.g., 162, 163) configured to rotate the head assembly (e.g., 103d) relative to the body assembly (e.g., 104d) with at least one degree of freedom, and during control of the robot head assembly to orient the head assembly in the direction of an interactive participant, the adaptive behavior control system controls the head assembly to rotate about a first axis (e.g., 160d) in the direction of the detected interactive participant.

[0183] Motorized assembly allows the head to rotate

[0184] In some embodiments, the head assembly includes a motorized assembly (e.g., 162, 163) configured to rotate the head assembly relative to the body assembly with at least two degrees of freedom, and during the control of the robot head assembly to orient the head assembly toward the direction of the detected face of the interactive participant, the adaptive behavior control system controls the head assembly to rotate about a first axis (e.g., 160d) in the direction of the interactive participant, and controls the head assembly to rotate about a second axis (e.g., 161d) toward the detected face. In some embodiments, the first axis is a vertical axis (e.g., 160d), and the second axis is a horizontal axis (e.g., 161d).

[0185] Testing participants

[0186] In some embodiments, the adaptive behavior control system detects participants by using at least one of temperature data captured by a thermal detection subsystem (e.g., an infrared camera, a thermal sensor, etc.), image data captured by a video capture subsystem (e.g., 101d), and audio data captured by an audio capture subsystem (e.g., 106d).

[0187] In some embodiments, the adaptive behavior control system determines the direction of the detected participant by using at least one of temperature data captured by a thermal detection subsystem (e.g., an infrared camera, a thermal sensor, etc.), image data captured by a video capture subsystem (e.g., 101d), audio data captured by an audio capture subsystem (e.g., 106d), and rotational position information of motor assemblies (e.g., 162, 163). In some embodiments, the adaptive behavior control system determines the direction of the detected participant by using at least one of temperature data captured by a thermal detection subsystem (e.g., an infrared camera, a thermal sensor, etc.), image data captured by a video capture subsystem (e.g., 101d), audio data captured by an audio capture subsystem (e.g., 106d), IMU data generated by an IMU of a human-computer interaction system, accelerometer data generated by an accelerometer of a human-computer interaction system, and rotational position information of motor assemblies (e.g., 162, 163).

[0188] In some embodiments, for each detected interacting participant, the adaptive behavior control system stores information identifying the detected participant in a world model (eg, 278) in association with information indicating the determined direction of the detected participant.

[0189] In some embodiments, the adaptive behavior control system determines the orientation of the detected face of the participant by using at least one of temperature data captured by a thermal detection subsystem (e.g., an infrared camera, a thermal sensor, etc.), image data captured by a video capture subsystem (e.g., 101d), audio data captured by an audio capture subsystem (e.g., 106d), and rotational position information of motor assemblies (e.g., 162, 163). In some embodiments, the adaptive behavior control system determines the orientation of the detected face of the participant by using at least one of temperature data captured by a thermal detection subsystem (e.g., an infrared camera, a thermal sensor, etc.), image data captured by a video capture subsystem (e.g., 101d), audio data captured by an audio capture subsystem (e.g., 106d), IMU data generated by an IMU of a human-computer interaction system, accelerometer data generated by an accelerometer of a human-computer interaction system, and rotational position information of motor assemblies (e.g., 162, 163).

[0190] In some embodiments, for each detected interacting participant, the adaptive behavior control system stores information identifying the detected participant in a world model (e.g., 278) in association with information indicating a determined direction of the detected participant and information indicating a determined facial orientation of the participant.

[0191] In some embodiments, the audio capture subsystem captures audio emitted by the participant and generates audio data corresponding to the captured audio, and the adaptive behavior control system uses the audio data to identify the presence and general location of the participant. In some embodiments, the audio capture subsystem includes a microphone array configured to generate directional information corresponding to the captured audio based on directions associated with microphones in the array that capture audio at a highest volume level among the volume levels of audio captured by microphones of the microphone array. Figure 1B As shown, microphone 120 includes a microphone array of six microphones. In some embodiments, the adaptive behavior control system determines that the participant emitting audio is closest to microphone 120, which captures the highest audio volume of the audio emitted by the participant.

[0192] In some embodiments, the audio capture subsystem captures audio emitted by a participant and generates audio data corresponding to the captured audio, and the adaptive behavior control system uses the audio data to identify the presence of the participant by identifying the audio emitted by the participant as matching audio emitted by a human. In some embodiments, the audio capture subsystem captures audio emitted by a participant and generates audio data corresponding to the captured audio, and the adaptive behavior control system uses the audio data to identify the presence of the participant by identifying the audio emitted by the participant as matching audio emitted by a human that is not a currently identified participant. In some embodiments, the audio capture subsystem captures audio emitted by a participant, generates audio data corresponding to the captured audio, determines whether the audio data corresponds to a human voice, determines whether the human voice corresponds to a current participant, and in response to determining that the human voice does not correspond to a current human interaction participant, the adaptive behavior control system identifies the presence of a new participant.

[0193] In some embodiments, the audio capture subsystem captures audio and generates audio data corresponding to the captured audio, the adaptive behavior control system determines whether the audio data corresponds to an environmental change corresponding to the presence of a new interactive participant, and in the case where the adaptive behavior control system determines that the audio data corresponds to an environmental change corresponding to the presence of a new interactive participant, the adaptive behavior control system identifies the presence of the new interactive participant. In some embodiments, the audio data corresponding to the environmental change corresponding to the presence of the new interactive participant includes audio data corresponding to an opening of a door. In some embodiments, the audio data corresponding to the environmental change corresponding to the presence of the new interactive participant includes audio data corresponding to footsteps. In some embodiments, the audio data corresponding to the environmental change corresponding to the presence of the new interactive participant includes audio data corresponding to breathing.

[0194] In some embodiments, by using a video capture subsystem (e.g., Figure 1D In some embodiments, the video capture subsystem captures an image and generates image data corresponding to the captured image, the adaptive behavior control system determines whether the image data corresponds to a human face, and in response to determining that the image data corresponds to a human face, the adaptive behavior control system performs a facial detection process to identify the orientation of the detected human face.

[0195] In some embodiments, the adaptive behavior control system is configured to detect the presence and corresponding position of the interactive participants and generate a three-dimensional world model of the detected position of the detected interactive participants. In some embodiments, the adaptive behavior control system is configured to detect the presence of the interactive participants by using a video capture subsystem of the human-computer interaction system, as described herein. In some embodiments, the adaptive behavior control system is configured to detect the presence of the interactive participants by using an audio capture subsystem of the human-computer interaction system, as described herein. In some embodiments, the adaptive behavior control system is configured to detect the position of the interactive participants by using a video capture subsystem, as described herein. In some embodiments, the adaptive behavior control system is configured to detect the position of the interactive participants by using an audio capture subsystem, as described herein.

[0196] In some embodiments, during initiation of an interaction with a previously detected interacting participant, the adaptive behavior control system controls the facial component to orient the facial component in the direction of the detected interacting participant by using the previously detected position included in the world model.

[0197] In some embodiments, during processing of a previously detected interactive participant's response to a performed action, the adaptive behavior control system controls the facial component to orient the facial component in the direction of the detected interactive participant using the previously detected position included in the world model.

[0198] In some embodiments, the adaptive behavior control system is configured to generate a three-dimensional world model of all detected faces, and wherein, during initiation of an interaction with a participant associated with a previously detected face, the adaptive behavior control system controls the facial components to orient the facial components in the direction of the detected face by using the previously detected direction included in the world model.

[0199] System Architecture

[0200] Figure 2 is a diagram illustrating an adaptive behavior control system (eg, Figure 1A 110) is a simplified diagram of the system architecture. In some embodiments, Figure 2The system is implemented as a single hardware device. In some embodiments, Figure 2 The system is implemented as a plurality of hardware devices. In some embodiments, Figure 2 The system is implemented as an ASIC (Application Specific Integrated Circuit), Figure 2 The system is implemented as an FPGA (Field Programmable Gate Array), Figure 2 The system is implemented as a SoC (system on chip). In some embodiments, the bus 201 interfaces with the processors 201A-N, the main memory 222 (e.g., random access memory (RAM)), the read-only memory (ROM) 206, the processor-readable storage medium 205, and the network device 211. In some embodiments, the bus 201 interfaces with at least one of a display device (e.g., 102c) and a user input device.

[0201] In some embodiments, bus 201 interfaces with audio system device 271. In some embodiments, audio system device 271 includes audio input processor 121. In some embodiments, audio system device 271 includes audio output controller 131. In some embodiments, audio system interface device 271 includes an audio system of human response detection subsystem 111. In some embodiments, audio system interface device 271 includes an audio system of interaction subsystem 111. In some embodiments, audio system device 271 includes a speaker (e.g., Figure 1B 120 Figure 1D In some embodiments, the audio system device 271 includes a microphone (e.g., Figure 1B 130 Figure 1D 106d).

[0202] In some embodiments, bus 201 interfaces with video system device 272. In some embodiments, video system device 272 includes a video system of human response detection subsystem 111. In some embodiments, video system device 272 includes a display system of interaction subsystem 111. In some embodiments, video system device 272 includes a camera (e.g., Figure 1C 101c, Figure 1D In some embodiments, the video system device 272 includes a display (e.g., Figure 1C 102C, Figure 1D 106d).

[0203] In some embodiments, bus 201 interfaces with motor controller 273. In some embodiments, motor controller 273 is configured to control Figure 1D In some embodiments, the motor controller 273 is configured to control at least one of the motors 162 and 163 of the robot system 100d. Figure 1D In some embodiments, the motor controller 273 is configured to control the attachment (e.g., 105d) of the robot system 100d. Figure 1D In some embodiments, the motor controller 273 is configured to control the motor of an accessory (e.g., 105d) of the robotic system 100d. Figure 1D A motor of the body component 104d of the robotic system 100d.

[0204] In some embodiments, the processor includes one or more of an ARM processor, an X86 processor, a GPU (graphics processing unit), etc. In some embodiments, at least one of the processors includes at least one arithmetic logic unit (ALU) that supports a SIMD (single instruction multiple data) system that provides native support for multiplication and accumulation operations.

[0205] In some embodiments, at least one of a central processing unit (processor), a GPU, and a multi-processor unit (MPU) is included.

[0206] In some embodiments, the processor and the main memory form a processing unit 299. In some embodiments, the processing unit includes one or more processors communicatively coupled to one or more of a RAM, a ROM, and a machine-readable storage medium; the one or more processors in the processing unit receive instructions stored by one or more of the RAM, the ROM, and the machine-readable storage medium via a bus; and the one or more processors execute the received instructions. In some embodiments, the processing unit is an ASIC (Application Specific Integrated Circuit). In some embodiments, the processing unit is a SoC (System on Chip).

[0207] In some embodiments, the processing unit includes at least one arithmetic logic unit (ALU) that supports a SIMD (single instruction multiple data) system that provides native support for multiplication and accumulation operations. In some embodiments, the processing unit is a central processing unit, such as an Intel Xeon processor. In other embodiments, the processing unit includes a graphics processing unit such as an NVIDIA Tesla.

[0208] The network adapter device 211 provides one or more wired or wireless interfaces for exchanging data and commands. Such wired and wireless interfaces include, for example, a universal serial bus (USB) interface, a Bluetooth interface, a Wi-Fi interface, an Ethernet interface, a near field communication (NFC) interface, etc.

[0209] In some embodiments, the network device 211 is communicatively coupled to another human-computer interaction system. In some embodiments, the network device 211 is communicatively coupled to another human-computer interaction system via a network (e.g., the Internet). In some embodiments, the network device 211 is communicatively coupled to a human-computer interaction server via a network (e.g., the Internet), and the human-computer interaction server is configured to provide a computer behavior strategy to the adaptive behavior control system via the network device 211. In some embodiments, the human-computer interaction server is configured to provide information of a detected human response to at least one action recognition of the computer behavior strategy of the adaptive behavior control system by another human-computer interaction system via the network device 211. In this way, Figure 2 The adaptive behavior control system is constructed based on the external human-machine interaction system (e.g., robot interaction system, such as Figure 1D 100d) detects human responses to update its computer behavior strategy.

[0210] The machine executable instructions in software programs (such as operating systems, application programs, and device drivers) are loaded into the memory (of the processing unit) from a processor readable storage medium, ROM, or any other storage location. During the execution of these software programs, the corresponding machine executable instructions are accessed by at least one of the processors (of the processing unit) via the bus and then executed by at least one of the processors. Data used by the software programs are also stored in the memory, and such data is accessed by at least one of the processors during the execution of the machine executable instructions of the software programs.

[0211] The processor readable storage medium 605 is one (or a combination of two or more) of a hard drive, a flash drive, a DVD, a CD, an optical disk, a floppy disk, a flash memory, a solid state drive, a ROM, an EEPROM, an electronic circuit, a semiconductor memory device, etc. The processor readable storage medium 205 includes machine executable instructions (and associated data) for the operating system 212, the software programs 213, the device driver 214, and the processor readable storage medium 205. Figures 4 to 7 In some embodiments, the processor-readable storage medium 205 includes data of at least one computer behavior policy 277. In some embodiments, the processor-readable storage medium 205 includes stored information indicating detected human reactions. In some embodiments, the processor-readable storage medium 205 includes a world model 278 of detected locations of detected interaction participants, as described herein.

[0212] In some embodiments, the processor-readable storage medium 205 includes machine executable instructions for the emotion detection module 279. In some embodiments, the emotion detection module 279 is configured to detect emotions based on the captured image data. In some embodiments, the emotion detection module 279 is configured to detect emotions based on the captured audio data. In some embodiments, the emotion detection module 279 is configured to detect emotions based on the captured image data and the captured audio data. In some embodiments, the emotions that can be detected by the emotion detection module 279 include anger, contempt, disgust, fear, happiness, neutrality, sadness, and surprise. In some embodiments, the emotions that can be detected by the emotion detection module 279 include happiness, sadness, anger, confusion, disgust, surprise, calmness, and unknown. In some embodiments, the emotion detection module 279 is configured to classify the detected emotions as positive, negative, or neutral. In some embodiments, the adaptive behavior control system uses the emotion detection module 279 to obtain the determined emotion classification (e.g., positive, neutral, negative) after performing an action of the first action type, and stores the determined emotion classification in association with the performed action (e.g., stored in the storage medium 205).

[0213] machine

[0214] The systems and methods of some embodiments and variations thereof may be at least partially embodied and / or implemented as a machine configured to receive a computer-readable medium storing computer-readable instructions. These instructions are preferably executed by a computer-executable component. The computer-readable medium may be stored on any suitable computer-readable medium, such as RAM, ROM, flash memory, EEPROM, optical device (CD or DVD), hard drive, floppy disk drive, or any suitable device. The computer-executable component is preferably a general-purpose processor or a special-purpose processor, but any suitable dedicated hardware or hardware / firmware combination device may alternatively or additionally execute these instructions.

[0215] in conclusion

[0216] As those skilled in the art will recognize from the foregoing detailed description and from the accompanying drawings and claims, modifications and variations may be made to the embodiments disclosed herein without departing from the scope as defined in the claims.

Claims

1. A method for human-computer interaction, comprising: The adaptive behavior control system of the robot detects a plurality of interacting participants by using at least one sensor; The adaptive behavior control system stores at least one computer behavior strategy for each detected interacting participant; The adaptive behavior control system controls the interaction subsystem to perform an action of a first action type for each detected interaction participant according to a computer behavior strategy selected for the interaction participant; For each detected interactive participant, the adaptive behavior control system detects a human response of the interactive participant to the performance of each action of the first action type based on the data received from the human response detection subsystem; For each detected human interaction participant, the adaptive behavior control system stores information indicative of each detected human response in association with information identifying an associated action; For each detected human interaction participant, the adaptive behavior control system updates the selected computer behavior strategy for the first action type of the interaction participant if the stored information indicating the detected human response to the first action type satisfies an update condition.

2. The method of claim 1, wherein the at least one sensor is at least one of: a sensor of a thermal detection subsystem, a sensor of a video capture subsystem, a sensor of an audio capture subsystem, a touch sensor, a piezoelectric pressure sensor, a capacitive touch sensor, a resistive touch sensor, a blood pressure sensor, a heart rate sensor, or a biometric sensor.

3. The method according to claim 1, in, The adaptive behavior control system stores information indicative of each detected human response in association with a corresponding computer behavior strategy, and The method further includes: in response to a reaction information request from an external system via an application programming interface of the adaptive behavior control system, the reaction information request specifying a computer behavior strategy, the adaptive behavior control system providing stored information indicating each detected human reaction for the specified computer behavior strategy to the external system as a response to the reaction information request.

4. The method of claim 1, further comprising: The adaptive behavior control system selects a computer behavior strategy for each detected interacting participant based on at least one of the following: an identity of the interacting participant, a current interaction context determined for the interacting participant, an action type, a current treatment goal, or a current therapy type determined for the interacting participant.

5. The method of claim 1, wherein at least one action for the first action type comprises performing a facial expression by controlling a robot head assembly of the robot.

6. The method of claim 5, wherein the robotic head assembly includes at least one mechanical facial feature, and the robotic head assembly is configured to control movement of the at least one mechanical facial feature to perform at least one facial expression.

7. The method of claim 5, wherein the robot head assembly includes at least one display device, and the robot head assembly is configured to control a display of the at least one display device to perform at least one facial expression.

8. The method of claim 5, wherein the robot head assembly includes at least one lighting subsystem, and the robot head assembly is configured to control lighting of the at least one lighting display system to perform at least one facial expression.

9. The method of claim 5, wherein the robot head assembly includes at least one LED ring, and the robot head assembly is configured to control at least one LED in the LED ring to perform at least one facial expression.

10. The method of claim 1, wherein at least one action for the first action type comprises controlling the robot head assembly by controlling an audio output subsystem of the robot head assembly to cause the robot head assembly to output audio in a specific voice.

11. The method of claim 1, wherein at least one action for the first action type comprises controlling the robot head assembly to output audible speech by controlling an audio output subsystem of the robot head assembly.

12. The method of claim 1, wherein at least one action for the first action type comprises controlling at least a first motorized appendage of the robot to perform movement according to at least one human interaction gesture.

13. The method of claim 1, wherein at least one human interaction gesture comprises a human-recognizable hand gesture performed by at least a first motorized appendage of the robot.

14. The method of claim 1, wherein at least one human interaction gesture comprises a human-recognizable arm gesture performed by at least a first motorized appendage of the robot.

15. The method of claim 1, wherein the adaptive behavior control system of the human-computer interaction system detects a plurality of interaction participants by using at least one sensor, comprising: A position of each detected interacting participant is detected, and a three-dimensional world model of the detected positions of the detected interacting participants is generated.

16. The method of claim 1, further comprising: The adaptive behavior control system selects a computer behavior strategy for at least one detected interacting participant based on current treatment goals.

17. The method of claim 1, further comprising: For at least one detected interacting participant, the adaptive behavior control system determines a current type of therapy being used during interaction with the participant and selects a computer behavior strategy based on the determined current type of therapy.

18. The method of claim 1, further comprising: For at least one detected interactive participant, the adaptive behavior control system selects a computer behavior strategy based on the identity of the interactive participant.

19. The method of claim 1, further comprising: For at least one detected interaction participant, the adaptive behavior control system determines a current interaction context of the participant and selects a computer behavior strategy based on the determined current interaction context.

20. The method of claim 1, wherein the adaptive behavior control system includes a computer behavior strategy for each of a plurality of therapy types for each of a plurality of interacting participants.

21. A system for human-computer interaction, comprising: Robot body components; a robotic head assembly mechanically attached to the body assembly via a motor assembly; The motor assembly is configured to rotate the robot head assembly about a vertical axis and a horizontal axis; at least one first motorized accessory configured to perform movement in accordance with at least one human interaction gesture; an adaptive behavior control system communicatively coupled to at least one of the motor assembly, the robotic head assembly, or the at least one first motorized attachment; a human response detection subsystem communicatively coupled to the adaptive behavior control system; as well as an interactive subsystem communicatively coupled to the adaptive behavior control system, Wherein, the adaptive behavior control system is constructed as follows: By detecting a plurality of interacting participants using at least one sensor, storing at least one computer behavior strategy for each detected interacting participant, controlling at least one of the interaction subsystem, the motor assembly, the at least one motorized appendage, the robot body assembly, and the robot head assembly to perform an action of a first action type for each detected interaction participant according to a computer behavior strategy selected for the interaction participant, for each detected interactive participant, detecting a human response of the interactive participant to each action of the first action type according to the data received from the human response detection subsystem, storing, for each detected human interaction participant, information indicative of each detected human response in association with information identifying an associated action, and For each detected human interaction participant, if the stored information indicating the detected human reaction to the first action type meets the update condition, the computer behavior strategy of the interaction participant for the first action type is updated.

22. A method for human-computer interaction, comprising: The adaptive behavior control system controls the interaction subsystem to perform a plurality of actions for a first action type according to the computer behavior policy, each action being a different alternative action for the action type; the adaptive behavior control system detecting a human response to the performance of each action of the first action type based on data received from the human response detection subsystem; The adaptive behavior control system stores information indicative of each detected human response in association with information identifying an associated action; as well as The adaptive behavior control system updates the computer behavior policy for the first action type in a condition that the stored information indicating the detected human response to the first action type satisfies an update condition.

23. The method of claim 22, in, The adaptive behavior control system stores information indicative of each detected human response in association with the computer behavior strategy, and The method further includes: in response to a reaction information request from an external system via an application programming interface of the adaptive behavior control system, the reaction information request specifying the computer behavior strategy, the adaptive behavior control system providing stored information indicating each detected human reaction to the computer behavior strategy to the external system as a response to the reaction information request.

24. The method of claim 22, Further including: In response to a policy information request from an external system via an application programming interface of the adaptive behavior control system, the policy information request specifying the computer behavior policy, the adaptive behavior control system provides an updated computer behavior policy to the external system as a response to the policy information request.

25. The method of claim 22, further comprising: The adaptive behavior control system detects the interacting participants by using at least one sensor; The adaptive behavior control system selects a computer behavior strategy based on a detected mood of a detected interacting participant.

26. The method of claim 22, further comprising: The adaptive behavior control system selects a computer behavior strategy based on the detected position.

27. The method of claim 22, further comprising: The adaptive behavior control system detects the interacting participants by using at least one sensor; as well as The adaptive behavior control system selects a computer behavior strategy based on detected identification of interacting participants.

28. The method of claim 27, wherein the at least one sensor is at least one of: a sensor of a thermal detection subsystem, a sensor of a video capture subsystem, a sensor of an audio capture subsystem, a touch sensor, a piezoelectric pressure sensor, a capacitive touch sensor, a resistive touch sensor, a blood pressure sensor, a heart rate sensor, or a biometric sensor.

29. The method of claim 27, wherein the adaptive behavior control system stores a plurality of computer behavior strategies for detected interacting participants.

Images