Digital character interacting with a customer in a physical domain

Abstract

Systems and methods for controlling a performance of a digital character portrayed at a display device are disclosed. According to at least one embodiment, a method for controlling a performance of a digital character portrayed at a display device includes determining a presence of a person located in a physical environment, and in response to determining the presence of the person, facilitating control of the performance of the digital character portrayed at the display device by a human operator, by an artificial intelligence (AI) game engine, or by a combination thereof.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Application No. 15 / 931,377 filed May 13, 2020 and U.S. Provisional Patent Application No. 62 / 860,188 filed June 11, 2019, the contents of all of which are hereby incorporated by reference in their entirety. BACKGROUND

[0003] Amusement parks can include various features to provide unique experiences to patrons. For example, amusement parks can have various rides and shows for patrons to enjoy. In addition, amusement parks can have show effects and props that can create an ideal environment or atmosphere for patrons. Such features can include entertainment figures (e.g., animated characters, animated figures) that can interact with patrons. For example, an entertainment figure can speak, wave, walk, or perform any other suitable action. SUMMARY

[0004] With respect to various embodiments disclosed herein, techniques are presented for controlling a performance of a digital character portrayed in a digital animation. According to various embodiments, the performance is controlled to facilitate an interaction between the digital character and a live patron.

[0005] According to at least one embodiment, a method for controlling a performance of a digital character portrayed at a display device is disclosed. The method includes determining a presence of a person located in a physical environment and, in response to determining the presence of the person, facilitating control of the performance of the digital character portrayed at the display device by a human operator, by an artificial intelligence (AI) game engine, or by a combination thereof.

[0006] According to at least one embodiment, an apparatus for controlling a performance of a digital character portrayed at a display device includes a network communication unit configured to transmit and receive data and one or more controllers. The one or more controllers are configured to determine a presence of a person located in a physical environment and, in response to determining the presence of the person, facilitate control of the performance of the digital character portrayed at the display device by a human operator, by an AI game engine, or by a combination thereof.

[0007] According to at least one embodiment, a machine-readable non-transitory medium has stored thereon machine-executable instructions for controlling a performance of a digital character portrayed at a display device. The instructions include determining a presence of a person located in a physical environment and, in response to determining the presence of the person, facilitating control of the performance of the digital character portrayed at the display device by a human operator, by an AI game engine, or by a combination thereof. BRIEF DESCRIPTION OF DRAWINGS

[0008] The foregoing aspects and features of the present disclosure will become more apparent with the following description of embodiments in conjunction with the accompanying drawings.

[0009] Figure 1 An example of a system for controlling a digital character to interact with a customer is described.

[0010] Figure 2A And Figure 2B An example of live motion capture for controlling a digital character is described.

[0011] Figure 3 An example of a rigid body in accordance with at least one embodiment is described.

[0012] Figure 4 A 3D view of a capture area is described.

[0013] Figure 5 An example delivery of a physical item for retrieval by a customer is described.

[0014] Figure 6 A flowchart of a method of controlling a performance of a digital character depicted at a display device in accordance with at least one embodiment is described.

[0015] Figure 7 An illustration of a computing environment in accordance with at least one embodiment is described.

[0016] Figure 8 A block diagram of a device in accordance with at least one embodiment is described. DETAILED DESCRIPTION

[0017] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration specific embodiments in which the application can be practiced. Those skilled in the art will realize that other embodiments can be practiced without departing from the scope of the present application. The following detailed description is therefore not to be taken in a limiting sense.

[0018] Embodiments disclosed herein relate to a system for controlling a digital character to interact with a customer. For example, the interaction can include delivering a physical product for retrieval by a customer. Exemplary embodiments include a system that visually resembles a food cart and outputs food for a customer, a system that sells movie tickets to a customer, and a system that facilitates the conduct of a rope-pulling contest between a digital character and a customer.

[0019] Exemplary embodiments will be described with reference to the drawings. Figure 1 Exemplary embodiments will be described with reference to the drawings.

[0020] Figure 1 A system 102 for controlling a digital character to interact with a customer is illustrated in accordance with at least one embodiment. The system 102 is located (at least partially) in an environment 100. Also present in the environment 100 is one or more customers 110. The system 102 includes a display device 104 and an output channel 108. The display device 104 is positioned in clear view of the customer 110. As will be explained in greater detail later, one or more items can be delivered via the output channel 108 for retrieval by the customer 110.

[0021] Moreover, the system 102 includes at least one video capture device (e.g., a camera) and at least one audio capture device (e.g., a microphone). The video capture device and the audio capture device can be positioned to capture actions (or lack thereof) occurring in the environment 100, particularly actions (or lack thereof) by the customer 110. Examples of such actions include movements or motions directed toward or away from the system 102 (e.g., toward or away from the display 104) by the customer 110, motions or gestures by the customer 110, and facial expressions and reactions by the customer 110. As will be described in greater detail below, this capture facilitates detecting the presence of the customer 110 in the environment 100 and / or interacting with the customer 110 (e.g., by one or more digital characters depicted at the display device 104).

[0022] As noted previously, the display device 104 of the system 102 is positioned in clear view of the customer 110. The display device 104 displays a video (e.g., a digital video). As will be described in greater detail below, the displayed video can include interactive elements and / or features.

[0023] For purposes of description, events depicted as occurring in the video displayed at the display device 104 will be referred to as events occurring in the digital realm. Separately, other events (e.g., events occurring in the environment 100) will be referred to as events occurring in the physical realm. From the perspective of the customer 110, the display device 104 can be perceived as a "window" into the physical space defined by the system 102, where events occurring in the digital realm are perceived as occurring in the physical space defined by one or more elements of the system 102.

[0024] With continued reference to Figure 1 , the digital character 106 is depicted in the video displayed at the display device 104. By way of example, the digital character 106 can be authored via live motion capture of a human actor.

[0025] For example, with reference to Figure 2AThe digital character 106 is authored via live motion capture of a human operator 202 located in an environment 200a. The environment 200a can be located proximate to the environment 100 or at a location remote from the environment 100. If the environment 200a is located proximate to the environment 100, the environment 200a can be configured such that the human operator is hidden from view of the customers 110 present in the environment 100.

[0026] In the environment 200a, a display device 204 is provided. The display device 204 outputs video and audio captured by the video and audio capture devices described previously with reference to the system 102. Thus, the human operator 202 can be informed in real-time of actions (or lack thereof) occurring in the environment 100. In this manner, the human operator 202 can effectively interact with the customers 110 present in the physical realm via the digital character 106 portrayed in the digital realm.

[0027] For example, the human operator 202 can ask the customers 110, "Does anyone want a cookie? Thus, via live motion capture, the digital character 106 is portrayed in the digital realm as asking, "Does anyone want a cookie? The customers 110 viewing the digital video displayed at the display device 104 perceive such actions occurring in the digital realm.

[0028] The customers 110 can respond accordingly. For example, one customer 110 can raise his / her hand. As another example, another customer 110 can shout, "Yes!"

[0029] By monitoring the display device 204, the human operator 202 is able to observe such actions made by the customers. For example, by continuing to monitor the camera footage, the human operator 202 is able to (1) change or shift his / her gaze such that the digital character 106 looks directly into the eyes of the customers 110 (e.g., if the customers 110 remain relatively still), and (2) reach toward the customers 110 (e.g., if the customers 110 reach or point toward a particular portion of the display device 104). In a similar manner, other interactions between the digital character 106 and the customers 110 can include shaking hands with the customers 110, hugging the customers 110, being physically affected by a push from a customer, and the like.

[0030] In at least one embodiment, bridging of the digital realm and the physical realm can be achieved by configuring the digital character to cause a limb to extend out to grasp the hand of the customer 110, or to present an item to the customer 110. By way of example, this can be achieved by controlling a physically operated puppet limb, operated in the physical realm by a human puppet operator hidden from view of the customer 110. The physically operated puppet limb can be controlled to extend out towards the customer 110 and to directly interact with the customer 110 in the physical realm. In this case, the physically operated puppet limb can be an animatronic arm and hand that moves realistically based on real-time control by the puppet operator. According to another embodiment, a digitally operated puppet limb can be controlled to extend out and directly interact with the customer 110. For example, the digitally operated puppet limb can be controlled via digital information from a performer (e.g., human operator 202) who is creating the character experience. In this case, the performer can control the limb to extend or retract in real-time by manually operating an on-screen interface, a keyboard trigger, a button, or a joystick input device. Real-time data from the performer will operate mechanical components of the hand, and cause the arm to move about in a controlled, robotically legal manner. According to another embodiment involving a digitally operated puppet limb, the performer can simply initiate a handshake by extending his / her arm out to a known region of the capture volume. Extension and retraction of the puppet limb, as well as other movements of the puppet limb, will be resolved by the performer, with the results also targeted to the animatronic puppet limb in a controlled, robotically legal manner.

[0031] In response to seeing and / or hearing the reaction by the customer 110, the human operator 202 can move within the environment 200a as if he is retrieving the box of cookies for the customer 110. Then, with reference to Figure 2B , the human operator 202 can move within the environment 200b as if he is delivering the retrieved box to one or more waiting customers 110.

[0032] The movement of the human operator 202 is replicated in the digital realm by the digital character 106. For example, in the digital realm, the digital character 106 retrieves the box of cookies and then moves in a direction that leads towards the output passageway 108.

[0033] In the foregoing, the movement of the human operator 202 can be monitored using a motion capture system. A motion capture system according to one or more embodiments will now be described in greater detail.

[0034] Optical systems (or optical motion capture systems) utilize data captured from image sensors to triangulate a subject's 3D position between two or more cameras calibrated to provide overlapping projections. Data can be captured using markers attached to an actor (e.g., human operator 202). Tracking a larger number of performers or expanding the capture area can be achieved by increasing the number of cameras.

[0035] With respect to optical systems, occlusions occur when model marker points are not present in the point cloud, for example, when the number of observed points is lower than the number of model marker points. Such events can occur for any of a variety of reasons, such as, for example, when one or more opaque objects intervene between the markers and the cameras or when the markers are beyond the camera field of view (FOV).

[0036] According to at least one embodiment, a drift correction system includes (e.g., in a hybrid fashion) one or more aspects of an optical motion capture system along with one or more aspects of an inertial motion capture system. Such a system can account for occlusion-free motion capture that is correct in position in a capture area (such as, for example, environment 200a) and will not drift over time.

[0037] Aspects of the inertial motion capture system include the use of an inertial body capture suit as known in the art. The inertial body capture suit can be of the type described previously with respect to Figure 2A For example, such an inertial body capture suit measures the joint angles of the human operator 202 and estimates his / her position from an initial starting point, for example, by counting steps. In this regard, errors can accumulate over time as the inertial motion capture system produces approximate results based on the magnetometers included in the suit. For example, the inertial motion capture system can approximate a true north position, but the error can be a result of the impact of natural magnetic interference on the accuracy of the magnetometers.

[0038] To help correct for phenomena such as occlusions and artifacts (such as those caused by magnetic interference), the system can use aspects of both the inertial motion capture system and the optical system. According to at least one embodiment, aspects of the optical system will be simpler than those of an optical system that can be typically used for full-body motion capture. For example, the optical system can be used to focus on the waist of the actor. According to at least one embodiment, the optical system includes the use of a rigid device (or rigid body) that can be worn by the human operator 202. According to at least one embodiment, data from the inertial motion capture system and data from the optical system are combined (or fused together) and used to drive the digital character (e.g., digital character 106).

[0039] By way of example, data regarding rotation and height of center of mass (see, e.g., Y-axis) can be received from an inertial body capture suit of an inertial motion capture system, and data regarding a horizontal plane (see, e.g., a plane defined by X-axis and Z-axis) can be received from an optical system. Figure 2A Figure 2A

[0040] As previously described, the optical motion capture system can include a rigid body. The rigid body can be stably attachable to a human operator 202 being drift corrected. According to at least one embodiment, the rigid body includes markers placed thereon. For example, the markers can be retro-reflective markers. Such markers can be placed on the rigid body in a unique pattern, such that the rigid body will have a unique identity (e.g., configuration or arrangement of markers) that is identifiable or identifiable by the optical motion capture system.

[0041] Figure 3 An example of a rigid body 502 according to at least one embodiment is illustrated. As illustrated, the rigid body 502 can take the form of a belt. The belt is configured to be worn around the waist of a person (e.g., human operator 202). The rigid body 502 is relatively small in volume, such that when the wearer of the rigid body moves, the rigid body moves with the wearer's hips, and does not move independently with respect to the wearer. The rigid body 502 can be made of a relatively strong material, and can have a uniform width (e.g., 1 or 2 inches). Markers 506 can be positioned along the perimeter of the rigid body 502. As previously described, the markers 506 can be positioned on the rigid body in a unique pattern. For example, in the example illustrated in Figure 3

[0042] Alternatively, instead of positioning markers 506 directly on the belt, the markers 506 can be positioned on the belt (indirectly) via one or more rigid bodies positioned (directly) on the belt. In this alternative example, the markers 506 can be placed directly on the rigid bodies, which in turn are placed on the belt. For example, five markers 506 can be placed on each of a total of five rigid bodies, which in turn are placed on the belt. The placement of the rigid bodies on the belt (and the placement of the markers on the rigid bodies) can be in a unique pattern.

[0043] It is understood that as few as one optical marker can be sufficient for purposes of achieving an acceptable level of drift correction. However, according to at least one embodiment, two or more optical markers are utilized to achieve a higher level of robustness with respect to occlusions.

[0044] ​​​With respect to the optical system, various embodiments will now be described with reference to a "non-intrinsic" system. According to such embodiments, an array of two or more cameras is mounted around the perimeter of a capture area (e.g., a room). As a rigid body (as worn by a human operator) moves around the capture area, the cameras capture views of the capture area and views of one or more markers (e.g., markers 506 on rigid body 502). The static cameras observe the moving markers and accordingly provide data (e.g., accurate rotation and translation) about the moving markers that is solved in computer animation software into a rigid object (e.g., a computer-generated skeleton).

[0045] Figure 4 A 3D view of a capture area (e.g., a room) 400 is illustrated. In at least one embodiment, one or more cameras are mounted on each tower placed in capture area 400. For example, as illustrated in FIG. 4A, four towers 402 are placed in capture area 400, one at each corner. For illustration purposes, each tower 402 can be an eight-foot tall box truss comprising two-foot wide, one-inch thick aluminum plates. Each of towers 402 can have three cameras 406 mounted thereon. Within one tower 402, cameras 406 can be spaced apart from each other, e.g., at approximately two feet apart from each other along the height of the tower. Figure 4

[0046] Cameras 406 can be located on small aluminum plates that are inserted into the box truss. A three-way positioning head with a geared connection can also be located on the plate. Cameras 406 can be attached to the positioning head, and the head can be geared to set in precise rotation, and once the aiming of the head is complete, the head is locked into place. As noted previously, towers 402 are positioned at the corners of capture area 400.

[0047] Each tower 402 generates its own data. During calibration of cameras 406, a static object (e.g., rigid body 502) can be placed in the middle of capture area 400. Cameras 406 can be calibrated simultaneously by configuring the optical system to triangulate the static object. Thus, the position of the static object within the room can be determined. As noted previously, data about a horizontal plane (see, e.g., the plane defined by the X and Z axes of Figure 2A Such data can be combined with data about rotation and height off the ground (see, e.g., the Y axis of Figure 2A ​combination (or fusion) of data from the optical system and data from the inertial system. The combination of data can be used to generate animations of the digital avatar. For example, rotational data received from the inertial system can be used to animate the skeleton of the digital avatar, and data received from the optical system can be used to correct the position of the hips of the digital avatar.

[0048] With respect to the optical system, various embodiments have been described with reference to a "extrinsic" system. According to at least another embodiment, an "intrinsic" system is utilized. In such a system, a camera is worn by a human operator to monitor markers located in a capture area (e.g., located on the ceiling, floor, and / or walls of a room) according to unique patterns. As the human operator moves about the room, the camera moves with the human operator and monitors the static markers so that an accurate estimate of the rotation and position of the camera itself in the room relative to the fixed markers located on the walls, floor, or ceiling of the room can be produced.

[0049] As previously described, markers placed on the rigid body and / or belt can be retro-reflective markers. According to at least one other embodiment, the markers can be light emitting devices (LEDs) that emit light at a wavelength that can be detected by the optical motion capture system.

[0050] As previously described, a drift correction system is created by combining (or mixing) aspects of the optical motion capture system with aspects of the inertial motion capture system. According to at least one other embodiment, a drift correction system is created by combining (or mixing) aspects of a magnetic motion capture system with aspects of the inertial motion capture system. Such an embodiment can operate in a similar manner as previously described with respect to embodiments that employ an optical motion capture system. For example, magnetic "markers" can be placed on the human operator. At either end of the room, a magnetic detection system can be placed. The magnetic detection system can emit a magnetic field and detect magnetic interference with that field caused by the magnetic markers. The detected interference can be used to determine the position of the magnetic markers.

[0051] In at least one embodiment, technology employing Wi-Fi technology can be utilized. Such an embodiment can operate in a similar manner as previously described with respect to embodiments that employ a magnetic motion capture system. For example, Wi-Fi antenna(s) can be placed around the room and a Wi-Fi transmitter can be placed on the human operator. As the human operator moves within the room, the Wi-Fi transmitter can move closer to one antenna or further from another. Characteristics of the signal received at the antenna(s) can be used to determine the position of the transmitter within the room.

[0052] In at least one embodiment, technology employing acoustic technology can be utilized. Such embodiments can operate in a manner similar to that previously described with respect to embodiments employing optical motion capture systems. In this regard, one of at least two configurations can be used. For example, in one configuration, microphones are placed on the human operator and ultrasonic transmitters are placed around the room. As another example, in another configuration, transmitters are placed on the human actor and microphones are placed around the room. In embodiments employing acoustic technology, time-of-flight (ToF) can be used to determine the location of whichever object is placed on the human operator (e.g., microphones or transmitters). In a manner often working with echolocation, the transmitters emit ultrasonic ping sounds and the microphones (or microphone array) detect the ping and calculate the distance traveled by the ping sound based on the delay (time-of-flight) from the transmitter to the microphone.

[0053] In at least one embodiment, one or more depth cameras can be utilized to determine the location of the human operator in the room. For example, a depth camera can project an infrared (IR) grid and then detect how the human operator is warping the grid to determine where the operator is located and movements made by the operator. As another example, an array of IR LEDs can flash in sequence and not flash to perform ToF calculations as light is emitted from the LEDs, reflects off the human operator, and then returns to the depth camera for depth calculations.

[0054] In at least one embodiment, markers need not be employed. For example, in a simpler system, one or more video cameras can be used to track specific features of the human operator or specific features located on a rigid body to determine the location of the human operator in the room.

[0055] Reference is made to the environment 300 of Figure 5 In the physical domain, a box 302 containing cookies is delivered at the output channel 108. For example, the system 102 can include a dispenser that dispenses items when controlled (e.g., by the human operator 202).

[0056] As previously referenced with respect to the environment 300 of Figure 1As previously described, the system 102 includes at least one video capture device (e.g., a camera) and at least one audio capture device (e.g., a microphone). It is understood that the camera and microphone can exist in the virtual world and be experienced by the customer in virtual reality. For example, the customer can control a digital avatar that exists in the digital realm through an input device such as a joystick, mouse, or virtual reality (VR) or motion capture input device. The interaction takes place within the digital realm, in addition to the sale / delivery of the item (e.g., the box 302) that still takes place in the physical realm. For example, the customer plays a game that interacts with a game character that corresponds to the customer, rather than directly with the customer. In this case, the customer enters the digital realm as a digital avatar and experiences the digital realm, the movements of the digital avatar being controlled by the customer based on input provided in the physical realm. As such, the avatar can be controlled to move in the digital realm in a certain way (e.g., walk, jump, fly, teleport). When the presence of the digital avatar is detected in the digital realm, interaction with the digital avatar is initiated similarly to the way in which interaction with the customer is initiated when the presence of the customer in the physical realm (e.g., the environment 100) is detected. Once the customer has triggered a change to take place in the physical realm (e.g., the delivery of an item), the change is implemented in the physical realm.

[0057] As previously described with reference to Figure 1 , Figure 2A , Figure 2B , Figure 3 and Figure 4 , a digital character (e.g., the digital character 106) can be authored via live motion capture of a human operator (e.g., the human operator 202). Live motion capture driven performance can be applied to a particular portion (or aspect) of the digital character (or an object such as the box 302 in Figure 5 According to one or more embodiments, the human operator uses a device such as a controller, keyboard, joystick, mouse, foot pedal, microphone, etc. to drive the performance of the digital character using real-time musical instrument digital interface (MIDI) input and / or digital puppetry manipulation. In such embodiments, live motion capture can not be exclusively used. For example, motion capture can be used to drive the performance of the face of the digital character, and the remainder of the performance is driven using triggered input devices. According to one or more other embodiments, the digital character 106 can be authored entirely via live motion capture of a human operator.

[0058] According to one or more other embodiments, the decision making and performance of the digital character is authored entirely by an artificial intelligence (AI) character that resides in the game engine.

[0059] According to one or more other embodiments, decision making and performance for a digital character can be achieved using a combination of AI game engine driven characters and human motion capture. By way of example, during certain time(s) (e.g., when the system is waiting for an exchange by a customer), a digital character can be driven at least in part by AI (e.g., by playing an environmental motion loop or other AI based character animation). At additional time(s), a transition from being driven by AI to being driven by motion capture can occur. For example, when the system detects the presence of a customer 110, the performance (e.g., animation control) of the digital character can transition to a human operator who is able to interact with the customer in a more humanized manner. In the event of such a transition, the system 102 can signal the human operator that he / she is being given at least partial control over the performance of the digital character. For example, the system 102 can provide audio and / or visual indication(s) to inform the human operator that he / she is being given control over the vocal performance of the digital character, the facial performance of the digital character, or the entire performance of the digital character.

[0060] In a similar manner, a transition from being driven by motion capture to being driven by AI game engine technology can occur. Decisions made by such AI technology can be based on analysis of data captured by video capture devices (e.g., cameras) and / or audio capture devices (e.g., microphones) of the system 102. For example, analysis of data captured by cameras can be used not only to identify the locations of various customers 110, but also to identify certain characteristics of particular customers, such as a blue colored garment (e.g., shirt) worn by one customer 110 or the blonde hair of another customer 110. As such, even when the performance of the digital character is being driven by AI game engine technology (rather than by a human operator), the digital character can still interact with customers 110 to some extent. For example, the eyes of the digital character can be driven to move such that the "eyes" of the digital character appear to follow movements made by customers 110. Also, for example, the digital character can be driven to wave to a particular customer 110 and / or express a greeting that is somewhat customized or personalized to the customer 110 wearing the blue colored garment or the customer 110 with blonde hair.

[0061] Similarly, analysis of data captured by the microphone can be used, at least to some extent, in interfacing with the customer 110. For example, where such analysis detects a sound as having certain characteristics (e.g., a sound that exceeds a certain threshold audio level), the performance of the digital character can be selectively driven by a particular clip of (pre-prepared) animation. In such a clip, the eyes of the digital character can move such that the gaze of the digital character appears to turn upward and outward, as if searching for the source of the sound heard. As another example, waveform analysis can be performed on speech audio captured by the microphone. Such analysis can be used in the identification of the mood or emotional state of the customer 110 who has spoken. In this case, the mood in which the digital character delivers an audio greeting to the customer 110 can be selected to match or reflect the mood identified by the waveform analysis. For example, if the mood of the customer 110 is identified as happy or cheerful, the digital character can be controlled to speak to the customer 110 in a happy or cheerful manner.

[0062] Accordingly, the AI game engine technology can be used to effectively execute a certain branch in the decision tree in response to some provocation (e.g., detection of a sound having certain characteristics) (e.g., using a selected animation clip to drive the performance of the digital character). This can improve the likelihood that the digital character appears realistic and capable of reacting to real-world events occurring in the physical domain (e.g., the physical domain of the environment 100). This can be superior to a performance of the digital character in which the character merely repeats certain actions while appearing oblivious to events occurring in the physical domain.

[0063] As such, the performance of the digital character need not always be driven by a human operator. For example, the performance of the digital character can be driven by the AI game engine technology during some (or most) of the time, and by a human operator during selected times (e.g., one or more key times). As another example, the performance of the digital character can be driven by the AI game engine technology during most of the day (e.g., 11 hours total, more than 12 hours of operation per day), and the control of the digital character can be assigned to a human operator during a selected period (e.g., selected hours during operation of such a day). This period can be selected so as to provide more personalized interaction with the customer 110 during this period. At the end of such a period, the human operator can return control of the performance of the digital character to the AI game engine technology. From the perspective of the customer 110, this return can be selected to occur once so as to provide a more seamless transition. The human operator can return control of the performance of the digital character at such times by, for example, manually operating an on-screen interface, a keyboard trigger, a button, or a joystick input device.

[0064] In at least one embodiment, the performance of the digital characters can sometimes be driven simultaneously by both AI game engine technology and human operators. For example, when a digital character is at least partially driven by AI in an interaction with a customer 110, a human operator can wish to augment the performance of the digital character. In this case, the human operator can control the AI-driven performance in one or more ways. For example, if the human operator recognizes the mood of the customer 110 as more than just happy or pleased, but specifically happy or pleased, the human operator can control the digital character to speak to the customer 110 in a similarly heightened manner. Such control can be performed by, for example, manually operating an on-screen interface, a keyboard trigger, a button, or a joystick input device. In this way, the performance of the digital character can be (temporarily) guided at least in part by the human operator.

[0065] According to embodiments to be described in greater detail later, a plurality of digital characters can be depicted at the display device 104. Each of such characters can be controlled via a respective different combination of AI game engine-driven characters and human motion-captured characters, as previously described.

[0066] As previously described with reference to various embodiments, actions (or lack thereof) occurring in the environment 100, for example, by the camera and microphone, specifically, by the customer 110, are captured. When the presence of the customer 110 in the environment 100 (e.g., in one or more particular areas within the environment 100) is detected, an interaction with the customer is initiated. It is understood that detection can additionally and / or alternatively be performed using other devices. Such other devices include pressure pads, depth cameras, sound wave range detection devices, light beam devices, and / or heat or sound detection devices.

[0067] Devices can not only detect the presence of a customer, but also other parameters such as the head height and body position of the customer. For example, a video camera, a depth sensor, or an interrupted light or sound beam can be used to detect a customer to determine the body position and height of the customer. Knowing such information (e.g., in relation to the position of the display device 104) can allow for an angular correction to the head or eye angle of the digital character 106 so that the line of sight of the digital character is more closely aligned with the line of sight of one or more of the customers. By more accurately detecting the head height of the customer, such devices are able to effectively "zero in" the line of sight of the digital character.

[0068] As also described previously with reference to various embodiments, the interaction includes one or more interactions between the digital character and the customers in both the digital realm and the physical realm. For example, in the digital realm, the digital character can perform steps to prepare or procure an item (e.g., a box of cookies 302) for the customers. In the physical realm, the item is given (or sold) to the customers in real-time such that the item can be retrieved or collected by the customers (e.g., at the output channel 108). According to one or more further embodiments, the item is provided such that it can be retrieved by the customers at another location (e.g., somewhere outside of the environment 100). According to one or more further embodiments, the item is delivered to the customers (e.g., by physical mail) at a later date and / or time.

[0069] According to one or more other embodiments, the interaction between the digital character and the customers includes an interaction that causes (or brings about) a change in the physical realm of the environment 100. For example, the system 102 can include a pulley to facilitate performance of a game of tug-of-war between the digital character and the customers.

[0070] By way of example, the human operator 202 controlling the performance of the digital character asks, "Does anyone want to play a game of tug-of-war with me? Thus, in the digital realm, the digital character 106 asks the customers 110, "Does anyone want to play a game of tug-of-war with me? In the physical realm, the customers 110 can respond accordingly. For example, one customer 110 can raise his / her hand. As another example, another customer 110 can say loudly, "Yes!"

[0071] In response to seeing and / or hearing the reactions made by the customers 110, the human operator 202 can move within the environment 200a as if the human operator 202 is retrieving a rope with which to play the game. The human operator 202 can then move within the environment 200b as if the human operator 202 is moving the retrieved rope toward one or more of the waiting customers 110.

[0072] The actions of the human operator 202 are replicated in the digital realm by the digital character 106. For example, in the digital realm, the digital character 106 retrieves the rope and moves in a direction that directs toward the output channel 108.

[0073] In the physical realm, the free end of the rope extends out through the output channel 108. The other end of the rope can be coupled to a pulley system hidden from view of the customers 110. After one of the customers grabs the rope, the game of tug-of-war can begin.

[0074] According to one or more further embodiments, examples of other interactions that cause changes in the physical realm of the environment 100 include: an object in the physical realm being moved (e.g., knocked over) in response to an action (e.g., a stumble or fall) by a digital character in the digital realm; a light in the physical realm being controlled to turn on / off in response to an action by a digital character in the digital realm; and some other aspect in the physical realm changing in response to an action by a digital character in the digital realm.

[0075] In a similar manner, actions by the customer in the physical realm that are detected by the system 102 can result in changes in the digital realm. For example, a facial response or expression made by the customer, a body posture of the customer, other movements made by the customer, or a vocal reaction or sound made by the customer can be detected and cause a related event to occur in the digital realm.

[0076] Interactions between the physical realm and the digital realm are not limited to those involving the customer and the digital characters. For example, the presence of one or more specific objects in the physical realm that are detected can result in changes in the digital realm, and vice versa. For example, the changes can involve corresponding or related objects in the digital realm. By way of example, movement of an object by the customer in the physical realm, either intentionally or unintentionally, can be followed by movement of a corresponding or related object by a digital character in the digital realm.

[0077] Changes in the physical realm of the environment 100 can be implemented to increase the likelihood that the customer will engage in communication with one or more digital characters in the digital realm. If no customer communication is detected, and the performance of the digital characters is authored, at least in part, by AI, then the system 102 can enter a loop state in which the system 102 continuously (or periodically) monitors for detected communication. Alternatively (or in addition), the system 102 can begin displaying subsequent animations in order to encourage the customer to engage in communication with the digital character(s). Alternatively (or in addition), the system 102 can return to a wait state in which a particular set of animations is displayed until the presence of the customer is detected.

[0078] As previously described with reference to various embodiments, an object such as a cookie tin can be delivered (e.g., via output channel 108) for a customer to retrieve. In this case, the object can be delivered relatively quickly after a request from the customer is detected (e.g., the customer says "Yes!"). In other cases, delivery may occur after a certain delay. For example, a toy sword can be delivered to a customer. In the physical realm, a pre-made toy sword can be retrieved and placed at the receiving end of the output channel. However, in the digital realm, a digital character can prepare the toy sword in a blacksmith shop (e.g., from raw materials). In such a case, an animation loop (or sequence) can be displayed on a display device (e.g., display device 104) to show the preparation of the toy sword. Such a loop of display can continue until feedback indicating that the delivery of the toy sword will occur is received in the physical realm. When such feedback is received, the loop of display ends, and in the digital realm, the preparation of the toy sword is completed. Examples of such feedback will be described in more detail later.

[0079] According to another example, the object can be an item that requires some preparation time in the physical realm. For example, the object can be food requested (or ordered) by a customer (e.g., a bowl of noodle soup).

[0080] Before a request is received, the presence of a customer can be detected first. As previously described with reference to various embodiments, movement of one or more customers (e.g., in environment 100) can be detected. When one or more movements are detected (e.g., one or more specific movements, such as movement toward display device 104), an action is taken to request communication from the customer. For example, when system 102 is designed to visually resemble the exterior of a kitchen or food cart, a sequence (e.g., a game sequence) is initiated, causing display device 104 to display one or more digital characters preparing food. For example, in the digital realm, the digital characters are cutting vegetables, cooking noodles, etc.

[0081] The ability to control the depiction of one or more objects (such as a spoon or pot for cooking) in the digital realm in many ways. For example, the object may have a counterpart existing in the physical realm, such as an actor capable of being tracked via motion capture (e.g., Figure 2AA human operator (202) carries a spoon (e.g., made of plastic or foam). As another example, when an object exists within a game engine like an object in a video game, the object's depiction can be digitally controlled. In this case, the actor can interact with the object by making a grasping gesture and placing his / her hand near the object to alert the game engine that he / she wants to pick it up. The actor can also trigger the grasping gesture via a controller or button press. In another example, the actor can trigger the object to appear in his / her hand from nowhere by pressing a button or by reaching into a specific 'area'.

[0082] As previously referenced Figure 2B The description describes a digital character 106 retrieving a cookie box in the digital realm. The actor can achieve this retrieval by extending his hand upwards into the air, causing the digital character 106's hand to enter an invisible cube within the digital realm, which exists outside the frame (e.g., exactly outside the frame) relative to the display (e.g., display 104). Once the digital character 106's hand is in this area, the box is placed in his hand within the digital realm. When the actor lowers his arm in the physical realm, the box appears in the digital character 106's hand in the digital realm, making it appear as if the digital character 106 has extended beyond the screen and pulled the box from an invisible shelf.

[0083] According to another example, a combination of one or more items in the physical realm along with one or more items in the digital realm can be used. For example, such a combination can be used to achieve the effect of a digital character cutting a carrot in the digital realm. This combination might involve a physical counterpart (e.g., a physical prop) held by an actor and corresponding to a knife in the digital realm, and a carrot that exists only in the digital realm. The movement of the physical prop is tracked so that the knife appears in the digital character's hand and moves accordingly. When the cutting edge of the knife moves near the carrot in the digital realm, the carrot is depicted as being sliced. When the side of the knife is brought near the carrot slice, the slice can be depicted as moving from the edge of the cutting board and into the pot. This movement of the carrot slice can also be depicted as being directly caused by the digital character's hand (e.g., caused by the actor moving to sweep the carrot slice into the pot). However, for greater realism, it is understood that only the knife (not the digital character's hand) can cause the carrot to be sliced.

[0084] Triggered motion(s) can also be achieved in a similar manner. Such motions include, for example, martial arts moves that a typical unskilled person cannot readily perform. These motions can be triggered by button presses on the controller that blend into one or more pre-recorded motions from live motion, or these motions can be triggered by the actor performing a gesture (e.g., an alternative gesture) that, when recognized by the system, initiates the triggered motion. For example, the actor can kick his / her leg, and the system can recognize the kick as a trigger for a special kung fu jump-kick sequence. According to further examples, the triggered motion does not drive the performance of the digital character entirely. For example, only one or more portions of the digital character's body can be driven by the triggered motion. By way of example, when the triggered motion is performed by the digital character's torso, arms, and / or legs, the digital character's face can still be controlled by the actor (e.g., via motion capture).

[0085] Returning to the detection of the presence of the customer, a request is received from the detected customer. Receiving the request can involve using natural language processing (to receive and process a spoken request). Alternatively (or additionally), receiving the request can involve using an interface that is operable by the customer. For example, the customer can operate a button located on a touchscreen or fill out an instruction card that is inserted into the machine and read by the machine.

[0086] After the customer requests a specific item, preparation of the item begins in the physical realm. For example, a food preparer (e.g., a chef) located near the environment 100 begins preparing the requested item.

[0087] Simultaneously, in the digital realm, actions that depict the preparation of the item occur. Such actions can be customized based on the customer's request. For example, based on the customer's request that the soup noodles include additional carrots, the sequence performed can include a depiction of carrots flying through the kitchen of the digital realm.

[0088] As the preparation of the requested item that is being performed in the physical realm is ongoing, the animation displayed at the display device 104 can be continuously controlled. For example, during the preparation of the requested item, feedback can be received from the food preparer, e.g., indicating that more time is required before the preparation is complete. Such feedback can cause additional loops of a particular sequence to be displayed. Here, it is understood that the animation displayed at the display device 104 can be controlled by both AI game engine technology and human operators simultaneously, similarly to the ways in which the performance of a particular digital character can be controlled as previously described with reference to various embodiments.

[0089] Additionally, animation can be controlled as the requested item nears completion or is completed. For example, feedback indicating that the item is ready for output can be received from the food preparer. This feedback could originate from a pressure sensor that senses an object (e.g., the requested item) has been placed at a determined location. Therefore, the food preparer can place the requested item at or near the pressure sensor to provide feedback indicating that the item is ready for output. As another example, system 102 may include an interface operable by the food preparer. Thus, the food preparer can operate buttons located on a touchscreen to signal that the item is ready for output.

[0090] In the physical realm, the item can be placed at output channel 108 for the customer to retrieve.

[0091] Based on the described features, the timing of events occurring in the digital domain (e.g., preparing items by one or more digital characters as depicted in the animation shown) can be better aligned with the timing of events occurring in the physical domain (e.g., preparing items by a food preparer). For example, the timing in the two domains can be better consistent, such that when a bowl of soup disappears from view in the digital domain, it reappears at output channel 108 in the physical domain. According to one or more specific embodiments, the items presented at output channel 108 are selected to visually match the items depicted in the digital domain.

[0092] Figure 6 A flowchart illustrating a method 600 for controlling the performance of a digital character depicted on a display device according to at least one embodiment.

[0093] At box 602, according to a specific embodiment, the performance of the digital character can be controlled using AI game engine technology. For example, see reference... Figure 1 The performance of digital character 106 is controlled using AI game engine technology.

[0094] At box 604, the presence of a customer located in the physical environment is detected. For example, continue to refer to... Figure 1 The presence of client 110 located in physical environment 100 was detected.

[0095] According to another embodiment, the detection of the customer's presence is based on data received from at least a camera or microphone located in the physical environment.

[0096] According to another embodiment, detecting the presence of a customer may include autonomously determining characteristics of the customer's appearance or the customer's emotional state. For example, data received from a camera may be used to determine that customer 110 is wearing blue clothing. As another example, data received from a microphone may be used to detect and determine whether customer 110 is happy or sad.

[0097] At block 606, responsive to detecting the presence of the customer, control of the performance (or at least aspects thereof) of the digital character by a human operator, an AI game engine, or a combination thereof is facilitated. For example, with reference to Figure 2A , responsive to detecting the presence of the customer 110, control of the performance of the digital character 106 by the human operator 202 is facilitated.

[0098] According to further embodiments, facilitating control of the performance of the digital character can include providing at least one option selectable by the human operator. The selectable option is to control the digital character to speak to the customer in accordance with the determined characteristic or the determined emotional state. For example, if the human operator 202 discerns that the mood of the customer 110 is particularly happy or joyful, the human operator can control the digital character 106 to speak to the customer 110 in such a particularly happy or joyful manner.

[0099] According to further embodiments, facilitating control of the performance of the digital character can include receiving motion capture data corresponding to the human operator. By way of example, receiving the motion capture data can include receiving data from an optical motion capture system (e.g., a system that utilizes cameras 406 of Figure 4 ).

[0100] According to further embodiments, the data received from the optical motion capture system can include data corresponding to one or more optical markers located at a waist region of the human operator. For example, the data received from the optical motion capture system can include data corresponding to markers 506 located at a waist region of the human operator 202.

[0101] In further embodiments, facilitating control of at least part of the performance of the digital character by the human operator is such that the performance of the digital character is simultaneously driven by both the human operator and the AI game engine technology.

[0102] At block 608, a request is received from the customer in accordance with particular embodiments. For example, with reference to Figure 1 , the customer 110 can request that a cookie box be provided.

[0103] At block 610, the request can be serviced by causing a change in the physical environment in which the customer is located. By way of example, the request can be serviced by providing a physical object to be delivered for retrieval by the customer. For example, with reference to Figure 5 , a box 302 containing cookies is provided to be delivered for retrieval by the customer 110.

[0104] In selected embodiments, the features and aspects described herein can be implemented within a computing environment 700, as shown in Figure 7 As shown in FIG. 7, the computing environment 700 can include one or more computer servers 701. The servers 701 can be operatively coupled to one or more data storage devices 702 (e.g., databases, indexes, files, or other data structures). The servers 701 can be connected to a data communications network 703, including a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a telephone network, a satellite or wireless communication network, or some combination of these or similar networks.

[0105] One or more client devices 704, 705, 706, 707, 708, 709, 710 can communicate with the servers 701 and corresponding data storage devices 702 via the data communications network 703. Such client devices 704, 705, 706, 707, 708, 709, 710 can include, for example, one or more laptop computers 707, desktop computers 704, smartphones and mobile phones 705, tablet computers 706, televisions 708, motion capture sensor(s) 709, camera(s) 710, or combinations thereof. In operation, such client devices 704, 705, 706, 707, 708, 709, 710 can send data or instructions to and receive data or instructions from the servers 701 in response to user input received from user input devices or other input devices. In response, the servers 701 can serve data from the data storage devices 702, alter data within the data storage devices 702, add data to the data storage devices 702, and the like, or combinations thereof.

[0106] In selected embodiments, the servers 701 can transmit one or more media files to one or more of the client devices 704, 705, 706, 707, 708, 709, 710 via the data communications network 703, the media files including audio and / or video content, encoded data, generated data, and / or metadata from the data storage devices 702. The devices can output the audio and / or video content from the media files using a display screen, a projector, or other display output device. In certain embodiments, the system 700 configured in accordance with the features and aspects described herein can be configured to operate within or support a cloud computing environment. For example, some or all of the data storage devices 702 and servers 701 can be located in a cloud server.

[0107] Reference is made to Figure 8An exemplary computer 800 is provided for illustration. One or more of the devices 704, 705, 706, 707, 708 of the system 700 can be configured as such a computer 800 or include such a computer 800.

[0108] In selected embodiments, the computer 800 can include a bus 803 (or multiple buses) or other communication mechanism, a processor 801, a main memory 804, a read only memory (ROM) 805, one or more additional storage devices 806, and / or a communication interface 802, among other things or sub-combinations thereof. The embodiments described herein can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a selective combination thereof. In all embodiments, the various components described herein can be implemented as single components or alternatively as separate components in various combinations.

[0109] The bus 803 or other communication mechanism (including multiple busses or mechanisms) can support communication within the computer 800. The processor 801 can be connected to the bus 803 and can process information. In selected embodiments, the processor 801 can be a specialized or dedicated microprocessor configured to perform specific tasks according to the features and aspects described herein by executing machine-readable software code defining the particular tasks. The main memory 804 (e.g., random access memory (or RAM (or other dynamic storage device)) can be connected to the bus 803 and can store information and instructions to be executed by the processor 801. The main memory 804 can also store temporary variables or other intermediate information during execution of such instructions.

[0110] The ROM 805 or some other static storage device can be connected to the bus 803 and can store static information and instructions for the processor 801. The additional storage device 806 (e.g., a magnetic disk, optical disk, memory card, etc.) can be connected to the bus 803. The main memory 804, the ROM 805, and the additional storage device 806 can include non-transitory computer-readable media that stores information, instructions or some combination thereof (e.g., instructions that, when executed by the processor 801, cause the computer 800 to perform one or more operations of a method as described herein). The communication interface 802 can also be connected to the bus 803. The communication interface 802 can provide or support two-way data communication with one or more external devices (e.g., other devices contained within a computing environment).

[0111] In selected embodiments, computer 800 can be connected (e.g., via bus 803) to a display 807. Display 807 can utilize any suitable mechanism to convey information to a user of computer 800. For example, display 807 can include a liquid crystal display (LCD), a light-emitting diode (LED) display, a projector, or other display device or utilize the same to present information in a visual display to a user of computer 800. One or more input devices 808 (e.g., alphanumeric keyboard, mouse, microphone) can be connected to bus 803 to communicate information and commands to computer 800. In selected embodiments, one input device 808 can provide or support control of positioning of a cursor to allow for selection and execution of various objects, files, programs, etc. provided by computer 800 and displayed by display 807.

[0112] Computer 800 can be used to transmit, receive, decode, display, etc. one or more video files. In selected embodiments, such transmitting, receiving, decoding, and displaying can be in response to processor 801 executing one or more sequences of instructions contained in main memory 804. Such instructions can be read into main memory 804 from another non-transitory computer-readable medium, such as a storage device.

[0113] Execution of the sequences of instructions contained in main memory 804 can cause processor 801 to perform one or more of the routines or steps described herein. In selected embodiments, one or more processors in a multi-processing arrangement can also be employed to execute sequences of instructions contained in main memory 804. Alternatively, or in addition, firmware can be used instead of, or in addition to, software instructions for implementing routines or steps according to the features and aspects described herein. Thus, embodiments according to the features and aspects described herein can not be limited to any specific combination of hardware circuitry and software.

[0114] A non-transitory computer-readable medium can refer to any medium that participates in providing instructions to a processor 801 for execution or that stores data for computer processing, and includes all computer-readable media, with the sole exception being a transitory propagating signal. Such non-transitory computer-readable media can include, but are not limited to, nonvolatile media, volatile media, and temporary storage media (e.g., cache memory). Nonvolatile media can include, for example, optical or magnetic disks, such as additional storage devices. Volatile media can include, for example, dynamic memory, such as main memory. Common forms of non-transitory computer-readable media can include, for example, a hard disk, a floppy disk, a magnetic tape, or any other magnetic medium, a CD-ROM, DVD, Blu-Ray, or other optical medium, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, a cartridge or tape that is magnetic, optical, or any other physical medium that can be used to store data temporarily or permanently. A computer program product can include a computer-readable medium in coupling with a processor (e.g., the processor 801).

[0115] In selected embodiments, the communication interface 802 can provide or support external, bidirectional data communications over a network link or via a network link. For example, the communication interface 802 can be a wireless network interface controller or a cellular radio that provides data communication network connectivity. Alternatively, the communication interface 802 can include a LAN card to provide data communication connectivity to a compatible LAN. In any such embodiments, the communication interface 802 can send and receive electrical, electromagnetic, or optical signals that convey information.

[0116] The network link can provide data communication through one or more networks to other data devices (e.g., client devices as shown in the computing environment 700). For example, the network link can provide a connection through a local network to a host computer or to data equipment operated by an Internet Service Provider (ISP). The ISP in turn can provide data communication services through the Internet. Thus, the computer 800 can send and receive commands, data, or a combination thereof, including program code, through the one or more networks, the network link, and the communication interface 802. As such, the computer 800 can be in interface with or otherwise communicate with a remote server (e.g., the server 701) through an interface or otherwise or some combination thereof.

[0117] The various apparatuses, modules, terminals, etc. described herein can be implemented in software including machine executable instructions read from a computer readable medium, as discussed above. In certain embodiments, a single computer can be used to implement several hardware aspects; in other embodiments, multiple computers, input / output systems, and hardware can be used to implement the system.

[0118] For software implementations, certain embodiments described herein can be implemented with a software module (such as routines and functions) that performs one or more of the functions and operations described herein. The software code can be written in any suitable programming language and can be stored in memory and executed by a controller or processor.

[0119] The foregoing embodiments and features are merely exemplary and are not to be interpreted as limiting the application. The current teachings can be readily applied to other types of apparatuses and processes. The description of the exemplary embodiments is intended to be illustrative, and not to limit the scope of the claims. Many alternatives, modifications, and variations will be apparent to those skilled in the art.

Claims

1. A method for controlling a performance of a digital character portrayed at a display device, the method comprising: determining a presence of a person located in a physical environment; and in response to determining the presence of the person, facilitating control of the performance of the digital character portrayed at the display device by a combination of a human operator and an artificial intelligence (Al) game engine, wherein, during one or more times, the digital character is capable of being driven at least in part by the Al game engine, wherein, while the digital character is driven by the Al game engine rather than by the human operator, the digital character is still capable of interacting with a person. The determination of the presence of the person is carried out based on data received from at least a camera or a microphone located in the physical environment.

2. The method of claim 1, wherein, Determining the presence of the person includes autonomously determining at least one of a characteristic of an appearance of the person, an emotional state of the person, or audio originating from the person.

3. The method of claim 1, wherein, Facilitating control of the performance of the digital character by the human operator includes providing at least one option selectable by the human operator for controlling the digital character to speak to the person in accordance with the determined characteristic or the determined emotional state.

4. The method of claim 3, wherein, Facilitating control of the performance of the digital character by the human operator includes receiving motion capture data corresponding to the human operator.

5. The method of claim 1, wherein, Receiving the motion capture data includes:

6. The method of claim 5, wherein, receiving data from an optical motion capture system; and receiving data from an inertial motion capture system. The data received from the optical motion capture system includes data corresponding to one or more optical markers located at a waist location of the human operator.

7. The method of claim 6, wherein, The data received from the optical motion capture system is used to carry out drift correction of a position of the human operator determined based on the data received from the inertial motion capture system.

8. The method of claim 6, wherein, Facilitating the control of the performance of the digital character by the human operator such that performance of the digital character is driven simultaneously by both the human operator and the Al game engine.

9. The method of claim 1, wherein, 10. The method of claim 1, further comprising: receiving a request from the person; and serving the request by causing a change in the physical environment in which the person is located, wherein serving the request includes providing a physical object to be delivered for retrieval by the person.

11. An apparatus for controlling a performance of a digital character portrayed at a display device, the apparatus comprising: a network communication unit configured to transmit and receive data; and one or more controllers configured to: determine a presence of a person located in a physical environment; and in response to determining the presence of the person, facilitate control of the performance of the digital character portrayed at the display device by a combination of a human operator and an artificial intelligence (Al) game engine, wherein, during one or more times, the digital character is capable of being driven at least in part by the Al game engine, wherein, while the digital character is driven by the Al game engine rather than by the human operator, the digital character is still capable of interacting with a person. ​ ​ ​ wherein, while the digital character is driven by the AI game engine rather than by the human operator, the digital character is still able to interact with people.

12. The apparatus of claim 11, wherein, The determination of the presence of the person is performed based on data received from at least a camera or microphone located in the physical environment.

13. The apparatus of claim 11, wherein, The one or more controllers are configured to determine the presence of the person by autonomously determining at least one of a characteristic of an appearance of the person, an emotional state of the person, or audio originating from the person.

14. The apparatus of claim 13, wherein, The one or more controllers are configured to facilitate control of the performance of the digital character by the human operator by providing at least one option selectable by the human operator for controlling the digital character to speak to the person in accordance with the determined characteristic or the determined emotional state.

15. The apparatus of claim 11, wherein, The one or more controllers are configured to facilitate control of the performance of the digital character by the human operator by receiving motion capture data corresponding to the human operator.

16. The apparatus of claim 15, wherein, Receiving the motion capture data includes: receiving data from an optical motion capture system; and receiving data from an inertial motion capture system.

17. The apparatus of claim 16, wherein, The data received from the optical motion capture system includes data corresponding to one or more optical markers located at a waist location of the human operator.

18. The apparatus of claim 16, wherein, The data received from the optical motion capture system is used to perform drift correction of a position of the human operator determined based on the data received from the inertial motion capture system.

19. The apparatus of claim 11, wherein, The control of the performance of the digital character by the human operator is facilitated such that performance of the digital character is driven simultaneously by both the human operator and the AI game engine.

20. A machine-readable non-transitory medium having stored thereon machine executable instructions for controlling a performance of a digital character depicted at a display device, the instructions comprising: determining a presence of a person located in a physical environment; and facilitating control of the performance of the digital character depicted at the display device by a combination of a human operator and an artificial intelligence (AI) game engine in response to determining the presence of the person, wherein, during one or more times, the digital character is able to be driven at least in part by the AI game engine, wherein, while the digital character is driven by the AI game engine rather than by the human operator, the digital character is still able to interact with people.

21. A method for controlling a performance of a digital character depicted at a display device in a physical environment, the method comprising: determining a presence of a first person located in a physical environment; and facilitating control of the performance of the digital character depicted at the display device by a combination of a second person and an artificial intelligence (AI) game engine in response to determining the presence of the first person, wherein determining the presence of the person includes autonomously determining at least one of a characteristic of an appearance of the person, an emotional state of the person, or audio originating from the person, wherein facilitating control of the performance of the digital character by the second person includes providing at least one option selectable by the second person for controlling the digital character to speak to the first person when interacting with the first person as a function of the determined characteristic or the determined emotional state of the first person, wherein, in the interaction with the first person, causing a change in the physical environment in which the first person is located in response to an action of the digital character depicted at the display device, wherein the change in the physical environment corresponds to a physical object different from the display device, wherein, during one or more times, the digital character is capable of being driven at least in part by the AI game engine, wherein, when the digital character is driven by the AI game engine and not by the human operator, the digital character is still capable of interacting with a person.

22. The method of claim 21, wherein, The determination of the presence of the first person is carried out based on data received from at least a camera or microphone located in the physical environment.

23. The method of claim 21, wherein, Facilitating control of the performance of the digital character by the second person includes receiving motion capture data corresponding to the human operator.

24. The method of claim 23, wherein, Receiving the motion capture data includes: receiving data from an optical motion capture system; and receiving data from an inertial motion capture system.

25. The method of claim 24, wherein, The data received from the optical motion capture system includes data corresponding to one or more optical markers located at a waist location of the second person.

26. The method of claim 24, wherein, The data received from the optical motion capture system is used to carry out drift correction of a position of the second person determined based on the data received from the inertial motion capture system.

27. The method of claim 21, wherein, Facilitating the control of the performance of the digital character by the second person such that performance of the digital character is driven simultaneously by both the second person and the AI game engine.

28. The method of claim 24, further comprising: receiving a request from the first person; and serving the request by causing a change in the physical environment in which the first person is located, wherein serving the request includes providing a physical object to be delivered for retrieval by the person.

29. An apparatus for controlling performance of a digital character depicted at a display device in a physical environment, the apparatus comprising: a network communication unit configured to transmit and receive data; and one or more controllers configured to: determine a presence of a first person located in a physical environment; and in response to determining the presence of the first person, facilitate control of the performance of the digital character depicted at the display device by a combination of a second person and an artificial intelligence (AI) game engine, wherein the one or more controllers are further configured to determine the presence of the first person by autonomously determining at least one of a characteristic of an appearance of the first person, an emotional state of the first person, or audio originating from the first person, wherein facilitating control of the performance of the digital character by the second person includes providing at least one option selectable by the second person for controlling the digital character to speak to the first person when interacting with the first person as a function of the determined characteristic or the determined emotional state of the first person, wherein, in the interaction with the first person, causing a change in the physical environment in which the first person is located in response to an action of the digital character depicted at the display device, wherein the change in the physical environment corresponds to a physical object different from the display device, wherein, during one or more times, the digital character is capable of being driven at least in part by the AI game engine, wherein, when the digital character is driven by the AI game engine and not by the human operator, the digital character is still capable of interacting with a person. wherein the one or more controllers are further configured to facilitate control of the performance of the digital character by the second person by providing at least one option selectable by the second person for controlling the digital character to speak to the first person when interacting with the first person in accordance with the determined characteristic or the determined emotional state of the first person, wherein, in the interaction with the first person, a change in the physical environment in which the first person is located is caused in response to an action of the digital character depicted at the display device, wherein the change in the physical environment corresponds to a physical object other than the display device, wherein, during one or more times, the digital character is capable of being driven at least in part by the AI game engine, wherein, when the digital character is driven by the AI game engine and not by the human operator, the digital character is still capable of interacting with a person.

30. The apparatus of claim 29, wherein, the determination of the presence of the first person is performed based on data received from at least a camera or microphone located in the physical environment.

31. The apparatus of claim 29, wherein, the one or more controllers are configured to facilitate control of the performance of the digital character by the second person by receiving motion capture data corresponding to the second person.

32. The apparatus of claim 31, wherein, receiving the motion capture data includes: receiving data from an optical motion capture system; and receiving data from an inertial motion capture system.

33. The apparatus of claim 32, wherein, the data received from the optical motion capture system includes data corresponding to one or more optical markers located at a waist location of the second person.

34. The apparatus of claim 32, wherein, the data received from the optical motion capture system is used to perform drift correction of a position of the second person determined based on the data received from the inertial motion capture system.

35. The apparatus of claim 29, wherein, the facilitating of the control of the performance of the digital character by the second person is such that performance of the digital character is driven simultaneously by both the second person and the AI game engine.

36. A machine-readable non-transitory medium having stored thereon machine executable instructions for controlling performance of a digital character depicted at a display device in a physical environment, the instructions comprising: determining a presence of a first person located in a physical environment; and in response to determining the presence of the first person, facilitating control of the performance of the digital character depicted at the display device by a combination of a second person and an artificial intelligence (AI) game engine, wherein determining the presence of the person includes autonomously determining at least one of a characteristic of an appearance of the person, an emotional state of the person, or audio originating from the person, wherein facilitating control of the performance of the digital character by the second person includes providing at least one option selectable by the second person for controlling the digital character to speak to the first person when interacting with the first person in accordance with the determined characteristic or the determined emotional state of the first person, in the interaction with the first person, causing a change in the physical environment in which the first person is located in response to an action of the digital character depicted at the display device, wherein the change in the physical environment corresponds to a physical object different from the display device, wherein, during one or more times, the digital character is capable of being driven at least in part by the AI game engine, wherein, while the digital character is driven by the AI game engine and not by the human operator, the digital character is still capable of interacting with a human.

Images