Digital character interacting with customer in physical realm

Abstract

To provide systems and methods for controlling performance of a digital character depicted at a display device.SOLUTION: According to at least one embodiment, a method for controlling performance of a digital character depicted at a display device includes steps of: determining presence of a person located in a physical environment; and in response to determination of the presence of the person, facilitating control of the performance of the digital character depicted at the display device by a human operator, by an artificial intelligence (AI) game-engine, or by a combination thereof.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Patent Application No. 15 / 931,377, filed May 13, 2020, and U.S. Provisional Patent Application No. 62 / 860,188, filed June 11, 2019, the disclosures of all of which are incorporated herein by reference in their entireties. [Background technology]

[0002] Amusement parks can include various features to create a unique experience for guests. For example, an amusement park can have various rides and shows that entertain guests. Additionally, an amusement park can have show effects and props that can create a desired environment or atmosphere for guests. Such features can include entertainment figures (e.g., animated characters, animated figures) that can interact with guests. For example, the entertainment figures can talk, wave, walk, or perform any other suitable action. Summary of the Invention [Problem to be solved by the invention]

[0003] In accordance with various embodiments disclosed herein, techniques are presented for controlling the performance of a digital character depicted in a digital animation, where the performance is controlled to facilitate interaction between the digital character and a real customer. [Means for solving the problem]

[0004] According to at least one embodiment, a method of controlling a performance of a digital character depicted on a display device is disclosed, the method including determining a presence of a person located within a physical environment, and responsive to determining the presence of the person, facilitating control of the performance of the digital character depicted on the display device by a human operator, or by an artificial intelligence (AI) game engine, or a combination thereof.

[0005] According to at least one embodiment, an apparatus for controlling a performance of a digital character depicted on a display device includes a network communication unit configured to transmit and receive data, and one or more controllers configured to determine a presence of a person located within a physical environment and, in response to determining the presence of the person, facilitate control of the performance of the digital character depicted on the display device by a human operator, or by an artificial intelligence (AI) game engine, or a combination thereof.

[0006] According to at least one embodiment, a machine-readable non-transitory medium stores machine-executable instructions for controlling a performance of a digital character presented on a display device, the instructions including determining a presence of a person located within a physical environment and facilitating control of the performance of the digital character presented on the display device by a human operator, by an artificial intelligence (AI) game engine, or a combination thereof, in response to determining the presence of the person.

[0007] The above and other aspects and features of the present disclosure will become more apparent from the following description of the embodiments, taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1]FIG. 1 illustrates a system for controlling a digital character to interact with a customer, according to at least one embodiment. [Figure 2A] FIG. 1 illustrates an embodiment of live motion capture for controlling a digital character. [Figure 2B] FIG. 1 illustrates an embodiment of live motion capture for controlling a digital character. [Diagram 3] FIG. 2 illustrates an example of a rigid body, according to at least one embodiment. [Figure 4] FIG. 1 shows a 3D view of a capture area. [Diagram 5] FIG. 1 illustrates an exemplary supply of physical items for pick-up by a customer. [Figure 6] FIG. 1 illustrates a flowchart of a method for controlling the performance of a digital character presented on a display device according to at least one embodiment. [Figure 7] FIG. 1 is an illustrative diagram of a computing environment in accordance with at least one embodiment. [Figure 8] FIG. 1 is a block diagram of a device according to at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and which show, by way of illustration, specific embodiments of the present invention. It will be understood by those skilled in the art that other embodiments may be utilized and structural and procedural changes may be made without departing from the scope of the present invention. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or similar elements.

[0010] Embodiments disclosed herein relate to systems for controlling digital characters to interact with customers. For example, the interaction can include the provision of a physical product for collection by the customer. Exemplary embodiments include a system that visually resembles a food truck and outputs food for customers, a system that sells movie tickets to customers, and a system that facilitates the performance of a tug of war between a digital character and a customer.

[0011] An exemplary embodiment will be described with reference to FIG.

[0012] 1 illustrates a system 102 for controlling a digital character to interact with a customer, according to at least one embodiment. The system 102 is located (at least partially) within an environment 100. Also present within the environment 100 are one or more customers 110. The system 102 includes a display device 104 and an output channel 108. The display device 104 is positioned so as to be clearly visible to the customer 110. As described in more detail below, one or more items can be provided via the output channel 108 for pick-up by the customer 110.

[0013] Additionally, the system 102 includes at least one video capture device (e.g., a camera) and at least one audio capture device (e.g., a microphone). The video and audio capture devices can be positioned to capture actions (or non-actions) occurring within the environment 100, and in particular actions (or non-actions) performed by the customer 110. Examples of such actions include movements or motions performed by the customer 110 towards or away from the system 102 (towards or away from the display 104), movements or gestures performed by the customer 110, and facial expressions and reactions performed by the customer 110. As described in more detail below, the capture facilitates detection of the presence of the customer 110 within the environment 100 and / or interaction with the customer 110 (e.g., via one or more digital characters shown on the display device 104).

[0014] As mentioned above, the display device 104 of the system 102 is positioned for clear viewing by the customer 110. The display device 104 displays a video (e.g., a digital video). As described in more detail below, the displayed video may include interactive elements and / or features.

[0015] For purposes of explanation, events shown occurring in the video displayed on display device 104 will be referred to as events occurring "in the digital realm." Separately, other events (e.g., events occurring in environment 100) will be referred to as events occurring "in the physical realm." From the perspective of customer 110, display device 104 can be perceived as a "window" into the physical space defined by system 102, while events occurring in the digital realm are perceived to be occurring in the physical space defined by one or more elements of system 102.

[0016] 1, the digital character 106 is depicted within a video displayed on the display device 104. As one example, the digital character 106 may be created via live motion capture of a human operator.

[0017] 2A, for example, the digital character 106 is created via live motion capture of a human operator 202 located within an environment 200a. The environment 200a can be located proximate to the environment 100 or remotely from the environment 100. When the environment 200a is located near the environment 100, the environment 200a can be configured such that the human operator is hidden from view of the customers 110 present within the environment 100.

[0018] In the environment 200a, a display device 204 is provided. The display device 204 outputs video and audio captured by the video and audio capture devices described above with respect to the system 102. Thus, the human operator 202 can be informed in real time of actions (or inactions) occurring in the environment 100. In this manner, the human operator 202 can effectively interact with the customers 110 present in the physical realm via the digital characters 106 depicted in the digital realm.

[0019] For example, the human operator 202 may ask the customers 110, "Does anyone want a cookie?" In response, via live motion capture, the digital character 106 is shown in the digital realm asking, "Does anyone want a cookie?" Customers 110 watching the digital video displayed on the display device 104 perceive such behavior occurring within the digital realm.

[0020] The customers 110 can react accordingly. For example, one customer 110 can raise his hand. As another example, another customer 110 can shout "Yes!"

[0021] The human operator 202 can observe such actions made by the customer by monitoring the display device 204. For example, by continuing to monitor the camera screen, the human operator 202 can (1) change or shift gaze so that the digital character 106 looks directly into the customer 110's eyes (e.g., if the customer 110 remains relatively still) and (2) reach out towards the customer 110 (e.g., if the customer 110 is reaching or pointing at a particular portion of the display device 104). Similarly, other interactions between the digital character 106 and the customer 110 can include shaking hands with the customer, hugging the customer 110 which is apparently affected by being pushed by the customer, and so forth.

[0022] In at least one embodiment, bridging the digital and physical realms can be accomplished by configuring a digital character to extend a limb to shake hands with or to hand an item to a customer 110. As an example, this can be accomplished by controlling the limbs of a physically manipulated puppet that is manipulated in the physical realm by a human puppeteer hidden from the view of the customer 110. The physically manipulated puppet limbs can be controlled to reach out towards the customer 110 and directly interact with the customer 110 in the physical realm. In this situation, the physically manipulated puppet limbs can be animatronic arms and hands that move realistically based on real-time control by the puppeteer. According to another embodiment, the digitally manipulated puppet limbs can be controlled to reach out and directly interact with the customer 110. For example, the digitally manipulated puppet limbs can be controlled via digital information from a performer (e.g., human operator 202) creating the character experience. In this situation, the performer can control the limb to extend or retract in real time by manually manipulating an on-screen interface, keyboard triggers, buttons, or joystick inputs. Real-time data from the performer will manipulate the hand mechanism to cause the arm to move in a controlled, robotically lawful manner. According to another embodiment including digitally manipulated puppet limbs, the performer can induce a handshake by simply reaching their arm into a known area of ​​the capture volume. Extension and retraction of the puppet limbs and other movements are resolved by the performer and the results are retargeted to the animatronic puppet limbs in a controlled, robotically lawful manner.

[0023] In response to watching and / or listening to the responses made by the customers 110, the human operator 202 may move within the environment 200a as if he were picking out a box of cookies for the customer 110. Then, referring to FIGURE 2B, the human operator 202 may move within the environment 200b as if he were delivering the picked out box to one or more waiting customers 110.

[0024] The movements of the human operator 202 are replicated in the digital domain by the digital character 106. For example, in the digital domain, the digital character 106 may pick up a box of cookies and then move in a direction towards the output channel 108.

[0025] As discussed above, the movements of the human operator 202 can be monitored using a motion capture system. A motion capture system according to one or more embodiments will now be described in further detail.

[0026] Optical systems (or optical motion capture systems) use data captured from image sensors to triangulate the 3D position of a subject between two or more cameras calibrated to provide overlapping projected images. Data can be captured using markers attached to the actors (e.g., human operator 202). Tracking of more performers or extending the capture area can be achieved by adding multiple cameras.

[0027] With respect to optical systems, occlusion occurs when a model marker point is not present within a large number of points, e.g., when the number of observation points is less than the number of model marker points. Such an event may occur due to any of a variety of reasons, such as the interposition of one or more opaque objects between the marker and the camera, or when the marker falls outside the camera field of view (FOV).

[0028] According to at least one embodiment, a drift correction system includes (e.g., hybridizes) one or more aspects of an optical motion capture system along with one or more aspects of an inertial motion capture system. Such a system can enable occlusion-free motion capture that is positionally correct within a capture area (such as environment 200a) and does not drift over time.

[0029] An aspect of the inertial motion capture system includes the use of an inertial body capture suit as known in the art. The inertial body capture suit may be of a type that facilitates live motion capture of the human operator 202, as described above with reference to FIG. 2A. For example, such an inertial body capture suit measures the joint angles of the human operator 202 to estimate the position of the operator from an initial starting point, for example, by counting steps. In this regard, the inertial motion capture system produces approximate results based on magnetometers included in the suit, so errors may accumulate over time. For example, the inertial motion capture system may approximate the location of true north, but errors may occur due to the effects of natural magnetic interference on the accuracy of the magnetometers.

[0030] To help correct for phenomena such as occlusions and artifacts, such as those caused by magnetic interference, the system can use aspects of both the inertial motion capture system and the optical system. According to at least one embodiment, aspects of the optical system may be simpler than those of optical systems typically available for full body motion capture. For example, the optical system may be used to focus on the waist of an actor. According to at least one embodiment, the optical system includes using a rigid device (or rigid body) wearable by the human operator 202. According to at least one embodiment, data from the inertial motion capture system and data from the optical system are combined (or fused together) and used to drive a digital character (e.g., digital character 106).

[0031] As an example, data regarding rotation and height above the ground (see the y-axis in FIG. 2A ) can be received from an inertial body capture suit of an inertial motion capture system, and data regarding the horizontal plane (e.g., see the plane defined by the x-axis and z-axis in FIG. 2A ) can be received from an optical system.

[0032] As discussed above, the optical motion capture system may include a rigid body. The rigid body may be stably attached to the human operator 202, which is drift-compensated. According to at least one embodiment, the rigid body includes markers disposed thereon. For example, the markers may be retro-reflective markers. Such markers may be disposed on the rigid body in a unique pattern such that the rigid body has a unique identity (e.g., configuration or arrangement of markers) that is recognizable or identifiable by the optical motion capture system.

[0033] FIG. 3 illustrates an example of a rigid body 502, according to at least one embodiment. As illustrated, the rigid body 502 can take the form of a belt. The belt is configured to be worn around the waist of a person (e.g., human operator 202). The rigid body 502 has a relatively low bulk such that while a wearer of the rigid body moves, the rigid body moves with the wearer's waist without moving independently relative to the wearer. The rigid body 502 can be made of a highly rigid material and can have a uniform width (e.g., 1 or 2 inches). The markers 506 can be positioned along the circumference of the rigid body 502. As discussed above, the markers 506 can be positioned on the rigid body in a unique pattern. For example, in the example illustrated in FIG. 3, a total of 25 markers 506 are positioned along the circumference of the rigid body 502 in a unique zigzag pattern.

[0034] Alternatively, instead of positioning the markers 506 directly on the belt, the markers 506 can be positioned (indirectly) on the belt via one or more rigid bodies that are positioned (directly) on the belt. In this alternative embodiment, the markers 506 can be positioned directly on the rigid bodies, which are positioned on the belt. For example, five markers 506 can be positioned on each of a total of five rigid bodies, which are positioned on the belt. The placement of the rigid bodies on the belt (as well as the placement of the markers on the rigid bodies) can be a unique pattern.

[0035] It will be appreciated that as few as one optical marker may be sufficient to achieve an acceptable level of drift correction, however, in accordance with at least one embodiment, two or more optical markers are utilized to achieve a higher level of robustness against occlusion.

[0036] With respect to the optical system, various embodiments are described herein with reference to an "external" system. According to such an embodiment, an array of two or more cameras is placed near the perimeter of a capture area (e.g., a room). The cameras capture views of the capture area as well as one or more markers (e.g., marker 506 on rigid body 502) as the rigid body (worn by a human operator) moves about the capture area. The stationary cameras observe the moving markers, thus providing data (e.g., precise rotation and translation) regarding the moving markers that are resolved to rigid objects (e.g., computer-generated bones) in computer animation software.

[0037] FIG. 4 shows a 3D view of a capture area (room) 400. In at least one embodiment, one or more cameras are mounted on each tower located within the capture area 400. For example, as illustrated in FIG. 4, four towers 402 are positioned in the capture area 400, one tower 402 located at each corner. For illustrative purposes, each tower 402 may be an 8-foot tall box truss that is 2 feet wide and includes 1-inch thick aluminum plates. Each of the towers 402 may have three cameras 406 mounted thereon. Within one tower 402, the cameras 406 may be spaced apart from one another—for example, about 2 feet between one another along the height of the tower.

[0038] The camera 406 can be on a small aluminum plate that is inserted into the box truss, and a geared three-way positioning head can be on the plate. The camera 406 can be attached to the positioning head, and the head can be geared to set for precise rotation and lock into place once head aiming is complete. As mentioned above, the towers 402 are positioned at the corners of the capture area 400.

[0039] Each tower 402 generates unique data. During the calibration of the camera 406, a stationary object (e.g., rigid body 502) can be placed in the center of the capture area 400. The camera 406 can be simultaneously calibrated by configuring the optical system to triangulate from the stationary object. Accordingly, the stationary object position in the room can be determined. As described above, data regarding the horizontal plane (see, e.g., the plane defined by the x-axis and the z-axis in FIG. 2A) can be received from the optical system. Such data can be combined (or fused) with data regarding rotation and height above the ground (see, e.g., the y-axis in FIG. 2A), which is received from the inertial body capture suit of the inertial motion capture system. The combination of data can be used to generate animation of the digital character. For example, rotation data received from the inertial coordinate system can be used to animate the skeleton of the digital character, and data received from the optical system can be used to correct the position of the waist of the digital character.

[0040] With regard to optical systems, various embodiments have been described with reference to "external" systems. According to at least another embodiment, an "intrinsic" system is utilized. In such a system, a camera is worn by a human operator to monitor markers located within a capture area (e.g., across the ceiling, floor and / or walls of a room) according to a unique pattern. As the human operator moves around the room, the camera moves with the human operator and monitors stationary markers such that an accurate estimate of the inherent rotation and position within the room relative to the fixed markers on the walls, floor or ceiling of the room can be generated.

[0041] As discussed above, the markers disposed on the rigid body and / or belt can be retroreflective markers, and according to at least one other embodiment, can be light emitting diodes (LEDs) that emit light at wavelengths detectable by an optical motion capture system.

[0042] As discussed above, a drift correction system is created by combining (or hybridizing) aspects of an optical motion capture system with aspects of an inertial motion capture system. According to at least one other embodiment, a drift correction system is created by combining (or hybridizing) aspects of a magnetic motion capture system with aspects of an inertial motion capture system. Such an embodiment may operate in a similar manner as described above with respect to the embodiment using an optical motion capture system. For example, a magnetic "marker" may be placed on a human operator. At either end of the room, a magnetic detection system may be placed. The magnetic detection system may emit a magnetic field to detect magnetic interference caused in the magnetic field by the magnetic marker. The detected interference may be used to determine the location of the magnetic marker.

[0043] In at least one embodiment, techniques using Wi-Fi technology can be utilized. Such an embodiment can operate in a similar manner as described above with respect to the embodiment using a magnetic motion capture system. For example, one or more Wi-Fi antennas can be placed around a room and a Wi-Fi emitter can be placed on the human operator. As the human operator moves within the room, the Wi-Fi emitter can move closer to or farther from one antenna or the other. Characteristics of the signal received at the antennas can be used to determine the location of the emitter within the room.

[0044] In at least one embodiment, techniques using acoustic technology can be utilized. Such an embodiment can operate in a similar manner as described above with respect to the embodiment using an optical motion capture system. In this regard, one of at least two configurations can be used. For example, in one configuration, microphones are placed on the human operator and ultrasonic emitters are placed around the room. As another example, in another configuration, emitters are placed on the actor and microphones are placed around the room. In an embodiment using acoustic technology, time of flight (ToF) can be used to determine the location of whatever object is placed on the human operator (e.g., microphone or emitter). Similar to how echolocation works, an emitter emits an ultrasonic ping and a microphone (or microphone array) detects the ping and calculates the distance the ping travels based on the delay (time of flight) from the emitter to the microphone.

[0045] In at least one embodiment, one or more depth cameras can be utilized to determine the location of a human operator within a room. For example, a depth camera can project an infrared (IR) grid and then detect how the human operator distorts the grid to determine where the operator is and the movements made by the operator. As another example, an array of IR LEDs can be sequentially flashed and a ToF calculation can be performed as the light from the LEDs illuminates and reflects off the human operator and back to the depth camera to make a depth calculation.

[0046] In at least one embodiment, the use of markers is not necessary: ​​for example, in a simpler system, one or more video cameras can be used to track specific features on the human operator or on a rigid body to determine the position of the human operator within the room.

[0047] 5, in the physical domain, a box 302 containing cookies is dispensed at an output channel 108. For example, the system 102 may include a dispenser that dispenses items when controlled (e.g., by a human operator 202).

[0048] As described above with reference to FIG. 1, the system 102 includes at least one video capture device (e.g., a camera) and at least one audio capture device (e.g., a microphone). It is understood that the camera and microphone can be present in a virtual world and experience by the customer in a virtual reality. For example, the customer can control a digital avatar that exists in the digital realm via an input device, such as a joystick, a mouse, or a virtual reality (VR) or motion capture input. Interaction occurs in the digital realm, except for the sale / supply of items (e.g., boxes 302), which still occurs in the physical realm. For example, the customer plays a game, and the game interacts with a game character that corresponds to the customer and does not interact with the customer directly. In this situation, the customer enters and experiences the digital realm as a digital avatar, and the movement of the digital avatar is controlled by the customer based on input provided in the physical realm. Thus, the avatar can be controlled to move (e.g., walk, jump, fly, teleport) in some way in the digital realm. When the presence of a digital avatar is detected in the digital realm, an interaction with the digital avatar is initiated in the same manner that an interaction with a customer is initiated when the presence of a customer is detected in a physical realm (e.g., environment 100). When a customer triggers a change to occur in the physical realm (e.g., the provision of an item), the change is realized in the physical realm.

[0049] Also, as described above with reference to FIGS. 1, 2A, 2B, 3, and 4, a digital character (e.g., digital character 106) can be created through live motion capture of a human operator (e.g., human operator 202). Live motion capture driving motion can be applied to specific parts (or aspects) of a digital character (or object, such as box 302 in FIG. 5), or can drive an entire digital character or prop when appropriate. According to one or more embodiments, a human operator drives the performance of a digital character through real-time Musical Instrument Digital Interface (MIDI) input and / or digital puppetry using devices such as controllers, keyboards, joysticks, mice, foot pedals, microphones, and the like. In such embodiments, live motion capture may not be used exclusively. For example, motion capture can be used to drive the facial performance of a digital character, with remaining aspects of the performance being driven using triggered input devices. According to one or more other embodiments, a digital character 106 can be created through live motion capture of a human operator.

[0050] According to one or more other embodiments, the performance and decision-making for the digital character is generated entirely by an artificial intelligence (AI) character resident in the game engine.

[0051] According to one or more other embodiments, the performance and decision-making for the digital character may be accomplished using a combination of AI game engine driven characters and human motion capture. As an example, during certain times, e.g., while the system is waiting for a customer to engage, the digital character may be at least partially driven by AI (e.g., by playing loops of environmental movements or other character-based animations). At other times, a transition from being driven by AI to being driven by motion capture may occur. For example, when the system detects the presence of a customer 110, the performance (e.g., animation control) of the digital character may be transitioned to a human operator, who may interact with the customer in a more familiar manner. When such a transition occurs, the system 102 may signal to the human operator that at least partial control over the performance of the digital character will be given. For example, the system 102 may provide audio and / or video indicators informing the human operator that control over the digital character's vocal performance, the digital character's facial performance, or the digital character's overall performance will be given.

[0052] Similarly, a transition from being driven by motion capture to being driven by AI game engine technology can occur. Decisions made by such AI technology can be based on analysis of data captured by the video capture devices (e.g., cameras) and / or audio capture devices (e.g., microphones) of the system 102. For example, analysis of data captured by the cameras can be used to identify the locations of various customers 110 as well as certain characteristics of particular customers, such as the blue clothing (e.g., shirt) worn by one customer 110 or the blonde hair of another customer 110. Thus, even when the performance of a digital character is driven by AI game engine technology (rather than by a human operator), the digital character can still interact with the customers 110 to some extent. For example, the eyes of the digital character can be driven such that the "gaze" of the digital character appears to follow the movements made by the customer 110. Also, for example, the digital character can be driven to wave to a particular customer 110 and / or to give a more or less customized or personalized greeting to a customer 110 wearing blue clothing or a customer 110 with blonde hair.

[0053] Similarly, analysis of data captured by a microphone can be used to at least some extent to interface with the customer 110. For example, upon detection of a sound by such analysis as having a particular characteristic (e.g., a sound above a particular threshold audio level), the performance of the digital character can be selectively driven by a particular clip of (prepared) animation. In such a clip, the eyes of the digital character can be moved to appear to shift the digital character's gaze upwards and outwards as if searching for the source of the sound heard. As another example, waveform analysis can be performed on speech audio captured by a microphone. Such analysis can be used in identifying the mood or emotional state of the customer 110 who was speaking. In this situation, the mood in which the digital character sends an audio greeting to the customer 110 can be selected to match or reflect the mood identified by the waveform analysis. For example, if the mood of the customer 110 is identified as happy or cheerful, the digital character can be controlled to address the customer 110 in a happy or cheerful manner.

[0054] Thus, AI game-engine techniques can be used to effectively execute a particular branch in a decision tree (e.g., using a selected clip of animation to drive the performance of a digital character) in response to a particular trigger (e.g., detection of a sound having a particular characteristic). This can increase the likelihood that the digital character will appear alive and be able to react to real-world events occurring within the physical realm (e.g., of the environment 100). This can be preferable to a digital character's performance in which the character simply repeats a particular action while appearing unaware of events occurring within the physical realm.

[0055] Thus, the performance of the digital character need not always be driven by a human operator. For example, the performance of the digital character may be driven by the AI ​​game-engine technology during some (or most) of the time, and by a human operator during selected times (e.g., one or more key times). As another example, the performance of the digital character may be driven by the AI ​​game-engine technology during a majority of a day (e.g., a total of 11 hours in a 12-hour day of operation), and control of the digital character may be assigned to a human operator for selected periods of time (e.g., selected times during such day of operation). The periods may be selected to provide a more personalized interaction with the customer 110 during such periods. At the end of such periods, the human operator may return control of the performance of the digital character to the AI ​​game-engine technology. This return may be chosen to occur at a time to provide a more seamless transition from the perspective of the customer 110. The human operator may return control of the performance of the digital character at such times by, for example, manually manipulating an on-screen interface, keyboard triggers, buttons, or joystick inputs.

[0056] In at least one embodiment, the performance of the digital character can be driven at some times by a human operator as well as by AI game engine technology simultaneously. For example, if the digital character is at least partially driven by an AI in its interactions with the customer 110, the human operator may wish to enhance the performance of the digital character. In this situation, the human operator can control the AI-driven performance in one or more ways. For example, if the human operator identifies the mood of the customer 110 as being particularly happy or energetic, rather than merely happy or energetic, the human operator can control the digital character to address the customer 110 in a similarly enhanced manner. Such control can be performed, for example, by manually manipulating an on-screen interface, keyboard triggers, buttons, or joystick inputs. In this manner, the performance of the digital character can be (temporarily) guided at least in part by the human operator.

[0057] According to embodiments described in more detail below, a number of digital characters may be presented on the display device 104. Each such character may be controlled via various combinations of AI game engine driven characters and human motion capture, respectively, as described above.

[0058] As described above with reference to various embodiments, actions (or non-actions) occurring within the environment 100, and in particular actions (or non-actions) taken by the customer 110, are captured by, for example, cameras and microphones. Upon detection of the presence of the customer 110 within the environment 100 (e.g., one or more particular areas within the environment 100), interaction with the customer is initiated. It is understood that detection may additionally and / or alternatively be performed using other devices. Such other devices include pressure pads, depth cameras, sonic range detection devices, light beam devices, and / or heat or sound detection devices.

[0059] The device can detect not only the presence of a customer, but also other parameters such as the customer's head height and body position. For example, the customer can be detected using a video camera, a depth sensor, or a broken light or sound beam to determine the customer's body position and height. Knowledge of such information (e.g., related to the position of the display device 104) can allow angle corrections to be made to the head or eyes of the digital character 106, such that the digital character's line of sight can more closely align with the line of sight of one or more customers. Such a device can effectively "adjust" the digital character's line of sight by more accurately detecting the customer's head height.

[0060] Also, as described above with reference to various embodiments, the interaction includes one or more interactions between the digital character and the customer for both the digital and physical realms. For example, in the digital realm, the digital character may perform a step of preparing or procuring an item (e.g., a box of cookies 302) for the customer. In the physical realm, the item is given (or sold) to the customer in real time, and the item is made available for retrieval or collection by the customer (e.g., at output channel 108). According to one or more further embodiments, the item is presented such that the item can be retrieved by the customer at another location (e.g., somewhere outside environment 100). According to one or more further embodiments, the item is provided to the customer at a later date or later (e.g., by regular mail).

[0061] According to one or more other embodiments, the interaction between the digital character and the customer includes an interaction that causes (or effects) a change in a physical area of ​​the environment 100. For example, the system 102 may include a pulley that facilitates conducting a tug-of-war game between the digital character and the customer.

[0062] As one example, a human operator 202 controlling the performance of a digital character may ask, "Who wants to play tug of war with me?" Thus, in the digital realm, the digital character 106 may ask the customers 110, "Who wants to play tug of war with me?" In the physical realm, the customers 110 may react accordingly. For example, one customer 110 may raise his hand. As another example, another customer 110 may shout, "Yes!"

[0063] In response to watching and / or listening to the reactions made by the patrons 110, the human operator 202 may move within the environment 200a as if the human operator 202 were retrieving a rope for a game to be played. The human operator 202 may then move within the environment 200b as if the human operator 202 were moving the retrieved rope towards one or more waiting patrons 110.

[0064] The actions of the human operator 202 are replicated in the digital domain by the digital character 106. For example, in the digital domain, the digital character 106 retrieves a rope and then moves in a direction towards the output channel 108.

[0065] Within the physical area, the free end of the rope is extended outward through output channel 108. The other end of the rope can be coupled to a pulley system that is hidden from view of the customers 110. After one of the customers grabs the rope, a game of tug-of-war can begin.

[0066] According to one or more further embodiments, examples of other interactions that cause changes in the physical area of ​​environment 100 include objects in the physical area being moved (e.g., knocked over) in response to actions by a digital character in the digital area (e.g., tripping or falling), lights in the physical area being controlled to turn on / off in response to actions by a digital character in the digital area, and any other aspect in the physical area that is changed in response to actions by a digital character in the digital area.

[0067] Similarly, actions by a customer in the physical realm detected by the system 102 may cause changes in the digital realm. For example, facial responses or expressions by a customer, a customer's body position, other movements made by a customer, or audio responses or sounds made by a customer may be detected and cause associated events to occur in the digital realm.

[0068] Interactions between the physical and digital realms are not limited to interactions involving a customer and a digital character. For example, the presence of one or more particular objects in the physical realm that are detected may result in changes in the digital realm, and vice versa. For example, the changes may include corresponding or related objects in the digital realm. As an example, an object moved by a customer in the physical realm, either intentionally or accidentally, may be followed by the movement of a corresponding or related object by a digital character in the digital realm.

[0069] Causing a change in the physical area of ​​the environment 100 can be performed to increase the likelihood that a customer will engage with one or more digital characters in the digital area. If customer engagement is not detected and the performance of the digital characters is at least partially created by AI, the system 102 can enter a loop state in which it continuously (or periodically) monitors for detected engagement. Alternatively (or additionally), the system 102 can begin displaying a tracking animation to motivate the customer to engage with the digital characters. Alternatively (or additionally), the system 102 can return to a standby state in which a particular set of animations is displayed until a customer presence is detected.

[0070] As described above with reference to various embodiments, an object such as a box of cookies may be provided (e.g., via output channel 108) for retrieval by a customer. In this situation, the object may be provided relatively immediately after a request by the customer (e.g., the customer says "Yes!") is detected. In other situations, the delivery may occur after a certain delay. For example, a toy sword may be provided to a customer. In the physical realm, a pre-fabricated toy sword may be retrieved for placement at the receiving end of the output channel. However, in the digital realm, a digital character may create the toy sword (e.g., from raw materials) at a blacksmith. In such a situation, an animation loop (or sequence) may be displayed on a display device (e.g., display device 104) to show the creation of the toy sword. The display of such a loop may continue until feedback is received in the physical realm indicating that the delivery of the toy sword is to occur. Once such feedback is received, the display of the loop ends and the creation of the toy sword is completed in the digital realm. Examples of such feedback are described in more detail below.

[0071] According to another example, the object may be an item that requires some preparation time in the physical domain, for example, the object may be a food item (e.g., a bowl of noodle soup) that is requested (or ordered) by a customer.

[0072] Before a request is received, the presence of a customer may be detected first. As described above with reference to various embodiments, one or more customer movements (e.g., in the environment 100) may be detected. When one or more movements (e.g., one or more specific movements, such as movements toward the display device 104) are detected, an action is performed that calls for customer engagement. For example, when the system 102 is designed to visually resemble the outside of a kitchen or food truck, a sequence (e.g., a game sequence) may be initiated in which the display device 104 displays one or more digital characters preparing food. For example, in the digital realm, the digital characters may chop vegetables, cook noodles, etc.

[0073] The representation of one or more items in the digital realm, such as a cooking spoon or pot, can be controlled in several ways. For example, an item can have a counterpart that exists in the physical realm, such as a spoon (e.g., made of plastic or foam) carried by an actor (e.g., human operator 202 in FIG. 2A) that can be tracked by motion capture. As another example, the representation of an item can be digitally controlled when the item exists in a game engine, such as an object in a video game. In this situation, the actor can interact with the item by making a grasping gesture and placing his hand near the item to alert the game engine that he wants to pick up the item. The actor can also trigger the grasping gesture via a controller or button press. In another example, the actor can trigger the item to appear from out of sight into the actor's hand by pressing a button or reaching into a particular 'zone'.

[0074] As described above with reference to FIG. 2B, in the digital realm, the digital character 106 retrieves a box of cookies. The actor can accomplish this retrieval of the box by reaching into the air so that the hand of the digital character 106 enters an invisible cube in the digital realm that exists off-frame (e.g., just outside of frame) relative to a display (e.g., display 104). When the hand of the digital character 106 enters this zone, the box is placed in the actor's hand in the digital realm. When the actor lowers his arm in the physical realm, the box appears in the hand of the digital character 106 in the digital realm, as if the digital character 106 is reaching off-screen and pulling the box out from an invisible shelf.

[0075] According to another embodiment, a combination of one or more items in the digital realm and one or more items in the physical realm is utilized. For example, such a combination can be utilized to achieve chopping of a carrot in the digital realm by a digital character. The combination can include a counterpart in the physical realm (e.g., a physical prop) that is held by an actor and corresponds to a knife in the digital realm, and a carrot that exists only in the digital realm. The movement of the physical prop is tracked so that the knife appears in the digital character's hand and moves accordingly. When the edge of the knife moves near the carrot in the digital realm, the carrot is shown to be sliced ​​into multiple pieces. When the side of the knife is brought near the carrot piece, the carrot piece can be shown to be moved off the edge of the cutting board and into the pot. This movement of the carrot piece can also be shown to be caused directly by the digital character's hand (e.g., by an actor gesturing to sweep the carrot piece into the pot). However, it is understood that to achieve a higher degree of realism, only the knife (and not the digital character's hand) can slice the carrot into multiple pieces.

[0076] Triggered movements can also be achieved in a similar manner. Such movements can include, for example, martial arts movements that a typical non-skilled person cannot easily execute. These movements can be triggered by a button press on a controller that blends a live movement with one or more pre-prepared movements, or by an actor making a gesture (e.g., a substitute gesture) that, when recognized by the system, initiates the triggered movement. For example, the actor can kick his / her leg, and the system can recognize this kick as the trigger for a special kung fu jump kick sequence. According to further embodiments, the triggered movement does not completely drive the performance of the digital character. For example, only one or more parts of the digital character's body can be driven by the triggered movement. As an example, the digital character's face can still be controlled by the actor (e.g., via motion capture) while the triggered movement is executed by the digital character's torso, arms, and / or legs.

[0077] Returning to detecting the presence of a customer, a request is received from the detected customer. Receiving the request may include using natural language processing (to receive and process a verbal request). Alternatively (or additionally), receiving the request may include using an interface operable by the customer. For example, the customer may operate buttons on a touch screen or fill out an instruction card that is inserted into the machine and read.

[0078] After a customer requests a particular item, preparation of the item begins within the physical domain, for example, a food preparer (e.g., a cook) located in proximity to environment 100 begins cooking the requested item.

[0079] Concurrently, actions occur in the digital realm indicating the preparation of the item. Such actions can be customized based on customer requests. For example, based on a customer request that a noodle soup include extra carrots, a sequence that is executed can include the display of carrots flying throughout the kitchen in the digital realm.

[0080] The animation displayed on the display device 104 can be continuously controlled while the cooking of the requested item in the physical domain is in progress. For example, during the cooking of the requested item, feedback can be received from the food preparer indicating, for example, that more time is needed before cooking is complete. Such feedback can cause further loops of a particular sequence to be displayed. It is understood that the animation displayed on the display device 104 can be controlled by AI game engine techniques as well as by a human operator simultaneously, similar to the manner in which the performance of a particular digital character can be controlled as described above with reference to various embodiments.

[0081] Additionally, the animation can continue to control when the requested item is nearing completion or is complete. For example, feedback can be received from the food preparer indicating that the item is ready for output. The feedback can come from a pressure sensor that senses an article (e.g., the requested item) being placed in a determined location. Thus, the food preparer can place the requested item near or at the pressure sensor to provide feedback indicating that the item is ready for output. As another example, the system 102 can include an interface that is operable by the food preparer. Thus, the food preparer can operate a button on a touch screen to signal that the item is ready for output.

[0082] Within the physical area, items may be placed in output channels 108 for pick-up by customers.

[0083] Based on the described features, the timing of events occurring in the digital realm (e.g., the creation of an item by a digital character as shown in a displayed animation) can be better aligned with the timing of events occurring in the physical realm (e.g., the preparation of an item by a food preparer). For example, the timing in both realms is better aligned such that as a bowl of noodle soup disappears from view in the digital realm, it appears in the physical realm of output channel 108. According to one or more particular embodiments, the items presented in output channel 108 are selected to match in appearance the items shown in the digital realm.

[0084] FIG. 6 illustrates a flowchart of a method 600 for controlling the performance of a digital character displayed on a display device according to at least one embodiment.

[0085] At block 602, according to certain embodiments, the performance of the digital character may be controlled using AI game engine techniques. For example, referring to FIG. 1, the performance of the digital character 106 is controlled using AI game engine techniques.

[0086] The presence of a customer located within the physical environment is detected at block 604. For example, with continued reference to Figure 1, the presence of a customer 110 located within the physical environment 100 is detected.

[0087] According to a further embodiment, the detection of the customer presence is performed based on data received from at least a camera or a microphone located in the physical environment.

[0088] According to further embodiments, detecting the presence of a customer may include autonomously determining an appearance characteristic of the customer or an emotional state of the customer. For example, data received from a camera may be used in determining that the customer 110 is wearing blue clothing. As another example, data received from a microphone may be used to detect and determine if the customer 110 is happy or sad.

[0089] In block 606, in response to detecting the presence of a customer, control of the performance of the digital character (or at least aspects thereof) by a human operator, an AI game engine, or a combination thereof is facilitated. For example, with reference to FIGURE 2A, in response to detecting the presence of a customer 110, control of the performance of the digital character 106 by a human operator 202 is facilitated.

[0090] According to a further embodiment, facilitating control of the performance of the digital character may include providing at least one option selectable by a human operator for controlling the digital character to address the customer according to the determined characteristics or the determined emotional state. For example, if the human operator 202 identifies the mood of the customer 110 as being particularly happy or cheerful, the human operator may control the digital character 106 to address the customer 110 in such a particularly happy or cheerful manner.

[0091] According to a further embodiment, facilitating control of the performance of the digital character can include receiving motion capture data corresponding to a human operator. By way of example, receiving the motion capture data can include receiving data from an optical motion capture system (e.g., a system utilizing camera 406 of FIG. 4) and receiving data from an inertial motion capture system (e.g., a system utilizing an inertial body capture suit worn by the human operator 202).

[0092] According to a further embodiment, the data received from the optical motion capture system can include data corresponding to one or more optical markers located on the waist region of the human operator. For example, the data received from the optical motion capture system can include data corresponding to marker 506 located on the waist region of the human operator.

[0093] In a further embodiment, control of at least a portion of the digital character's performance by a human operator is facilitated such that the digital character's performance is driven simultaneously by the human operator and AI game engine technology.

[0094] At block 608, according to certain embodiments, a request is received from a customer. For example, referring to Figure 1, customer 110 may request that a box of cookies be provided.

[0095] At block 610, the request may be processed by causing a change in the physical environment in which the customer is located. As an example, the request may be processed by providing a physical object that is submitted for retrieval by the customer. For example, referring to FIG. 5, a box 302 containing cookies is provided and submitted for retrieval by the customer 110.

[0096] In selected embodiments, features and aspects described herein may be implemented in a computing environment 700, which may include one or more computer servers 701, as shown in Figure 7. The servers 701 may be operatively coupled to one or more data stores 702 (e.g., databases, indexes, files, or other data structures). The servers 701 may be connected to a data communications network 703, which may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a telephone network, a satellite or wireless communications network, or any combination of these or similar networks.

[0097] One or more client devices 704, 705, 706, 707, 708, 709, 710 can communicate with the server 701 and corresponding data store 702 via a data communications network 703. Such client devices 704, 705, 706, 707, 708, 709, 710 can include, for example, one or more laptop computers 707, desktop computers 704, smart phones and mobile phones 705, tablet computers 706, televisions 708, motion capture sensors 709, cameras 710, or combinations thereof. In operation, such client devices 704, 705, 706, 707, 708, 709, 710 can send data or instructions to or receive data or instructions from the server 701 in response to user input received from a user input device or other input. In response, server 701 may process data from data store 702, modify data in data store 702, add data to data store 702, or the like, or any combination thereof.

[0098] In selected embodiments, the server 701 can send one or more media files including audio and / or video content, encoded data, generated data, and / or metadata from the data store 702 to one or more of the client devices 704, 705, 706, 707, 708, 709, 710 via the data communications network 703. The devices can output the audio and / or video content from the media files using a display screen, projector, or other display output device. In certain embodiments, the system 700 configured according to features and aspects described herein can be configured to operate within or support a cloud computing environment. For example, some or all of the data store 702 and the server 701 can reside within a cloud server.

[0099] 8, shown is an illustration of an exemplary computer 800. One or more of the devices 704, 705, 706, 707, 708 of the system 700 may be configured as or include such a computer 800.

[0100] In selected embodiments, computer 800 may include a bus 803 (or multiple buses) or other communication mechanism, a processor 801, a main memory 804, a read only memory (ROM) 805, one or more additional storage devices 806, and / or a communication interface 802, or the like, or subcombinations thereof. The embodiments described herein may be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microphone controllers, microphone processors, other electronic units designed to perform the functions described herein, or any selected combination thereof. In all embodiments, the various components described herein may be implemented as a single component, or alternatively may be implemented as various separate components.

[0101] The bus 803 or other communication mechanism may include a plurality of such buses or mechanisms to support communication of information within the computer 800. The processor 801 is coupled to the bus 803 and may process information. In selected embodiments, the processor 801 may be a specialized or dedicated microphone processor configured to perform particular tasks according to the features and aspects described herein by executing machine-readable software code that defines the particular tasks. A main memory 804 (e.g., a random access memory or RAM or other dynamic storage device) is coupled to the bus 803 and may store information and instructions to be executed by the processor 801. The main memory 804 may also store temporary variables or other intermediate information during the execution of such instructions.

[0102] A ROM 805 or some other static storage device may be coupled to the bus 803 to store static information and instructions for the processor 801. An additional storage device 806 (e.g., magnetic disks, optical disks, memory cards, etc.) may be coupled to the bus 803. The main memory 804, the ROM 805 and the additional storage device 806 may include non-transitory computer-readable media that retain information, instructions, or any combination thereof, such as instructions that, when executed by the processor 801, cause the computer 800 to perform one or more operations of the methods described herein. A communication interface 802 may also be coupled to the bus 803. The communication interface 802 may provide or support two-way data communication between the computer 800 and one or more external devices (e.g., other devices contained within the computing environment).

[0103] In selected embodiments, computer 800 may be connected to a display 807 (e.g., via bus 803). Display 807 may use any suitable mechanism for communicating information to a user of computer 800. For example, display 807 may include or utilize a liquid crystal display (LCD), a light emitting diode (LED) display, a projector, or other display device to present information in a visual display to a user of computer 800. One or more input devices 808 (e.g., an alphanumeric keyboard, a mouse, a microphone) may be connected to bus 803 for communicating information and commands to computer 800. In selected embodiments, one input device 808 may be provided by computer 800 and provide or support control of cursor positioning to enable selection and execution of various objects, files, programs, etc. displayed by display 807.

[0104] Computer 800 may be used to transmit, receive, decode, display, etc., one or more video files. In selected embodiments, such transmitting, receiving, decoding, and displaying may be in response to processor 801 executing one or more sequences of one or more instructions contained in main memory 804. Such instructions may be read into main memory 804 from another non-transitory computer-readable medium, such as a storage device.

[0105] Execution of the sequences of instructions contained in main memory 804 may cause processor 801 to perform one or more of the procedures or steps described herein. In selected embodiments, one or more processors of a multiprocessing arrangement may also be used to execute the sequences of instructions contained in main memory 804. Alternatively, or additionally, firmware may be used in place of, or in combination with, software instructions to implement procedures or steps in accordance with features and aspects described herein. Thus, embodiments in accordance with features and aspects described herein may not be limited to any specific combination of hardware circuitry and software.

[0106] A non-transitory computer readable medium may refer to any medium involved in holding instructions for execution by the processor 801 or storing data for processing by a computer, and may include all computer readable media, except for transitory propagating signals. Such non-transitory computer readable media may include, but are not limited to, non-volatile media, volatile media, and temporary storage media (e.g., cache memory). Non-volatile media may include optical or magnetic disks, such as additional storage devices. Volatile media may include dynamic memory, such as main memory. Common forms of non-transitory computer readable media may include, for example, hard disks, floppy disks, magnetic tapes, or any other magnetic media, CD-ROMs, DVDs, Blu-rays, or optical media, RAMs, PROMs, EPROMs, flash EPROMs, any other memory cards, chips, or cartridges, or any other memory media that can be read by a computer.

[0107] In selected embodiments, communication interface 802 may provide or support external bidirectional data communication to or through a network link. For example, communication interface 802 may be a wireless network interface controller or a cellular radio that provides a data communication network connection. Alternatively, communication interface 802 may include a LAN card that provides a data communication connection to a compatible LAN. In any such embodiment, communication interface 802 may send and receive electrical, electromagnetic, or optical signals that carry information.

[0108] The network link may provide data communication through one or more networks to other data devices (e.g., client devices as shown in computing environment 700). For example, the network link may provide a connection through a local network of a host computer or to data facilities operated by an Internet Service Provider (ISP). The ISP may in turn provide data communication services over the Internet. Thus, computer 800 may send and receive commands, including program code, data, or any combination thereof, through one or more networks, network links, and communication interface 802. In this manner, computer 800 may interface with or otherwise communicate with a remote server (e.g., server 701) or any combination thereof.

[0109] The various devices, modules, terminals, and the like described herein may be implemented on a computer by execution of software, including machine instructions, read from a computer-readable medium, as discussed above. In certain embodiments, some hardware aspects may be implemented using a single computer, while in other embodiments, the system may be implemented using multiple computers, input / output systems, and hardware.

[0110] For software implementations, certain embodiments described herein may be implemented with separate software modules, such as procedures and functions, each of which performs one or more of the functions and operations described herein. The software code may be implemented in a software application written in any suitable programming language and may be stored in a memory and executed by a controller or processor.

[0111] The above described embodiments and features are merely illustrative and should not be construed as limiting the present invention. The teachings of the present invention can be readily applied to other types of devices and processes. The description of such embodiments is intended to be illustrative and not limiting of the scope of the claims. Many alternatives, modifications, and variations will be apparent to those skilled in the art. [Explanation of symbols]

[0112] 602: Controlling the performance of digital characters using artificial intelligence (AI) game engine technology 604: Detecting the presence of a person located within a physical environment 606: Facilitate control of digital character performance by a human operator, an AI game engine, or a combination of both 608: Receive requests from people 610: Process requests by inducing changes in the physical environment

Claims

1. 1. A method for controlling the performance of a digital character presented on a display device, comprising: determining the presence of a person located within the physical environment; facilitating control of a performance of the digital character shown on the display device by a human operator or by an artificial intelligence (AI) game engine, or a combination thereof, in response to determining the presence of the person; A method comprising:

2. The method of claim 1 , wherein the determination of the presence of a person is performed based on data received from at least a camera or a microphone located in the physical environment.

3. The method of claim 1 , wherein determining the presence of the person comprises autonomously determining at least one of an appearance characteristic of the person, an emotional state of the person, or audio emanating from the person.

4. 4. The method of claim 3, wherein facilitating control of the performance of the digital character by the human operator comprises providing at least one option selectable by the human operator to control the digital character to act on the person in response to the determined characteristics or the determined emotional state.

5. The method of claim 1 , wherein facilitating control of a performance of the digital character by the human operator comprises receiving motion capture data corresponding to the human operator.

6. The step of receiving the motion capture data includes: receiving data from an optical motion capture system; receiving data from an inertial motion capture system; The method of claim 5 , comprising:

7. The method of claim 6 , wherein the data received from the optical motion capture system includes data corresponding to one or more optical markers located in a waist region of the human operator.

8. 7. The method of claim 6, wherein the data received from the optical motion capture system is for performing drift correction of the position of the human operator determined based on the data received from the inertial motion capture system.

9. 2. The method of claim 1, wherein the control of the digital character's performance by the human operator is facilitated such that the digital character's performance is driven simultaneously by both the human operator and the AI ​​game engine.

10. receiving a request from the person; Processing the request by causing a change in the physical environment in which the person is located; Further comprising: The method of claim 1 , wherein processing the request includes providing a physical object to be submitted for retrieval by the person.

11. 1. An apparatus for controlling the performance of a digital character shown on a display device, comprising: a network communication unit configured to transmit and receive data; one or more controllers configured to determine the presence of a person located within a physical environment and, in response to determining the presence of the person, facilitate control of a performance of the digital character presented on the display device by a human operator, or by an artificial intelligence (AI) game engine, or a combination thereof; An apparatus comprising:

12. The apparatus of claim 11 , wherein the determination of the presence of a person is performed based on data received from at least a camera or a microphone located in the physical environment.

13. 12. The apparatus of claim 11, wherein the one or more controllers are configured to determine the presence of the person by autonomously determining at least one of an appearance characteristic of the person, an emotional state of the person, or audio emanating from the person.

14. 14. The apparatus of claim 13, wherein the one or more controllers are configured to facilitate control of the performance of the digital character by the human operator by providing at least one option selectable by the human operator to control the digital character to act on the person in accordance with the determined characteristics or the determined emotional state.

15. 12. The apparatus of claim 11, wherein the one or more controllers are configured to receive motion capture data corresponding to the human operator to facilitate control of a performance of the digital character by the human operator.

16. Receiving the motion capture data includes: receiving data from an optical motion capture system; and receiving data from an inertial motion capture system; 16. The apparatus of claim 15, comprising:

17. 17. The apparatus of claim 16, wherein the data received from the optical motion capture system includes data corresponding to one or more optical markers located in a waist region of the human operator.

18. 17. The apparatus of claim 16, wherein the data received from the optical motion capture system is for performing drift correction of the position of the human operator determined based on the data received from the inertial motion capture system.

19. 12. The apparatus of claim 11, wherein the control of the digital character's performance by the human operator is facilitated such that the digital character's performance is driven simultaneously by both the human operator and the AI ​​game engine.

20. A machine-readable non-transitory medium storing machine-executable instructions for controlling performance of digital characters presented on a display device, comprising: The instruction: Determining the presence of a person located within a physical environment; and facilitating control of the performance of the digital character shown on the display device by a human operator, or by an artificial intelligence (AI) game engine, or a combination thereof, in response to determining the presence of the person; A machine-readable non-transitory medium, comprising:

Images