Digital characters that interact with customers in the physical realm

Abstract

To provide systems and methods for controlling performance of a digital character depicted at a display device.SOLUTION: According to at least one embodiment, a method for controlling performance of a digital character depicted at a display device includes steps of: determining presence of a person located in a physical environment; and in response to determination of the presence of the person, facilitating control of the performance of the digital character depicted at the display device by a human operator, by an artificial intelligence (AI) game-engine, or by a combination thereof.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Patent Application No. 15 / 931,377, filed on May 13, 2020, and U.S. Provisional Patent Application No. 62 / 860,188, filed on June 11, 2019, and the entire disclosure of each of these is hereby incorporated by reference in its entirety.

Background Art

[0002] An amusement park can include various features to provide a unique experience for visitors. For example, an amusement park can have various rides and shows to entertain visitors. Further, an amusement park can have show effects and props that can create a desirable environment or atmosphere for visitors. Such features can include entertainment figures (e.g., anime characters, anime figures) that can interact with visitors. For example, an entertainment figure can talk, wave its hands, walk, or perform any other suitable actions.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Regarding various embodiments disclosed herein, a technique for controlling the performance of digital characters presented in digital animation is presented. According to various embodiments, the performance is controlled to facilitate interaction between the digital characters and actual customers.

Means for Solving the Problems

[0004] According to at least one embodiment, a method for controlling the performance of a digital character displayed on a display device is disclosed. The method includes the steps of determining the presence of a person located in a physical environment, and, in response to determining the presence of a person, facilitating the control of the performance of the digital character displayed on the display device by a human operator, an artificial intelligence (AI) game engine, or a combination thereof.

[0005] According to at least one embodiment, a device for controlling the performance of a digital character shown on a display device includes a network communication unit configured to send and receive data, and one or more controllers. The one or more controllers are configured to determine the presence of a person located in the physical environment and, in response to determining the presence of a person, to facilitate the control of the performance of the digital character shown on the display device by a human operator, an artificial intelligence (AI) game engine, or a combination thereof.

[0006] According to at least one embodiment, a machine-readable non-temporary medium stores machine-executable instructions for controlling the performance of a digital character shown on a display device. The instructions include determining the presence of a person located in a physical environment, and, in response to determining the presence of a person, facilitating the control of the performance of the digital character shown on the display device by a human operator, an artificial intelligence (AI) game engine, or a combination thereof.

[0007] The above and other aspects and features of this disclosure will become more apparent from the following description of embodiments with reference to the accompanying drawings. [Brief explanation of the drawing]

[0008] [Figure 1]This figure shows a system for controlling a digital character to interact with a customer, according to at least one embodiment. [Figure 2A] This figure shows an example of live motion capture for controlling a digital character. [Figure 2B] This figure shows an example of live motion capture for controlling a digital character. [Figure 3] This figure shows an example of a rigid body according to at least one embodiment. [Figure 4] This is a 3D view of the capture area. [Figure 5] This figure illustrates the supply of physical items for customer retrieval. [Figure 6] This figure shows a flowchart of a method for controlling the performance of a digital character displayed on a display device, according to at least one embodiment. [Figure 7] This is an illustrative diagram of a computing environment according to at least one embodiment. [Figure 8] This is a block diagram of a device according to at least one embodiment. [Modes for carrying out the invention]

[0009] In the following detailed description, reference is made to the accompanying drawings, which form part of the description and illustrate specific embodiments of the invention. Those skilled in the art will understand that other embodiments may be utilized and structural and procedural modifications may be made without departing from the scope of the invention. Wherever possible, the same reference numerals are used throughout the drawings to indicate the same or similar elements.

[0010] Embodiments disclosed herein relate to systems for controlling a digital character to interact with a customer. For example, the interaction may include the supply of physical products to be picked up by the customer. Exemplary embodiments include a system that visually resembles a food truck and dispenses food for a customer, a system for selling movie tickets to a customer, and a system for facilitating a tug-of-war performance between a digital character and a customer.

[0011] An exemplary embodiment will be described with reference to Figure 1.

[0012] Figure 1 shows a system 102 for controlling a digital character to interact with a customer, according to at least one embodiment. The system 102 is located (at least partially) within an environment 100, which also contains one or more customers 110. The system 102 includes a display device 104 and an output channel 108. The display device 104 is positioned so that it is clearly visible to the customer 110. One or more items can be supplied via the output channel 108 for retrieval by the customer 110, as will be described in more detail below.

[0013] Furthermore, system 102 includes at least one video capture device (e.g., a camera) and at least one audio capture device (e.g., a microphone). The video and audio capture devices can be positioned to capture actions (or non-actions) occurring within the environment 100, in particular actions (or non-actions) made by the customer 110. Examples of such actions include movements or motions made by the customer 110 toward or away from system 102 (towards or away from display 104), movements or gestures made by the customer 110, and facial expressions and reactions made by the customer 110. As will be described in more detail below, the capture facilitates the detection of the presence of the customer 110 within the environment 100 and / or interaction with the customer 110 (e.g., by one or more digital characters shown on display device 104).

[0014] As described above, the display device 104 of system 102 is positioned so that the customer 110 can see it clearly. The display device 104 displays video (e.g., digital video). As will be described in more detail below, the displayed video may include interactive elements and / or features.

[0015] For explanatory purposes, events that appear to occur in the video displayed on the display device 104 will be referred to as events occurring "within the digital domain." Separately, other events (for example, events occurring within the environment 100) will be referred to as events occurring "within the physical domain." From the customer's perspective 110, the display device 104 can be perceived as a "window" to the physical space defined by the system 102, while events occurring within the digital domain are perceived as occurring within the physical space defined by one or more elements of the system 102.

[0016] Continuing to refer to FIG. 1, digital character 106 is shown within the video displayed on display device 104. As an example, digital character 106 can be created via live motion capture of a human operator.

[0017] For example, referring to FIG. 2A, digital character 106 is created via live motion capture of human operator 202 located within environment 200a. Environment 200a can be located in the vicinity of environment 100 or at a location remote from environment 100. When environment 200a is located near environment 100, environment 200a can be configured to be hidden from the view of customer 110 where the human operator is present within environment 100.

[0018] In environment 200a, display device 204 is provided. Display device 204 outputs the video and audio captured by the video and audio capture devices described above with respect to system 102. Accordingly, human operator 202 can be informed in real time of the actions (or inactions) occurring in environment 100. In this way, human operator 202 can effectively interact with customer 110 present within the physical region via digital character 106 shown in the digital region.

[0019] For example, human operator 202 can ask customer 110, "Does anyone want a cookie?" In response, via live motion capture, digital character 106 is shown in the digital region asking, "Does anyone want a cookie?" Customer 110, who is viewing the digital video displayed on display device 104, perceives such actions occurring within the digital region.

[0020] Customer 110 can respond accordingly. For example, one Customer 110 can raise a hand. As another example, another Customer 110 can shout "Yes!".

[0021] Human operator 202 can observe such actions made by the customer by monitoring display device 204. For example, by continuously monitoring the camera screen, human operator 202 can (1) change or shift the gaze so that digital character 106 looks directly into the eyes of customer 110 (for example, when customer 110 remains relatively stationary), (2) extend a hand towards customer 110 (for example, when customer 110 is reaching or pointing a hand at a specific part of display device 104). Similarly, other interactions between digital character 106 and customer 110 can include a handshake with the customer, an embrace of customer 110 clearly affected by being pushed by the customer, and others.

[0022] In at least one embodiment, bridging the digital and physical realms can be achieved by configuring a digital character to extend its limbs to shake hands with a customer 110 or to hand an item to a customer 110. As an example, this can be achieved by controlling the limbs of a physically operated puppet, which is operated in the physical realm by a human puppeteer hidden from the customer 110's field of vision. The limbs of the physically operated puppet can be controlled to reach out to the customer 110 and interact directly with the customer 110 in the physical realm. In this scenario, the limbs of the physically operated puppet can be animatronic arms and hands that move realistically based on real-time control by the puppeteer. According to another embodiment, the limbs of a digitally operated puppet can be controlled to reach out and interact directly with the customer 110. For example, the limbs of a digitally operated puppet can be controlled via digital information from a performer (e.g., human operator 202) creating the character experience. In this scenario, the performer can control the limbs to extend or retract in real time by manually operating an on-screen interface, keyboard triggers, buttons, or joystick inputs. Real-time data from the performer manipulates the hand mechanism to move the arm in a controlled, robotically lawful manner. According to another embodiment, which includes digitally manipulated puppet limbs, the performer can trigger a handshake simply by extending the arm into a known area of ​​capture capacity. Extending and retracting the puppet's limbs, as well as other movements, are resolved by the performer, and the results are retargeted onto the animatronic puppet's limbs in a controlled, robotically lawful manner.

[0023] In response to viewing and / or hearing the reaction made by customer 110, the human operator 202 may move within environment 200a as if he were retrieving a box of cookies for customer 110. Then, referring to Figure 2B, the human operator 202 may move within environment 200b as if he were supplying the retrieved boxes to one or more waiting customers 110.

[0024] The movements of the human operator 202 are reproduced in the digital domain by the digital character 106. For example, in the digital domain, the digital character 106 picks up a box of cookies and then moves toward the output channel 108.

[0025] In the above, the movement of the human operator 202 can be monitored using a motion capture system. Next, a motion capture system according to one or more embodiments will be described in more detail.

[0026] An optical system (or optical motion capture system) uses data captured from an image sensor to triangulate the 3D position of a subject between two or more cameras calibrated to provide overlapping projected images. The data can be captured using markers attached to the actor (e.g., human operator 202). Tracking or expanding the capture area for multiple performers can be achieved by increasing the number of cameras.

[0027] In optical systems, occlusion occurs when a model marker point is not located within a large number of points, for example, when the number of observation points is less than the number of model marker points. Such events can be caused by a variety of reasons, such as the intervention of one or more opaque objects between the marker and the camera, or when the marker moves out of the camera's field of view (FOV).

[0028] According to at least one embodiment, the drift compensation system includes one or more embodiments of an optical motion capture system (e.g., in a hybrid configuration) along with one or more embodiments of an inertial motion capture system. Such a system can enable positionally correct and occlusion-free motion capture within a capture area (such as environment 200a) that does not drift over a period of time.

[0029] Embodiments of the inertial motion capture system include the use of inertial body capture suits known in the art. The inertial body capture suit may be of a type that facilitates live motion capture of a human operator 202, as described above with reference to Figure 2A. For example, such an inertial body capture suit estimates the operator's position from an initial starting point by measuring the joint angles of the human operator 202, for example, by counting steps. In this regard, since the inertial motion capture system produces approximate results based on a magnetometer included in the suit, errors may accumulate over time. For example, the inertial motion capture system may approximate the true north position, but errors may arise due to the effect of natural magnetic interference on the accuracy of the magnetometer.

[0030] To help compensate for phenomena such as occlusion and artifacts caused by magnetic interference, the system can utilize both inertial motion capture and optical systems. According to at least one embodiment, the optical system would be simpler than that of an optical system typically used for full-body motion capture. For example, the optical system can be used to focus on the actor's waist. According to at least one embodiment, the optical system includes using a rigid device (or rigid body) wearable by a human operator 202. According to at least one embodiment, data from the inertial motion capture system and data from the optical system are combined (or fused together) and used to drive a digital character (e.g., digital character 106).

[0031] For example, data relating to rotation and height from the ground (see the Y-axis in Figure 2A) can be received from the inertial body capture suit of the inertial motion capture system, while data relating to the horizontal plane (see, for example, the plane defined by the x-axis and Z-axis in Figure 2A) can be received from the optical system.

[0032] As described above, the optical motion capture system may include a rigid body. The rigid body can be stably mounted on a drift-compensated human operator 202. According to at least one embodiment, the rigid body includes markers placed thereon. For example, the markers may be retroreflective markers. Such markers may be arranged on the rigid body in a unique pattern such that the rigid body has a unique identity (e.g., the configuration or arrangement of the markers) that makes it recognizable or identifiable by the optical motion capture system.

[0033] Figure 3 shows an embodiment of the rigid body 502 according to at least one embodiment. As illustrated, the rigid body 502 can take the form of a belt. The belt is configured to be worn around the waist of a person (e.g., a human operator 202). The rigid body 502 is relatively small in size, and as the wearer of the rigid body moves, the rigid body moves with the wearer's waist without moving independently of the wearer. The rigid body 502 can be made of an extremely rigid material and can have a uniform width (e.g., 1 or 2 inches). Markers 506 can be positioned along the perimeter of the rigid body 502. As described above, the markers 506 can be positioned on the rigid body in a specific pattern. For example, in the embodiment illustrated in Figure 3, a total of 25 markers 506 are arranged along the perimeter of the rigid body 502 in a specific zigzag pattern.

[0034] Alternatively, instead of positioning the marker 506 directly on the belt, the marker 506 can be (indirectly) positioned on the belt via one or more rigid bodies that are (directly) positioned on the belt. In this alternative embodiment, the marker 506 can be placed directly on the rigid bodies, and the rigid bodies are placed on the belt. For example, five markers 506 can be placed on each of a total of five rigid bodies, and the rigid bodies are placed on the belt. The arrangement of the rigid bodies on the belt (and the arrangement of the markers on the rigid bodies) can be a unique pattern.

[0035] It is understood that a single optical marker may suffice to achieve an acceptable level of drift correction. However, according to at least one embodiment, two or more optical markers are used to achieve a higher level of robustness against occlusion.

[0036] With respect to the optical system, various embodiments will be described here with reference to the “external” system. According to such embodiments, an array of two or more cameras is installed near the periphery of the capture area (e.g., a room). The cameras capture a view of the capture area and one or more markers (e.g., marker 506 on rigid body 502) as a rigid body (worn by a human operator) moves around the vicinity of the capture area. A stationary camera observes the moving marker and thus provides data (e.g., precise rotation and translation) about the moving marker, which is resolved in computer animation software into a rigid object (e.g., computer-generated bone).

[0037] Figure 4 shows a 3D view of the capture area (room) 400. In at least one embodiment, one or more cameras are installed on each tower located within the capture area 400. For example, as illustrated in Figure 4, four towers 402 are arranged within the capture area 400, with one tower 402 at each corner. For illustrative purposes, each tower 402 may be an 8-foot-high box truss including an aluminum plate 2 feet wide and 1 inch thick. Each tower 402 may house three cameras 406. Within one tower 402, the cameras 406 can be spaced apart from each other—for example, about 2 feet apart along the height of the tower.

[0038] Camera 406 can reside on a small aluminum plate inserted into a box truss. A geared three-way positioning head can also reside on the plate. Camera 406 can be mounted on the positioning head, which can be geared to set up for precise rotation and lock into place once the head is aimed. As described above, tower 402 is positioned at the corner of the capture area 400.

[0039] Each tower 402 generates its own unique data. During the calibration of the camera 406, a stationary object (e.g., a rigid body 502) can be positioned in the center of the capture area 400. The camera 406 can be simultaneously calibrated by configuring the optical system to triangulate from the stationary object. This allows the position of the stationary object in the room to be determined. As described above, data relating to the horizontal plane (e.g., see the plane defined by the x and Z axes in Figure 2A) can be received from the optical system. Such data can be combined (or fused) with data relating to rotation and height from the ground (e.g., see the Y axis in Figure 2A), and this data is received from the inertial body capture suit of the inertial motion capture system. The combination of data can be used to generate animations of a digital character. For example, rotation data received from the inertial coordinate system can be used to animate the skeleton of a digital character, and data received from the optical system can be used to correct the position of the digital character's waist.

[0040] With respect to optical systems, various embodiments have been described with reference to “external” systems. In at least one other embodiment, a “proprietary” system is used. In such a system, a camera is worn by a human operator to monitor markers located within a capture area (e.g., across the ceiling, floor, and / or walls of a room) according to a proprietary pattern. As the human operator moves around the room, the camera moves with the human operator, monitoring stationary markers to generate accurate estimations of the proprietary rotation and position within the room relative to fixed markers on the walls, floor, or ceiling of the room.

[0041] As described above, the markers placed on the rigid body and / or belt can be retroreflective markers. According to at least one other embodiment, they can be light-emitting elements (LEDs) that emit light of wavelengths detectable by an optical motion capture system.

[0042] As described above, the drift compensation system is generated by combining (or hybridizing) an aspect of an optical motion capture system with an aspect of an inertial motion capture system. According to at least one other embodiment, the drift compensation system is created by combining (or hybridizing) an aspect of a magnetic motion capture system with an aspect of an inertial motion capture system. Such an embodiment can operate in a similar manner to the method described above with respect to embodiments using an optical motion capture system. For example, a magnetic "marker" can be placed on a human operator. Magnetic detection systems can be placed at both ends of the room. The magnetic detection systems can emit a magnetic field and detect magnetic interference induced in the magnetic field by the magnetic marker. The detected interference can be used to determine the position of the magnetic marker.

[0043] In at least one embodiment, a technique using Wi-Fi technology can be utilized. Such an embodiment can operate similarly to the method described above with respect to embodiments using a magnetic motion capture system. For example, one or more Wi-Fi antennas can be arranged around a room, and a Wi-Fi emitter can be placed on a human operator. As the human operator moves within the room, the Wi-Fi emitter can move closer to or further away from one antenna or the other. The characteristics of the signals received by the antennas can be used to determine the position of the emitter within the room.

[0044] In at least one embodiment, a technique using acoustic technology can be utilized. Such an embodiment can operate similarly to the method described above with respect to embodiments using an optical motion capture system. In this regard, at least one of two configurations can be used. For example, in one configuration, microphones are positioned on a human operator and ultrasonic emitters are positioned around the room. In another example, in another configuration, emitters are positioned on an actor and microphones are positioned around the room. In embodiments using acoustic technology, time of flight (ToF) can be used to determine the position of whatever object is positioned on the human operator (e.g., microphone or emitter). Similar to how echolocation works, the emitter emits an ultrasonic ping sound, and the microphone (or microphone array) detects the ping and calculates the distance the ping sound travels based on the delay (time of flight) from the emitter to the microphone.

[0045] In at least one embodiment, one or more depth cameras can be used to determine the position of a human operator in a room. For example, the depth camera can project an infrared (IR) grid and then detect how the human operator distorts the grid to determine where the operator is and the movement made by the operator. In another embodiment, an array of IR LEDs can flash sequentially, and depth calculations can be performed by performing a Time of Flight (TF) calculation as the light from the LEDs lights up, reflects off the human operator, and returns to the depth camera.

[0046] In at least one embodiment, the use of markers is not required. For example, in a simpler system, one or more video cameras can be used to track a human operator or specific features on a rigid body to determine the position of the human operator in a room.

[0047] Referring to the environment 300 in Figure 5, in the physical realm, the box 302 containing the cookies is supplied via output channel 108. For example, system 102 may include a dispenser that dispenses items when controlled (e.g., by a human operator 202).

[0048] As described above with reference to Figure 1, System 102 includes at least one video capture device (e.g., a camera) and at least one audio capture device (e.g., a microphone). The camera and microphone can exist in a virtual world, which the customer can experience in virtual reality. For example, the customer can control a digital avatar that exists in the digital realm via an input device such as a joystick, mouse, or virtual reality (VR) or motion capture input. Interactions occur in the digital realm, with the exception of the sale / supply of items (e.g., box 302) which still occur in the physical realm. For example, the customer plays a game, and the game interacts with a game character corresponding to the customer, but does not interact directly with the customer. In this scenario, the customer enters and experiences the digital realm as a digital avatar, and the movement of the digital avatar is controlled by the customer based on inputs provided in the physical realm. Thus, the avatar can be controlled to move in the digital realm in some way (e.g., walking, jumping, flying, teleporting). When the presence of a digital avatar is detected in the digital realm, interaction with the digital avatar is initiated in the same way that interaction with a customer is initiated when the presence of a customer is detected in the physical realm (e.g., environment 100). When the customer triggers a change occurring in the physical realm (e.g., supply of an item), the change is realized in the physical realm.

[0049] Furthermore, as described above with reference to Figures 1, 2A, 2B, 3, and 4, a digital character (e.g., digital character 106) can be created via live motion capture of a human operator (e.g., human operator 202). The live motion capture drive motion can be applied to a specific part (or aspect) of the digital character (or an object such as box 302 in Figure 5), or, when appropriate, to drive the entire digital character or props. According to one or more embodiments, the human operator drives the performance of the digital character by digital puppetry using devices such as a real-time digital interface for musical instruments (MIDI) input and / or a controller, keyboard, joystick, mouse, foot pedal, microphone, etc. In such embodiments, live motion capture may not be used exclusively. For example, motion capture can be used to drive the performance of the digital character's face, and the remaining aspects of the performance are driven using triggered input devices. According to one or more other embodiments, the digital character 106 can be created via live motion capture of a human operator.

[0050] According to one or more other embodiments, the performance and decision-making related to the digital character are entirely generated by an artificial intelligence (AI) character residing in the game engine.

[0051] According to one or more other embodiments, performance and decision-making regarding a digital character can be achieved using a combination of an AI game engine-driven character and human motion capture. For example, during a specific period of time, for instance, while the system is waiting for a customer to engage, the digital character can be at least partially driven by the AI ​​(e.g., by playing a loop of environmental behavior or other character-based animation). At another time, a transition from AI-driven to motion capture-driven operation can occur. For example, when the system detects the presence of a customer 110, the performance of the digital character (e.g., animation control) can be transferred to a human operator, who can then interact with the customer in a more familiar way. Upon reaching such a transition, the system 102 can signal to the human operator that at least partial control over the digital character's performance will be given. For example, the system 102 can provide audio and / or video indicators to inform the human operator that control will be given over the digital character's voice performance, the digital character's facial performance, or the digital character's overall performance.

[0052] Similarly, a transition can occur from motion capture-driven to AI game engine technology-driven. Decisions made by such AI technology can be based on the analysis of data captured by the system 102's video capture device (e.g., camera) and / or audio capture device (e.g., microphone). For example, the analysis of data captured by the camera can be used to identify not only the positions of various customers 110, but also specific characteristics of a particular customer, such as the blue clothing (e.g., shirt) worn by one customer 110 or the blonde hair of another customer 110. Thus, even when the performance of a digital character is driven by AI game engine technology (rather than by a human operator), the digital character can still interact with customers 110 to some extent. For example, the eyes of a digital character can be driven to appear as if the digital character's "gaze" is following movements made by the customer 110. Also, for example, the digital character can be driven to wave to a particular customer 110 and / or to give a somewhat customized or personalized greeting to a customer 110 wearing blue clothing or a customer 110 with blonde hair.

[0053] Similarly, analysis of data captured by the microphone can be used to interact with the customer 110 to at least some extent. For example, if such analysis detects a sound as having certain characteristics (e.g., a sound exceeding a certain threshold audio level), the performance of the digital character can be selectively driven by a specific clip of (prepared) animation. In such a clip, the digital character's eyes can be moved to appear as if shifting its gaze upward and outward, as if searching for the source of the sound it heard. As another example, waveform analysis can be performed on speech audio captured by the microphone. Such analysis can be used to identify the mood or emotional state of the customer 110 who was speaking. In this situation, the mood in which the digital character sends a voice greeting to the customer 110 can be selected to match or reflect the mood identified by the waveform analysis. For example, if the customer 110's mood is identified as happy or cheerful, the digital character can be controlled to interact with the customer 110 in a happy or cheerful manner.

[0054] Therefore, AI game engine technology can be used to effectively execute specific branches in a decision tree (e.g., using selected clips of animation to drive the performance of a digital character) in response to specific triggers (e.g., the detection of sounds with specific characteristics). This increases the likelihood that the digital character will appear alive and be able to react to real-world events occurring within the physical domain (e.g., environment 100). This can be preferable to the performance of a digital character where the character simply repeats specific actions while appearing unaware of events occurring within the physical domain.

[0055] Therefore, the performance of a digital character does not always need to be driven by a human operator. For example, the performance of a digital character can be driven by AI game engine technology for some (or most) of the time, and by a human operator for selected times (e.g., one or two or more critical times). In another embodiment, the performance of a digital character can be driven by AI game engine technology for most of the day (e.g., 11 hours total in a 12-hour day), and control of the digital character can be assigned to a human operator for selected periods (e.g., selected times during such a day's operation). The period can be chosen to provide a more personalized interaction with the customer 110 during that period. At the end of such a period, the human operator can return control of the digital character's performance to the AI ​​game engine technology. This return can be chosen to occur all at once to provide a more seamless transition from the customer 110's perspective. The human operator can return control of the digital character's performance at that time, for example, by manually operating an on-screen interface, keyboard triggers, buttons, or joystick inputs.

[0056] In at least one embodiment, the performance of a digital character can be driven simultaneously, at a given time, by a human operator and by AI game engine technology. For example, if the digital character is at least partially driven by AI in its interaction with customer 110, the human operator may want to enhance the digital character's performance. In this situation, the human operator can control the AI-driven performance in one or more ways. For example, if the human operator identifies customer 110's mood not merely as happy or cheerful, but specifically as happy or cheerful, the human operator can control the digital character to deal with customer 110 in a similarly enhanced manner. Such control can be performed, for example, by manually manipulating an on-screen interface, keyboard triggers, buttons, or joystick inputs. In this way, the performance of a digital character can be (temporarily) guided, at least partially, by a human operator.

[0057] According to embodiments described in more detail below, multiple digital characters can be displayed on the display device 104. Each of these characters can be controlled via various combinations of AI game engine-driven characters and human motion capture, as described above.

[0058] As described above with reference to various embodiments, actions (or non-actions) occurring within the environment 100, particularly actions (or non-actions) performed by the customer 110, are captured, for example, by cameras and microphones. When the presence of the customer 110 is detected within the environment 100 (e.g., one or more specific areas within the environment 100), interaction with the customer is initiated. It is understood that detection can be performed additionally and / or alternatively using other devices. Such other devices include pressure pads, depth cameras, acoustic range detection devices, optical beam devices, and / or thermal or sound detection devices.

[0059] The device can detect not only the presence of a customer but also other parameters such as the customer's head height and body position. For example, the customer's body position and height can be detected using a video camera, depth sensor, or a broken light or sound wave beam. Recognition of such information (e.g., related to the position of the display device 104) allows for angle correction to be performed on the head or eyes of the digital character 106, so that the digital character's gaze can be more closely aligned with the gaze of one or more customers. Such a device can effectively "adjust" the gaze of the digital character by more accurately detecting the customer's head height.

[0060] Furthermore, as described above with reference to various embodiments, the interaction includes one or more interactions between the digital character and the customer in both the digital and physical domains. For example, in the digital domain, the digital character can perform the step of preparing or procuring an item (e.g., a box of cookies 302) for the customer. In the physical domain, the item is given (or sold) to the customer in real time, and the item is made available for the customer to take out or collect (e.g., via output channel 108). According to one or more further embodiments, the item is presented so that the customer can take out the item at another location (e.g., somewhere outside environment 100). According to one or more further embodiments, the item is supplied to the customer at a later date or later (e.g., by regular mail).

[0061] According to one or more other embodiments, the interaction between the digital character and the customer includes interactions that cause (or bring about) changes in the physical domain of the environment 100. For example, system 102 may include a pulley to facilitate the performance of a tug-of-war game between the digital character and the customer.

[0062] For example, a human operator 202 controlling the performance of a digital character asks, "Is there anyone who wants to play tug-of-war with me?" Thus, in the digital realm, the digital character 106 asks customer 110, "Is there anyone who wants to play tug-of-war with me?" In the physical realm, customer 110 can respond accordingly. For example, one customer 110 might raise their hand. In another example, another customer 110 might shout, "Yes!"

[0063] In response to viewing and / or hearing the response made by customer 110, human operator 202 may move within environment 200a as if human operator 202 were taking out the rope for the game to be played. Subsequently, human operator 202 may move within environment 200b as if human operator 202 were moving the rope that has been taken out toward one or more waiting customers 110.

[0064] The actions of the human operator 202 are reproduced in the digital domain by the digital character 106. For example, in the digital domain, the digital character 106 takes out a rope and then moves toward the output channel 108.

[0065] Within the physical domain, the free end of the rope is stretched outward through the output channel 108. The other end of the rope can be coupled to a pulley system hidden from the view of customer 110. After one of the customers grabs the rope, a tug-of-war game can begin.

[0066] According to one or more further embodiments, other embodiments of interactions that cause changes in the physical domain of environment 100 include objects in the physical domain that are moved (e.g., flipped over) in response to actions (e.g., tripping or falling) by a digital character in the digital domain, lights in the physical domain that are controlled to turn on / off in response to actions by a digital character in the digital domain, and any other aspects in the physical domain that are changed in response to actions by a digital character in the digital domain.

[0067] Similarly, customer actions in the physical domain detected by system 102 may result in changes in the digital domain. For example, a customer's facial response or expression, their posture, other movements made by the customer, or an audio response or sound made by the customer can be detected, and the related events can be generated in the digital domain.

[0068] Interactions between the physical and digital domains are not limited to interactions involving customers and digital characters. For example, the presence of one or more specific objects in the detected physical domain may result in changes in the digital domain, and vice versa. For instance, these changes may include corresponding or related objects in the digital domain. As an example, an object moved intentionally or accidentally by a customer in the physical domain may be followed by the movement of a corresponding or related object by a digital character in the digital domain.

[0069] Inducing changes within the physical domain of environment 100 can be done to increase the likelihood that the customer will interact with one or more digital characters in the digital domain. If no customer interaction is detected and the performance of the digital characters is at least partially created by the AI, system 102 can enter a loop state that continuously (or periodically) monitors for detected interaction. Alternatively (or additionally), system 102 can begin displaying tracking animations to motivate the customer to engage with the digital characters. Alternatively (or additionally), system 102 can return to a waiting state where a specific set of animations is displayed until the presence of a customer is detected.

[0070] As described above with reference to various embodiments, an object such as a cookie box can be supplied for customer pickup (e.g., via output channel 108). In this scenario, the object can be supplied relatively soon after a customer request (e.g., the customer saying "yes!") is detected. In other scenarios, supply can occur after a certain delay. For example, a toy sword can be supplied to a customer. In the physical domain, a pre-made toy sword can be picked up and placed at the receiving end of the output channel. However, in the digital domain, a digital character can forge the toy sword (e.g., from raw materials) at a blacksmith. In such a scenario, an animation loop (or sequence) can be displayed on a display device (e.g., display device 104) to show the forging of the toy sword. The display of such a loop can continue until feedback is received in the physical domain indicating that the supply of the toy sword will occur. Once such feedback is received, the display of the loop ends, and the forging of the toy sword is completed in the digital domain. Examples of such feedback are described in more detail below.

[0071] In another embodiment, the object can be an item that requires some preparation time within the physical domain. For example, the object can be a food item requested (or ordered) by a customer (e.g., a bowl of noodle soup).

[0072] The presence of a customer can be detected before a request is received. As described above with reference to various embodiments, the movement of one or more customers (e.g., in environment 100) can be detected. When one or more movements (e.g., one or more specific movements such as movement toward display device 104) are detected, an action is performed to solicit customer engagement. For example, when system 102 is designed to visually resemble the outside of a kitchen or food truck, a sequence (e.g., a game sequence) is initiated so that display device 104 displays one or more digital characters preparing food. For example, in the digital realm, the digital characters chop vegetables, cook noodles, and so on.

[0073] The representation of one or more items in the digital realm, such as a cooking spoon or pot, can be controlled in several ways. For example, an item may have a corresponding physical object, such as a spoon (e.g., made of plastic or foam) held by an actor (e.g., human operator 202 in Figure 2A) that can be tracked by motion capture. In another embodiment, the representation of an item can be controlled digitally when the item exists within a game engine, such as an object in a video game. In this scenario, the actor can interact with the item by making a grasping gesture and placing their hand near the item to alert the game engine that they want to pick up the item. The actor can also trigger a grasping gesture via a controller or button press. In another embodiment, the actor can trigger the item to appear in the actor's hand from an invisible location by pressing a button or reaching into a specific 'zone'.

[0074] As described above with reference to Figure 2B, in the digital realm, the digital character 106 takes a box of cookies. The actor can achieve this takeaway of the box by reaching into the air so that the digital character 106's hand enters an invisible cube in the digital realm that is outside the frame (e.g., just outside the frame) relative to the display (e.g., display 104). When the digital character 106's hand enters this zone, the box is positioned in the actor's hand in the digital realm. When the actor lowers their arm in the physical realm, the box appears in the digital realm in the digital character 106's hand, as if the digital character 106 were reaching off-screen to pull the box from an invisible shelf.

[0075] In another embodiment, a combination of one or more items in the digital realm and one or more items in the physical realm is utilized. For example, such a combination can be used to achieve the shredding of a carrot in the digital realm by a digital character. This combination may include a physical counterpart (e.g., a physical prop) in the digital realm corresponding to a knife held by an actor and in the digital realm, and a carrot that exists only in the digital realm. The movement of the physical prop is tracked so that the knife appears in the digital character's hand and moves accordingly. As the tip of the knife moves near the carrot in the digital realm, the carrot is shown to be sliced ​​into multiple pieces. As the side of the knife is brought near the carrot pieces, the carrot pieces can be shown to be moved away from the edge of the cutting board and into a pot. This movement of the carrot pieces can also be shown to be directly caused by the digital character's hand (e.g., by an actor gesturing to sweep the carrot pieces into the pot). However, to achieve a higher degree of realism, it is understood that only the knife (and not the digital character's hand) can slice the carrot into multiple pieces.

[0076] Triggered movements can also be achieved in a similar manner. Such movements may include, for example, martial arts movements that are not easily performed by a typical novice. These movements can be triggered by pressing a button on a controller that merges a live movement with one or more pre-prepared movements, or by an actor performing a gesture (e.g., an alternative gesture) that initiates the triggered movement when recognized by the system. For example, an actor may kick their leg, and the system may recognize this kick as the trigger for a special kung fu jump kick sequence. According to further embodiments, the triggered movement does not entirely drive the performance of the digital character. For example, only one or more parts of the digital character's body may be driven by the triggered movement. As an example, while the triggered movement is performed by the torso, arms and / or legs of the digital character, the face of the digital character can still be controlled by the actor (e.g., via motion capture).

[0077] Returning to customer detection, requests from detected customers are received. Receiving requests may involve using natural language processing (to receive and process verbal requests). Alternatively (or additionally), receiving requests may involve using a customer-operable interface. For example, the customer may operate buttons on a touchscreen or fill out an instruction card that is inserted into and read by the machine.

[0078] After a customer requests a specific item, the preparation of that item begins within the physical domain. For example, a food cook (e.g., a chef) located near environment 100 begins preparing the requested item.

[0079] Simultaneously, actions representing the preparation of an item occur in the digital realm. Such actions can be customized based on customer requests. For example, based on a customer request that noodle soup include extra carrots, the sequence performed could include a display of carrots flying around the entire kitchen in the digital realm.

[0080] The animation displayed on the display device 104 can be continuously controlled while the cooking of the requested item in the physical domain is in progress. For example, during the cooking of the requested item, feedback can be received from the food cook indicating that more time is needed before cooking is complete. Such feedback can be used to display further loops of a particular sequence. It is understood that the animation displayed on the display device 104 can be controlled simultaneously by AI game engine technology and by a human operator, as can be the way in which the performance of a particular digital character can be controlled as described above with reference to various embodiments.

[0081] In addition, the animation can continue to be controlled as the requested item approaches completion or when it is completed. For example, feedback can be received from the food cook indicating that the item is ready for output. The feedback can originate from a pressure sensor that senses that an item (e.g., the requested item) is placed in a determined location. Thus, the food cook can place the requested item near or on the pressure sensor to provide feedback indicating that the item is ready for output. As another example, system 102 may include an interface operable by the food cook. Thus, the food cook can operate a button on a touchscreen to signal that the item is ready for output.

[0082] Within the physical domain, items can be placed on output channel 108 in preparation for retrieval by the customer.

[0083] Based on the described features, the timing of events occurring in the digital domain (e.g., the creation of an item by a digital character, as shown in a displayed animation) can be better aligned with the timing of events occurring in the physical domain (e.g., the preparation of an item by a food cook). For example, the timing in both domains can be better aligned so that when a bowl of noodle soup disappears from view in the digital domain, it reappears in the physical domain of output channel 108. According to one or more specific embodiments, the items presented in output channel 108 are selected so as to match the appearance of the items shown in the digital domain.

[0084] Figure 6 shows a flowchart of a method 600 for controlling the performance of a digital character displayed on a display device, according to at least one embodiment.

[0085] In block 602, according to certain embodiments, the performance of a digital character can be controlled using AI game engine technology. For example, referring to Figure 1, the performance of digital character 106 is controlled using AI game engine technology.

[0086] In block 604, the presence of a customer located within the physical environment is detected. For example, referring again to Figure 1, the presence of a customer 110 located within the physical environment 100 is detected.

[0087] According to a further embodiment, the detection of a customer's presence is performed based on data received from at least a camera or microphone located within the physical environment.

[0088] In a further embodiment, the step of detecting the presence of a customer may include a step of autonomously determining the customer's physical characteristics or emotional state. For example, data received from a camera can be used to determine that customer 110 is wearing blue clothing. In another example, data received from a microphone can be used to detect and determine whether customer 110 is happy or sad.

[0089] In block 606, in response to the detection of a customer's presence, the performance (or at least a mode thereof) of the digital character by a human operator, an AI game engine, or a combination thereof is facilitated. For example, referring to Figure 2A, in response to the detection of a customer's presence 110, the performance of the digital character 106 by the human operator 202 is facilitated.

[0090] In a further embodiment, the step of facilitating control of the digital character's performance may include providing at least one selectable option by a human operator. The selectable option is for controlling the digital character to interact with the customer according to a determined characteristic or determined emotional state. For example, if a human operator 202 identifies that customer 110 is particularly happy or cheerful, the human operator can control the digital character 106 to interact with customer 110 in such a particularly happy or cheerful manner.

[0091] In a further embodiment, the step of facilitating control of the digital character's performance may include the step of receiving motion capture data corresponding to a human operator. For example, the step of receiving motion capture data may include receiving data from an optical motion capture system (e.g., a system utilizing camera 406 in Figure 4) and receiving data from an inertial motion capture system (e.g., a system utilizing an inertial body capture suit worn by a human operator 202).

[0092] In a further embodiment, the data received from the optical motion capture system may include data corresponding to one or more optical markers located in the waist region of the human operator. For example, the data received from the optical motion capture system may include data corresponding to marker 506 located in the waist region of the human operator.

[0093] In a further embodiment, control of at least a portion of the performance of a digital character by a human operator is facilitated so that the performance of the digital character is driven simultaneously by the human operator and the AI ​​game engine technology.

[0094] In block 608, according to a particular embodiment, a request is received from a customer. For example, referring to Figure 1, customer 110 may request that one box of cookies be provided.

[0095] In block 610, a request can be processed by causing a change in the physical environment in which the customer is located. For example, a request can be processed by providing a physical object to be delivered for the customer to pick up. Referring to Figure 5, for example, a box 302 containing cookies is provided and delivered for the customer 110 to pick up.

[0096] In selected embodiments, the features and aspects described herein can be implemented within a computing environment 700 which may include one or more computer servers 701, as shown in Figure 7. A server 701 may be operably coupled to one or more data stores 702 (e.g., databases, indexes, files, or other data structures). A server 701 may be connected to a data communications network 703 which may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a telephone network, a satellite or wireless communication network, or any combination thereof.

[0097] One or more client devices 704, 705, 706, 707, 708, 709, 710 can communicate with server 701 and corresponding data store 702 via data communication network 703. Such client devices 704, 705, 706, 707, 708, 709, 710 may include, for example, one or more laptop computers 707, desktop computers 704, smartphones and mobile phones 705, tablet computers 706, televisions 708, motion capture sensors 709, cameras 710, or combinations thereof. During operation, such client devices 704, 705, 706, 707, 708, 709, 710 may send or receive data or commands to server 701 in response to user input received from user input devices or other inputs. In response, server 701 may process data from datastore 702, modify data within datastore 702, add data to datastore 702, or similar actions, or a combination thereof.

[0098] In a selected embodiment, the server 701 can send one or more media files containing audio and / or video content, encoded data, generated data, and / or metadata from the data store 702 to one or more of the client devices 704, 705, 706, 707, 708, 709, and 710 via the data communication network 703. The devices can output the audio and / or video content from the media files using a display screen, projector, or other display output device. In a particular embodiment, the system 700 configured according to the features and aspects described herein can operate in or be configured to support a cloud computing environment. For example, some or all of the data store 702 and the server 701 can reside in a cloud server.

[0099] Referring to Figure 8, an illustrative example of a computer 800 is shown. One or more of the devices 704, 705, 706, 707, and 708 of system 700 can be configured as such a computer 800, or may contain such a computer 800.

[0100] In selected embodiments, the computer 800 may include a bus 803 (or multiplex bus) or other communication mechanism, a processor 801, main memory 804, read-only memory (ROM) 805, one or more additional storage devices 806, and / or a communication interface 802, or similar, or a partial combination thereof. The embodiments described herein may be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microphone controllers, microphone processors, other electronic units designed to perform the functions described herein, or a selective combination thereof. In all embodiments, the various components described herein may be implemented as a single component or alternatively as various separate components.

[0101] Bus 803 or other communication mechanism may include multiple such buses or mechanisms and may support the communication of information within the computer 800. Processor 801 may be connected to bus 803 and process information. In a selected embodiment, processor 801 may be a specific or dedicated microphone processor configured to perform a specific task according to the features and aspects described herein by executing machine-readable software code that defines a particular task. Main memory 804 (e.g., random access memory or RAM or other dynamic storage device) may be connected to bus 803 and may store information and instructions executed by processor 801. Main memory 804 may also store temporary variables or other intermediate information during the execution of such instructions.

[0102] ROM 805 or some other static storage device can be connected to bus 803 to store static information and instructions for processor 801. Further storage devices 806 (e.g., magnetic disks, optical disks, memory cards, etc.) can be connected to bus 803. Main memory 804, ROM 805 and further storage devices 806 may include non-temporary computer-readable media that hold information, instructions, or any combination thereof, for example, instructions that, when executed by processor 801, cause computer 800 to perform one or more operations in the manner described herein. Communication interface 802 can also be connected to bus 803. Communication interface 802 can provide or support bidirectional data communication between computer 800 and one or more external devices (e.g., other devices housed within the computing environment).

[0103] In a selected embodiment, the computer 800 may be connected to the display 807 (for example, via the bus 803). The display 807 may use any suitable mechanism to convey information to the user of the computer 800. For example, the display 807 may include or utilize a liquid crystal display (LCD), a light-emitting diode (LED) display, a projector, or other display device to present information to the user of the computer 800 in a visual display. One or more input devices 808 (for example, an alphanumeric keyboard, a mouse, a microphone) may be connected to the bus 803 to communicate information and commands to the computer 800. In a selected embodiment, one input device 808 may provide or support control of cursor positioning to enable the selection and execution of various objects, files, programs, etc., provided by the computer 800 and displayed by the display 807.

[0104] Computer 800 can be used to transmit, receive, decode, display, and otherwise transmit one or more video files. In a selected embodiment, such transmission, reception, decoding, and display may be in response to a processor 801 executing one or more sequences of one or more instructions contained in main memory 804. Such instructions may be read into main memory 804 from another non-temporary computer-readable medium (e.g., a storage device).

[0105] The execution of an instruction sequence contained in main memory 804 can cause the processor 801 to perform one or more of the procedures or steps described herein. In a selected embodiment, one or more processors in a multiprocessing configuration can also be used to execute the instruction sequence contained in main memory 804. Alternatively or additionally, firmware can be used in place of or in connection with software instructions that perform procedures or steps according to the features and embodiments described herein. Thus, embodiments according to the features and embodiments described herein can be made less limited to any particular combination of hardware circuits and software.

[0106] Non-temporary computer-readable media can refer to any medium that is involved in holding instructions to be executed by processor 801 or that stores data to be processed by a computer, and can include all computer-readable media except for temporary propagated signals. Examples of such non-temporary computer-readable media include, but are not limited to, non-volatile media, volatile media, and temporary storage media (e.g., cache memory). Non-volatile media can include optical or magnetic disks, such as further storage devices. Volatile media can include dynamic memory, such as main memory. Common forms of non-temporary computer-readable media can include, for example, hard disks, floppy disks, magnetic tapes, or any other magnetic media, CD-ROMs, DVDs, Blu-rays, or optical media, RAM, PROMs, EPROMs, flash EPROMs, any other memory cards, chips, or cartridges, or any other memory media that can be read by a computer.

[0107] In a selected embodiment, the communication interface 802 can supply or support external bidirectional data communication to or over a network link. For example, the communication interface 802 may be a wireless network interface controller or cellular radio providing data communication network connectivity. Alternatively, the communication interface 802 may include a LAN card providing data communication connectivity with a compatible LAN. In any such embodiment, the communication interface 802 may send and receive electrical, electromagnetic, or optical signals that transmit information.

[0108] A network link can communicate data to other data devices (e.g., client devices as shown in computing environment 700) via one or more networks. For example, a network link can provide connection to a data facility operated by an Internet Service Provider (ISP) via the host computer's local network. The ISP can then provide data communication services over the Internet. Thus, computer 800 can send and receive commands, data, or combinations thereof, including program code, via one or more networks, network links, and communication interface 802. In this way, computer 800 can communicate with a remote server (e.g., server 701) or any combination thereof via interface connection or other means.

[0109] The various devices, modules, terminals, and similar entities described herein can be implemented on a computer by the execution of software containing machine language instructions read from a computer-readable medium, as discussed above. In certain embodiments, some hardware configurations can be implemented using a single computer, while in other embodiments, the system can be implemented using multiple computers, input / output systems, and hardware.

[0110] With respect to the software implementation, the specific embodiments described herein may be implemented in separate software modules, such as procedures and functions, each of which may perform one or more of the functions and operations described herein. The software code may be implemented in a software application written in any suitable programming language, stored in memory, and executed by a controller or processor.

[0111] The embodiments and features described above are merely illustrative and should not be construed as limiting the invention. The teachings of the invention can be readily applied to other types of apparatus and processes. Such descriptions of embodiments are intended for illustrative purposes and do not limit the scope of the claims. Many alternative, modified, and variant forms will be apparent to those skilled in the art. [Explanation of Symbols]

[0112] 602: Using artificial intelligence (AI) game engine technology to control the performance of digital characters 604: Detect the presence of a person located within the physical environment. 606: Facilitate control of digital character performance through human operators, AI game engines, or a combination thereof. 608: Receiving a request from someone 610: Processing requests by causing changes within the physical environment.

Claims

1. A method for enabling a digital character displayed on a display device to interact with a first person located in a physical environment, The step of receiving a request from the first person, The steps of providing interaction between the digital character and the first person by controlling the performance of the digital character shown on the display device by a second person, or by an artificial intelligence (AI) game engine, or a combination thereof, During the interaction between the digital character and the first person, the process includes the step of causing a change in the physical environment in which the first person is located in response to the actions of the digital character shown on the display device, The aforementioned change in the physical environment is associated with a physical object different from the display device. The step of causing a change in the physical environment includes processing the request by providing a physical object different from the display device to be sent out for retrieval by the first person, A physical object different from the aforementioned display device includes a pre-created item. The method further includes the step of controlling the display device to display at least one animation loop or sequence indicating the preparation of the items before providing the physical object which includes the pre-prepared items, method.

2. The process further includes the step of determining whether or not the first person is present in the physical environment, The step of providing interaction between the digital character and the first person is performed in response to determining whether or not the first person is present. The method according to claim 1.

3. The method according to claim 2, wherein the determination of whether or not the first person is present is performed based on data received from at least a camera or microphone located within the physical environment.

4. The method according to claim 1, wherein the step of causing the change in the physical environment includes moving or displacing a physical object different from the display device within the physical environment.

5. A physical object different from the display device includes a light source placed in the physical environment. The method according to claim 1, wherein the step of causing the change in the physical environment includes controlling the light source to turn on or off.

6. A method for enabling a digital character displayed on a display device to interact with a first person located in a physical environment, The step of receiving a request from the first person, The steps of providing interaction between the digital character and the first person by controlling the performance of the digital character shown on the display device by a second person, or by an artificial intelligence (AI) game engine, or a combination thereof, During the interaction between the digital character and the first person, the process includes the step of causing a change in the physical environment in which the first person is located in response to the actions of the digital character shown on the display device, The aforementioned change in the physical environment is associated with a physical object different from the display device. The step of causing a change in the physical environment includes processing the request by providing a physical object different from the display device to be sent out for retrieval by the first person, A physical object different from the display device includes an item prepared in the physical environment after the request is received. The method further includes the step of controlling the display device to display at least one animation loop or sequence indicating the preparation of the item before providing the physical object containing the item, method.

7. The claim further includes the step of receiving information indicating that the preparation of the item in the physical environment is not yet complete, The method according to claim 6, wherein the step of controlling the display device to display at least a first animation loop or sequence further includes the step of controlling the display device to display at least a second animation loop or sequence in response to receiving the information.

8. The further step includes receiving information indicating that the preparation of the item in the physical environment is complete, The method according to claim 6, wherein the step of controlling the display device to display at least a first animation loop or sequence includes the step of controlling the display device to display an animation showing an item in a completed state.

9. A device for enabling a digital character displayed on a display device to interact with a first person located in a physical environment, A network communication unit configured to send and receive data, One or more controllers configured to receive requests from the first person and to control the performance of the digital character shown on the display device by the second person, or by an artificial intelligence (AI) game engine, or a combination thereof, thereby providing interaction between the digital character and the first person, and to cause changes in the physical environment where the first person is located in response to actions by the digital character shown on the display device during the interaction between the digital character and the first person, Equipped with, The aforementioned change in the physical environment is associated with a physical object different from the display device. The one or more controllers are further configured to receive a request from the first person, provide a physical object to be sent out for retrieval by the first person, and process the request to cause a change in the physical environment. The physical object, which is different from the display device, includes a pre-created item. The one or more controllers are further configured to control the display device to display at least one animation loop or sequence indicating the preparation of the items before providing the physical object, which includes the pre-generated items. Device.

10. The one or more controllers are further configured to determine whether or not the first person is present in the physical environment. The apparatus according to claim 9, wherein providing interaction between the digital character and the first person is performed in response to determining the presence or absence of the first person.

11. The apparatus according to claim 10, wherein the determination of whether or not the first person is present is performed based on data received from at least a camera or microphone placed in the physical environment.

12. The apparatus according to claim 9, wherein the one or more controllers are further configured to cause a change in the physical environment by moving or displacing a physical object different from the display device within the physical environment.

13. The physical object, which is different from the display device, is composed of a light source placed within the physical environment. The apparatus according to claim 9, wherein the one or more controllers are further configured to cause a change in the physical environment by controlling the light source to turn on or off.

14. A device for enabling a digital character displayed on a display device to interact with a first person located in a physical environment, A network communication unit configured to send and receive data, One or more controllers configured to provide interaction between the digital character and the first person by controlling the performance of the digital character shown on the display device by a second person, or by an artificial intelligence (AI) game engine, or a combination thereof, and to cause changes in the physical environment in which the first person is located in response to actions by the digital character shown on the display device during the interaction between the digital character and the first person, Equipped with, The aforementioned change in the physical environment is associated with a physical object different from the display device. The one or more controllers are further configured to receive a request from the first person, provide a physical object to be sent out for retrieval by the first person, and process the request to cause a change in the physical environment. The physical object, which is different from the display device, includes an item prepared in the physical environment after the request is received. The one or more controllers are configured to control the display device to display at least one first animation loop or sequence indicating the preparation of the item before providing the physical object containing the item. Device.

15. The apparatus according to claim 14, wherein the one or more controllers are further configured to receive information indicating that the items in the physical environment are not ready, and to control the display device to display at least a second animation loop or sequence in response to receiving the information.

16. A machine-readable non-temporary medium that stores machine-executable instructions for enabling a digital character shown on a display device to interact with a first person located in a physical environment, The command receives a request from the first person and, by the second person, or by an artificial intelligence (AI) game engine, or a combination thereof, controls the performance of the digital character shown on the display device, thereby providing interaction between the digital character and the first person, and The interaction between the digital character and the first person includes causing a change in the physical environment in which the first person is located in response to an action by the digital character shown on the display device, The aforementioned change in the physical environment is associated with a physical object different from the display device, Causing a change in the physical environment includes processing the request by providing a physical object different from the display device to be delivered for retrieval by the first person, A physical object different from the aforementioned display device includes a pre-created item. The instruction further includes controlling the display device to display at least one animation loop or sequence indicating the preparation of the items before providing the physical object containing the pre-created items. Machine-readable non-temporary medium.

17. A machine-readable non-temporary medium that stores machine-executable instructions for enabling a digital character shown on a display device to interact with a first person located in a physical environment, The command receives a request from the first person and, by the second person, or by an artificial intelligence (AI) game engine, or a combination thereof, controls the performance of the digital character shown on the display device, thereby providing interaction between the digital character and the first person, and The interaction between the digital character and the first person includes causing a change in the physical environment in which the first person is located in response to an action by the digital character shown on the display device, The aforementioned change in the physical environment is associated with a physical object different from the display device, Causing a change in the physical environment includes processing the request by providing a physical object different from the display device to be delivered for retrieval by the first person, A physical object different from the display device includes an item prepared in the physical environment after the request is received. The instruction further includes controlling the display device to display at least one animation loop or sequence indicating the preparation of the item before providing the physical object containing the item. Machine-readable non-temporary medium.

Images