Interactive video game system

Abstract

An interactive video game system includes at least one sensor and at least one display device disposed near an active area. The system also includes a controller communicatively coupled to the at least one sensor and the at least one display device, wherein the controller is configured to: receive scan data of a participant in the active area from the at least one sensor; generate at least one model based on the scan data of the participant; recognize the movement of the participant in the active area based on the at least one model; generate a virtual representation for the participant based on the at least one model and the movement of the participant; and present the virtual representation of the participant in the virtual environment on the display device, wherein the movement of the virtual representation is enhanced relative to the movement of the participant.

Description

Background Art

[0001] The present disclosure relates generally to video game systems and, more particularly, to interactive video game systems that enable simultaneous multi-player gaming activities.

[0002] Video game systems typically enable participants to control characters in a virtual environment to achieve a predetermined goal or task. Traditional video game systems typically rely on manual input devices, such as joysticks, game controllers, keyboards, etc., to enable participants to control characters within the virtual environment of the game. Additionally, some modern video game systems may include cameras that can track the movement of participants, thereby enabling participants to control video game characters based on their movement. However, these systems typically suffer from the problem of occlusion, in which a portion of a participant is at least temporarily obscured from the camera and, therefore, the system is no longer able to accurately track the position or movement of the participant. For example, occlusion can cause jitter or stuttering of the movement of a character in a virtual environment, as well as other inaccurate or erroneous conversions of participant actions to character actions within the game. Additionally, for multi-participant video game systems, the potential for occlusion increases significantly with the number of participants. Summary of the invention

[0003] The present embodiment relates to an interactive video game system, which includes at least one sensor and at least one display device disposed near an active area. The system also includes a controller, which is communicatively coupled to the at least one sensor and the at least one display device, wherein the controller is configured to: receive scan data of a participant in the active area from the at least one sensor; generate at least one model based on the scan data of the participant; recognize the movement of the participant in the active area based on the at least one model; generate a virtual representation for the participant based on the at least one model and the movement of the participant; and present a virtual representation of the participant in a virtual environment on the display device, wherein the movement of the virtual representation is enhanced relative to the movement of the participant.

[0004] The present embodiment also relates to a method for operating an interactive video game system. The method includes: receiving scan data of a participant positioned in an active area via a processing circuit of a controller of the interactive video game system; generating a shadow model and a skeletal model of the participant based on the scan data via the processing circuit; generating a virtual representation of the participant based on the shadow model via the processing circuit, wherein the virtual representation is associated with an enhanced capability; identifying an action of the participant in the active area based on the skeletal model via the processing circuit, wherein the action triggers an enhanced capability associated with the virtual representation; and presenting a virtual representation in a virtual environment that performs the enhanced capability via a display device of the interactive video game system.

[0005] The present embodiment also relates to an interactive video game system, which includes a controller configured to: receive scan data of a participant in an active area from at least one sensor of the interactive video game system; generate a shadow model and a skeletal model based on the scan data of the participant; generate a virtual representation for the participant based on the shadow model; identify the participant's actions in the active area based on the participant's skeletal model, wherein the actions trigger enhanced capabilities associated with the virtual representation; and present the virtual representation in a virtual environment that executes the enhanced capabilities on a display device of the interactive video game system. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] These and other features, aspects and advantages of the present disclosure will become better understood when the following detailed description is read with reference to the accompanying drawings, wherein like characters refer to like parts throughout the drawings, and wherein:

[0007] Figure 1 is a schematic diagram of an embodiment of an interactive video game system according to the present technology, the interactive video game system enabling multiple participants to control corresponding virtual representations by performing actions in a three-dimensional (3D) activity area;

[0008] Figure 2 is a schematic diagram of another embodiment of an interactive video game system having a two-dimensional (2D) active area according to the present technology;

[0009] Figure 3 is a diagram illustrating an example of a skeletal model and a shadow model representing a participant in a 3D activity area and a corresponding virtual representation of the participant presented in a virtual environment according to the present technology;

[0010] Figure 4 is a diagram illustrating an example of a skeletal model and a shadow model representing a participant in a 2D activity area and a corresponding virtual representation of the participant presented in a virtual environment according to the present technology;

[0011] Figure 5 is a flow chart illustrating an embodiment of a process for operating an interactive gaming system according to the present technology;

[0012] Figure 6 is a flow chart illustrating an example embodiment of a process by which an interactive video game system performs Figure 5 certain actions indicated in the flowchart of ; and

[0013] Figure 7 , 8 , 9, 10, 11, 12 and 13 respectively illustrate example embodiments of an interactive video game system that enables generation of a virtual representation having an enhanced appearance and / or movement relative to the detected appearance and / or movement of a participant. DETAILED DESCRIPTION

[0014] As used herein, "scan data" refers to two-dimensional (2D) or three-dimensional (3D) data collected by sensing (e.g., measuring, imaging, ranging) the visible external surfaces of participants in the activity area. More specifically, as used herein, "volume scan data" refers to 3D scan data such as point cloud data, and can be contrasted with "2D scan data" such as image data.

[0015] As used herein, a "participant model" is a 2D or 3D model generated from scan data of a participant, which generally describes the outer surface of the participant and may include texture data. More specifically, as used herein, a "volume participant model" or "volume model" refers to a 3D participant model generated from volume scan data of a participant, and may be contrasted with a "2D participant model" generated from 2D scan data of a participant.

[0016] As used herein, a "shadow model" refers to a textureless volumetric model of a participant generated from scan data of the participant, either directly or via a participant model. Thus, when presented on a 2D surface such as a display device, the shadow model of the participant has a shape substantially similar to the shadow or silhouette of the participant when illuminated from behind.

[0017] As used herein, a "skeletal model" refers to a 3D model generated from the scanned data of a participant that defines the predicted positioning and location of certain skeletons of the participant (e.g., skeletons associated with arms, legs, head, spine) to describe the positioning and posture of the participant in the activity area. Thus, the skeletal model is used to determine the movement and actions of the participant in the activity area to trigger events in the virtual environment and / or the activity area.

[0018] The present embodiment is directed to an interactive video game system that enables multiple participants (e.g., up to 12) to perform actions in a physical activity area to control the virtual representation of the participants in a displayed virtual environment. The disclosed interactive video game system includes one or more sensors (e.g., cameras, light sensors, infrared (IR) sensors) arranged around the activity area to capture scan data (e.g., 2D or volume scan data) of the participants. For example, certain embodiments of the disclosed interactive video game system include an array with two or more volume sensors, such as a depth camera and a light detection and ranging (LIDAR) device, which is capable of performing a volume scan of each participant. The system includes suitable processing circuitry that generates a model (e.g., a participant model, a shadow model, a skeletal model) for each participant based on the scan data collected by the one or more sensors, as discussed below. During game activity, the one or more sensors capture the actions of the participants in the activity area, and the system determines the nature of these actions based on the generated participant models. Therefore, the interactive video game system continuously updates the virtual representation of the participants and the virtual environment based on the actions of the participants and their corresponding in-game effects.

[0019] As mentioned, the disclosed interactive video game system includes one or more sensors arranged around the activity area to monitor the actions of participants in the activity area. For example, in some embodiments, an array including multiple sensors can be used to generally ensure that a skeletal model of each participant can be accurately generated and updated throughout the game activity, regardless of potential occlusions from the perspective of one or more sensors of the array. In other embodiments, fewer sensors (e.g., a single camera) can be used, and a machine learning algorithm can be used to process the data, which generates a complete skeletal model for the participant regardless of potential occlusions. For such embodiments, a machine learning tool can be pre-trained using a full set of scan data, in which the actual skeletal model of the participant (e.g., manually identified by a human, identified using another skeletal tracking algorithm) is known, and portions of one or more participants are occluded. Therefore, after training, the machine learning tool may then be able to generate a skeletal model of the participant based on the scan data, regardless of potential occlusions.

[0020] Additionally, the processing circuitry of the system can use the scan data to generate aspects (e.g., size, shape, outline) of a virtual representation of each participant within the virtual environment. In some embodiments, certain aspects (e.g., color, texture, scale) of the virtual representation of each participant can be further adjusted or modified based on information associated with the participant. As discussed below, the information can include information related to game activities (e.g., items acquired, achievements unlocked), as well as other information about the participant's activities outside of the game (e.g., participant performance in other games, items purchased by the participant, locations visited by the participant). In addition, the scan data collected by the sensor can be used by the processing circuitry of the gaming system to generate additional content, such as souvenir images, in which the participant model is illustrated as being within the virtual world.

[0021] In addition, the processing circuit of the system can use the scan data to enhance the movement of the virtual representation of each participant. For example, in some embodiments, the processing circuit of the system can use the scan data to generate a skeleton model, which indicates that the participant is moving or posing in a special way. In response, the processing circuit can enhance the virtual representation of the participant so that the virtual representation can move or change in a way that exceeds the actual movement or posture of the participant, so that the movement and or appearance of the virtual representation are enhanced or enhanced. For example, in an embodiment in which the virtual representation (such as a special video game character) has a special enhanced ability (for example, extremely high jumping ability, extremely fast swimming ability, the ability to fly), then a certain participant movement or posture (for example, a small jumping movement, a swimming cycle through the air, a flapping movement) can be detected, and these enhanced abilities in the virtual representation of the participant can be triggered. Therefore, the disclosed interactive video game system realizes the immersion and participation experience of multiple simultaneous participants.

[0022] Considering the foregoing, Figure 1 1 is a schematic diagram of an embodiment of an interactive video game system 10 that enables multiple participants 12 (e.g., participants 12A and 12B) to respectively control corresponding virtual representations 14 (e.g., virtual representations 14A and 14B) by performing actions in an active area 16. It can be noted that although for simplicity, the present description is directed to two participants 12 using the interactive video game system 10, in other embodiments, the interactive video game system 10 can support more than two (e.g., 6, 8, 10, 12, or more) participants 12. Figure 1The active area 16 of the interactive video game system 10 illustrated in FIG. 1 is described herein as a 3D active area 16A. The term "3D active area" as used herein refers to an active area 16 having a width (corresponding to the x-axis 18), a height (corresponding to the y-axis 20), and a depth (corresponding to the z-axis 22), wherein the system 10 generally monitors the movement of the participant 12 along the x-axis 18, the y-axis 20, and the z-axis 22, respectively. In response to the participant 12 moving throughout the active area 16, the interactive video game system 10 updates the positioning of the virtual representation 14 presented on the display device 24 along the x-axis 26, the y-axis 28, and the z-axis 30 in the virtual environment 32. Although the 3D active area 16A is illustrated as being substantially circular, in other embodiments, the 3D active area 16A may be square, rectangular, hexagonal, octagonal, or any other suitable 3D shape.

[0023] Figure 1 The embodiment of the interactive video game system 10 illustrated in FIG. 1 includes a main controller 34 having memory circuitry 33 and processing circuitry 35, which generally provides control signals to control the operation of the system 10. Thus, the main controller 34 is communicatively coupled to an array 36 of sensing cells 38 disposed about the 3D active area 16A. More specifically, the array 36 of sensing cells 38 may be described as being symmetrically distributed about the perimeter of the active area 16. In some embodiments, at least a portion of the array 36 of sensing cells 38 may be positioned above the active area 16 (e.g., suspended from a ceiling or on an elevated platform or stand) and directed at a downward angle to image the active area 16. In other embodiments, at least a portion of the array 36 of sensing cells 38 may be positioned near the floor of the active area 16 and directed at an upward angle to image the active area 16. In some embodiments, the array 36 of the interactive video game system 10 may include at least two sensing cells 38 for each participant (e.g., participants 12A and 12B) in the active area 16. Thus, in some embodiments, array 36 of sensing elements 38 is appropriately positioned to image a substantial portion of potential vantage points around active area 16 to reduce or eliminate potential participant occlusions. However, as mentioned above, and in other embodiments, array 36 may include fewer sensing elements 38 (e.g., a single sensing element), and processing circuitry 35 may rely on machine learning tools to handle potential occlusion situations.

[0024] In the illustrated embodiment, each sensing unit 38 includes a corresponding sensor 40, which can be a volumetric sensor (e.g., an infrared (IR) depth camera, a LIDAR device, or another suitable range-finding device) or a 2D imaging device (e.g., an optical camera). For example, in some embodiments, all sensors 40 of the sensing units 38 in the array 36 are IR depth cameras or LIDAR devices, while in other embodiments, a mixture of both IR depth cameras, LIDAR devices, and / or optical cameras are present within the array 36. It is currently recognized that both IR depth cameras and LIDAR devices can be used to perform a volumetric scan of each participant 12, and the collected volumetric scan data can be used to generate various models of the participant, as discussed below. For example, in some embodiments, the IR depth camera in the array 36 can be used to collect data to generate a skeletal model, while the data collected by the LIDAR device in the array 36 can be used to generate a participant model and / or a shadow model for the participant 12, which is discussed in more detail below. It is also recognized that a LIDAR device that collects point cloud data is generally capable of scanning and mapping a larger area than a depth camera, and the scanning and mapping generally has better accuracy and resolution. Thus, in certain embodiments, at least one sensing unit 38 of array 36 includes a corresponding volumetric sensor 40 that is a LIDAR device to enhance the accuracy or resolution of array 36 and / or reduce the overall number of sensing units 38 present in array 36 .

[0025] In addition, each illustrated sensing unit 38 includes a sensor controller 42 having suitable memory circuits 44 and processing circuits 46. The processing circuits 46 of each sensing unit 38 execute instructions stored in the memory circuits 44 to enable the sensing unit 38 to scan the participants 12 to generate scan data (e.g., volumetric and / or 2D scan data) for each participant 12. For example, in the illustrated embodiment, the sensing units 38 are communicatively coupled to the main controller 34 via a high-speed Internet Protocol (IP) network 48 that enables low-latency data exchange between devices of the interactive video game system 10. Additionally, in some embodiments, the sensing units 38 may each include a respective housing that encapsulates the sensor controller 42 with the sensor 40.

[0026] It may be noted that in other embodiments, the sensing unit 38 may not include a corresponding sensor controller 42. For such embodiments, the processing circuit 35 of the main controller 34 or other suitable processing circuit of the system 10 is communicatively coupled to the corresponding sensors 40 of the array 36 to directly provide control signals to the sensors 40 and directly receive data signals from the sensors 40. However, it is currently recognized that the processing (e.g., filtering, bone mapping) of the volume scan data collected by each of these sensors 40 may be processor intensive. Therefore, in certain embodiments, it may be advantageous to divide the workload by utilizing a dedicated processor (e.g., a processor 46 of each sensor controller 42) to process the scan data collected by the corresponding sensor 40 and then send the processed data to the main controller 34. For example, in the illustrated embodiment, each processor 46 of the sensor controller 42 processes the scan data collected by their corresponding sensor 40 to generate a partial model (e.g., a partial volume model or 2D model, a partial bone model, a partial shadow model) of each participant 12, and the processing circuit 35 of the main controller 34 receives and fuses or combines the partial models to generate a complete model of each participant 12, as discussed below.

[0027] Additionally, in some embodiments, the master controller 34 may also receive information from other sensing devices in and around the activity area 16. For example, the illustrated master controller 34 is communicatively coupled to a radio frequency (RF) sensor 45 disposed near the 3D activity area 16A (e.g., above the activity area 16A, below the activity area 16A, adjacent to the activity area 16A). The illustrated RF sensor 45 receives a unique identification RF signal from a wearable device 47 (such as a bracelet or headband with a radio frequency identification (RFID) tag worn by each participant 12). In response, the RF sensor 45 provides a signal regarding the identity and relative position of the participant 12 in the activity area 16 to the master controller 34. Thus, for the illustrated embodiment, the processing circuit 35 of the master controller 34 receives and combines data collected by the array 36 and, potentially, other sensors (e.g., the RF sensor 45) to determine the identity, location, and actions of the participant 12 in the activity area 16 during the gaming activity. Additionally, the illustrated master controller 34 is communicatively coupled to a database system 50 or any other suitable data repository storing participant information. The database system 50 includes processing circuitry 52 that executes instructions stored in memory circuitry 54 to store and retrieve information associated with the participants 12, such as various models associated with the participants (e.g., participant models, shadow models, and / or skeletal models), participant statistics (e.g., wins, losses, scores, total game activity time), participant attributes or inventory (e.g., abilities, textures, items), participant purchases at gift shops, participant scores in loyalty reward programs, etc. The processing circuitry 35 of the main controller 34 can query, retrieve, and update the information related to the participants 12 stored by the database system 50 to enable the system 10 to operate as described herein.

[0028] Additionally, Figure 1The embodiment of the interactive video game system 10 illustrated in FIG. 1 includes an output controller 56 that is communicatively coupled to the main controller 34. The output controller 56 typically includes a processing circuit 58 that executes instructions stored in a memory circuit 60 to control the output of stimuli (e.g., audio signals, video signals, lights, physical effects) observed and experienced by the participants 12 in the active area 16. Thus, the illustrated output controller 56 is communicatively coupled to an audio device 62 and a display device 24 to provide appropriate control signals to operate these devices to provide a particular output. In other embodiments, the output controller 56 may be coupled to any suitable number of audio and / or display devices. The display device 24 may be any suitable display device, such as a projector and screen, a flat screen display device, or an array of flat screen display devices that are arranged and designed to provide a suitable view of the virtual environment 32 to the participants 12 in the active area 16. In some embodiments, the audio device 62 may be arranged in an array around the active area 16 to increase participant immersion during gaming activities. For example, in some embodiments, each audio device 62 (e.g., each speaker) in such an array may be independently controlled by the master controller 34 so that each participant 12 can hear different sounds relative to other participants and unique to their own actions. In yet other embodiments, the activity area 16 may include robotic elements (e.g., robots, robotic animals, etc.) that can be actuated in the real world in response to signals provided by the output controller based on the participant's actions during the game activity. For example, in addition to or in place of the virtual representation of the participant presented on the display device 24, the robotic representation of the participant 12 provides a non-virtual representation that is controlled in response to the movement and behavior of the participant 12 in the activity area 16. In other embodiments, the system 10 may not include the output controller 56, and the processing circuit 35 of the master controller 34 may be communicatively coupled to the audio device 62, the display device 24, etc. to generate various stimuli for the participant 12 to observe and experience in the activity area 16.

[0029] Figure 2 1 is a schematic diagram of another embodiment of an interactive video game system 10 that enables multiple players 12 (eg, players 12A and 12B) to control virtual representations 14 (eg, virtual representations 14A and 14B) by performing actions in an active area 16. Figure 2 The embodiment of the interactive video game system 10 illustrated in FIG. 1 includes the embodiments described herein with reference to Figure 1 Many of the features discussed include the main controller 34, the array 36 of sensing elements 38, the output controller 56, and the display device 24. However, Figure 2The embodiment of the interactive video game system 10 illustrated in FIG. 1 is described herein as having a 2D active area 16B. As used herein, the term "2D active area" refers to an active area 16 having a width (corresponding to the x-axis 18) and a height (corresponding to the y-axis 20), wherein the system 10 generally monitors the movement of each participant 12 along the x-axis 18 and the y-axis 20. Figure 2 , participants 12A and 12B are assigned segments 70A and 70B of the 2D activity area 16B, respectively, and participants 12 do not wander outside their respective assigned segments during game play. However, it will be appreciated that other embodiments of the interactive video game system 10 may include a sufficient number of sensors (e.g., LIDAR sensors or other suitable sensors 40 located above the participants) that each participant 12 may be continuously tracked as they freely move around the entire activity area 16B, with the system accounting for potential occlusion by other participants as they move. The interactive video game system 10 updates the positioning of the virtual representation 14 presented on the display device 24 along the x-axis 26 and y-axis 28 in the virtual environment 32 in response to the participant 12 moving within the 2D activity area 16B (e.g., running along the x-axis 18, jumping along the y-axis 20). As mentioned, in some embodiments, the array 36 may include fewer sensors (e.g., a single camera).

[0030] Additionally, Figure 2 The embodiment of the interactive video game system 10 illustrated in FIG. 1 includes an interface panel 74 that enables enhanced player interaction. Figure 2 As shown in FIG. 1 , the interface panel 74 includes a plurality of input devices 76 (e.g., cranks, wheels, buttons, sliders, blocks) designed to receive input from the participants 12 during the game activity. Thus, the illustrated interface panel 74 is communicatively coupled to the master controller 34 to provide signals to the controller 34 indicating how the participants 12 manipulate the input devices 76 during the game activity. The illustrated interface panel 74 also includes a plurality of output devices 78 (e.g., audio output devices, visual output devices, physical stimulation devices) designed to provide audio, visual and / or physical stimulation to the participants 12 during the game activity. Thus, the illustrated interface panel 74 is communicatively coupled to the output controller 56 to receive control signals and provide appropriate stimulation to the participants 12 in the activity area 16 in response to appropriate signals from the master controller 34. For example, the output devices 78 may include audio devices, such as speakers, horns, alarms, etc. The output devices 78 may also include visual devices, such as lights or display devices of the interface panel 74.

[0031] In some embodiments, the output device 78 of the interface panel 74 includes a physical effect device, such as an electrically controlled release valve coupled to a compressed air line, which provides a burst of hot air or cold air or mist in response to a suitable control signal from the main controller 34 or the output controller 56. It can be appreciated that the output device is not limited to those incorporated in the interface panel 74. In some embodiments, the activity area 16 may include an output device that indirectly provides physical effects to the participant, such as through the air. For example, in an embodiment, when the participant reaches a special posture to trigger the ability or action of the virtual representation, the participant can experience the corresponding physical effect. By way of specific example, in an embodiment in which the participant has the ability to throw snowballs, in response to the participant extending their hands in a special way, the participant can receive a blast of cold air on the palm of their exposed hand. In an embodiment in which the participant has the ability to throw fireballs, in response to the participant extending their hands in a special way, the participant can receive a blast of hot air or IR radiation (e.g., heat). In still other embodiments, the participant may receive tactile feedback (e.g., ultrasonic tactile feedback) in response to the participant's virtual representation interacting with an object in the virtual world. For example, when the participant's virtual representation punches a wall in the virtual environment, the participant may receive some physically feelable effect on a part of their body (e.g., an extended fist) that corresponds to the activity in the virtual environment.

[0032] like Figure 2 , the array 36 of sensing units 38 disposed around the 2D active area 16B of the illustrated embodiment of the interactive video game system 10 includes at least one sensing unit 38. That is, although Figure 1 Some embodiments of the interactive video game system 10 illustrated in FIG. 1 include an array 36 having at least two sensing units 38 per participant, but Figure 2 The embodiment of the interactive video game system 10 illustrated in includes an array 36 having as few as one sensing unit 38 regardless of the number of participants. In some embodiments, the array 36 may include at least two sensing units disposed at right angles (90°) relative to the participants 12 in the 2D active area 16B. In some embodiments, the array 36 may additionally or alternatively include at least two sensing units disposed on opposite sides (180°) relative to the participants 12 in the active area 16B. By way of specific example, in some embodiments, the array 36 may include only two sensing units 38 disposed on different (e.g., opposite) sides of the participants 12 in the 2D active area 16B.

[0033] As mentioned, Figure 1 and 2The array 36 illustrated in FIG. 1 is capable of collecting scan data (e.g., volume scan data or 2D scan data) for each participant 12 in the active area 16. In some embodiments, the collected scan data can be used to generate various models (e.g., participants, shadows, skeletons) for each participant, and these models can then be updated based on the movement of the participants during the game activity, as discussed below. However, it is currently recognized that the use of a volume model including the texture data is substantially more processor intensive (e.g., involving additional filtering, additional data processing) than the use of a shadow model lacking texture data. For example, in some embodiments, the processing circuit 35 of the main controller 34 can generate a shadow model of each participant 12 based on the scan data (e.g., 2D scan data) collected via the array 36 by using edge detection techniques to distinguish between the edges of the participant 12 and its surrounding environment in the active area 16. It is currently recognized that such edge detection techniques are substantially less processor intensive and involve substantially less filtering than the use of a volume model including texture data. Thus, it is presently recognized that certain embodiments of the interactive video game system 10 generate and update a shadow model rather than a volume model including textures, thereby achieving a reduction in size, complexity, and cost of the processing circuitry 35 of the main controller 34. Additionally, as discussed below, the processing circuitry 35 may generate a virtual representation 14 of the participant 12 based at least in part on the generated shadow model.

[0034] As mentioned, the scan data collected by the array 36 of the interactive video game system 10 may be used to generate various models (e.g., 2D or volumetric participant models, shadow models, skeletal models) for each participant. Figure 3 is a diagram illustrating skeletal models 80 (eg, skeletal models 80A and 80B) and shadow models 82 (eg, shadow models 82A and 82B) representing participants in 3D activity area 16A. Figure 3 Also illustrated are corresponding virtual representations 14 (e.g., virtual representations 14A and 14B) of these participants presented in a virtual environment 32 on a display device 24 in accordance with the present technology. As illustrated, the represented participants are located at different locations within a 3D active area 16A of the interactive video game system 10 during game play, as indicated by the positioning of the skeletal model 80 and the shadow model 82. The illustrated virtual representations 14 of the participants in the virtual environment 32 are generated at least in part based on the shadow models 82 of the participants. As mentioned above, as the participants move within the 3D active area 16A, the master controller 34 tracks these movements and generates updated skeletal models 80 and shadow models 82 and virtual representations 14 of each participant accordingly.

[0035] Additionally, if Figure 1 and3 As shown in FIG. 1 , an embodiment of an interactive video game system 10 having a 3D active area 16A implements tracking and player movement along the z-axis 22 and translates it into movement of the virtual representation 14 along the z-axis 30. Figure 3 , this enables the participant represented by the skeletal model 80A and the shadow model 82A to move the leading edge 84 of the 3D active area 16A and causes the corresponding virtual representation 14A to be presented at a relatively deeper point or level 86 along the z-axis 30 in the virtual environment 32. This also enables the participant represented by the skeletal model 80B and the shadow model 82B to move to the trailing edge 88 of the 3D active area 16A, which causes the corresponding virtual representation 14B to be presented at a substantially shallower point or level 90 along the z-axis 30 in the virtual environment 32. In addition, for the illustrated embodiment, the size of the presented virtual representation 14 is modified based on the position of the participant along the z-axis 22 in the 3D active area 16A. That is, the virtual representation 14A positioned relatively deep along the z-axis 30 in the virtual environment 32 is presented as being significantly smaller than the virtual representation 14B positioned at a shallower depth or layer along the z-axis 30 in the virtual environment 32.

[0036] It may be noted that for embodiments of the interactive video game system 10 having a 3D participant area 16A, Figure 1 and 3 , the virtual representation 14 may only be able to interact with virtual objects that are positioned at a similar depth along the z-axis 30 in the virtual environment 32. For example, for Figure 3 , virtual representation 14A is able to interact with virtual object 92 that is positioned deeper along z-axis 30 in virtual environment 32, while virtual representation 14B is able to interact with another virtual object 94 that is positioned at a relatively shallow depth along z-axis 30 in virtual environment 32. That is, virtual representation 14A cannot interact with virtual object 94 unless the participant represented by models 80A and 82A changes position along z-axis 22 in 3D active area 16A so that virtual representation 14A moves to a depth similar to virtual object 94 in virtual environment 32.

[0037] For comparison, Figure 4 is a diagram illustrating examples of skeletal models 80 (eg, skeletal models 80A and 80B) and shadow models 82 (eg, shadow models 82A and 82B) representing participants in a 2D activity area 16B. Figure 4 Also illustrated are virtual representations 14 of the participants (e.g., virtual representations 14A and 14B) presented on the display device 24. As mentioned above, as the participants move in the 2D activity area 16B, the master controller 34 tracks these movements and updates the skeletal model 80, shadow model 82, and virtual representations 14 of each participant accordingly. Figure 2 and 4 The embodiment of the interactive video game system 10 with a 2D active area 16B illustrated in FIG. 1 does not track the player's movement along the z-axis (e.g., Figure 1 and 3 Alternatively, for embodiments with a 2D activity area 16B, the size of the presented virtual representation 14 may be modified based on the state or situation of the participants inside and / or outside the game activity. Figure 4 14B. In some embodiments, the size of virtual representations 14A and 14B may be enhanced or enlarged in response to virtual representations 14A or 14B interacting with a particular item, such as in response to virtual representation 14A gaining a power-up during a current or previous round of game play. In other embodiments, the enlarged size of virtual representation 14A and other modifications of the virtual representation (e.g., texture, color, transparency, items worn or carried by the virtual representation) may be the result of a corresponding participant interacting with an object or item external to interactive video game system 10, as discussed below.

[0038] It is currently recognized that Figure 2 and 4 As shown in FIG. 1 , an embodiment of the interactive video game system 10 utilizing the 2D active area 16B implements the same Figure 1 The particular advantages of the embodiment of the interactive video game system 10 utilizing the 3D active area 16A illustrated in FIG. Figure 2 The array 36 of sensing units 38 in the interactive video game system 10 with a 2D active area 16B illustrated in FIG. Figure 1 The interactive video game system 10 with 3D active area 16A shown in FIG. 1 includes fewer sensing units 38. That is, for example, Figure 2 and 4 16B, the depth (e.g., Figure 1 16B) is not tracked. Additionally, because participants 12A and 12B remain in their respective assigned segments 70A and 70B of 2D active area 16B, the potential for occlusion is significantly reduced. For example, by keeping participants within their assigned segments 70 of 2D active area 16B, occlusions between participants can only occur predictably along x-axis 18. Thus, by using 2D active area 16B, Figure 2 The embodiment of the interactive video game system 10 illustrated in Figure 1This enables the use of a smaller array 36 with fewer sensing units 38 to track participants 12, compared to embodiments of the interactive video game system 10.

[0039] Thus, it is recognized that the smaller array 36 of sensing elements 38 used by an embodiment of the interactive video game system 10 having a 2D active area 16B also generates significantly less data to be processed than an embodiment having a 3D active area 16A. For example, because Figure 2 and 4 In 2D active area 16B, occlusions between players 12 are significantly more limited and predictable, so fewer sensing elements 38 may be used in array 36 while still covering a significant portion of potential vantage points around active area 16. Thus, for embodiments of interactive video game system 10 having 2D active area 16B, processing circuitry 35 of main controller 34 may be smaller, simpler, and / or more power efficient relative to processing circuitry 35 of main controller 34 for embodiments of interactive video game system 10 having 3D active area 16A.

[0040] As mentioned, the interactive video game system 10 is capable of generating various models of the participants 12. More specifically, in some embodiments, the processing circuitry 35 of the master controller 34 is configured to receive partial model data (e.g., partial participant, shadow, and / or skeletal models) from the various sensing units 38 of the array 36 and fuse the partial models into a complete model (e.g., a complete volumetric, shadow, and / or skeletal model) of each participant 12. Set forth below are examples in which the processing circuitry 35 of the master controller 34 fuses partial skeletal models received from the various sensing units 38 of the array 36. It can be appreciated that in some embodiments, the processing circuitry 35 of the master controller 34 can use a similar process to fuse partial shadow model data into shadow models and / or fuse partial volumetric model data.

[0041] In the example, a partial skeleton model is generated by each sensing unit 38 of the interactive video game system 10 and subsequently fused by the processing circuit 35 of the main controller 34. In particular, the processing circuit 35 may perform a one-to-one mapping of the corresponding skeleton of each participant 12 in each partial skeleton model generated by different sensing units 38 positioned at different angles (e.g., opposite sides, vertically) relative to the active area 16. In some embodiments, when the partial skeleton models generated by different sensing units 38 are fused by the processing circuit 35, relatively small differences between the partial skeleton models may be averaged to provide smoothness and prevent jerky movement of the virtual representation 14. Additionally, when the partial skeleton model generated by a particular sensing unit is significantly different from the partial skeleton models generated by at least two other sensing units, the processing circuit 35 of the main controller 34 may determine that the data is erroneous and, therefore, not include the data in the skeleton model 80. For example, if a particular partial skeleton model is missing a skeleton that is present in other partial skeleton models, the processing circuit 35 may determine that the missing skeleton is likely the result of an occlusion and may discard all or some of the partial skeleton models in response.

[0042] It may be noted that precise coordination of the components of the interactive video game system 10 is desirable to provide smooth and responsive movement of the virtual representation 14 in the virtual environment 32. In particular, in order to properly fuse the partial models (e.g., partial skeleton, volumetric, and / or shadow models) generated by the sensing unit 38, the processing circuit 35 may take into account the time at which each partial model was generated by the sensing unit 38. In some embodiments, the interactive video game system 10 may include a system clock 100, such as Figure 1 and 2, which is used to synchronize operations within the system 10. For example, the system clock 100 can be a component of the main controller 34 or another suitable electronic device that is capable of generating a time signal that is broadcast on the network 48 of the interactive video game system 10. In some embodiments, various devices coupled to the network 48 can receive and use the time signal to adjust the corresponding clocks at a particular time (e.g., at the beginning of a game activity), and when providing game activity data to the main controller 34, the devices can then include timing data based on the signals from these corresponding clocks. In other embodiments, various devices coupled to the network 48 continuously (e.g., at regular microsecond intervals) receive time signals from the system clock 100 throughout the game activity, and when providing data (e.g., volume scan data, partial model data) to the main controller 34, the devices then include timing data from the time signal. Additionally, the processing circuit 35 of the main controller 34 can determine whether the partial model (e.g., partial volume, shadow or skeleton model) generated by the sensing unit 38 is fresh enough (e.g., recent, belonging to the same period as other data) to be used to generate or update the complete model, or whether the data should be discarded because it is not fresh. Thus, in some embodiments, the system clock 100 enables the processing circuitry 35 to appropriately fuse the partial models generated by the various sensing units 38 into a suitable volumetric, shadow, and / or skeletal model of the participant 12.

[0043] Figure 5 1 is a flow chart illustrating an embodiment of a process 110 for operating an interactive video game system 10 in accordance with the present technology. It can be appreciated that in other embodiments, certain steps of the illustrated process 110 may be performed in a different order, repeated multiple times, or skipped altogether, in accordance with the present disclosure. Figure 5 The process 110 illustrated in FIG. 1 may be performed by the processing circuit 35 of the main controller 34 alone or in combination with other suitable processing circuits of the system 10 (eg, the processing circuits 46 , 52 , and / or 58 ).

[0044] The illustrated embodiment of process 110 begins with the interactive video game system 10 collecting (block 112) scan data for each participant. In some embodiments, Figure 1-4, participants 12 may be scanned or imaged by sensing units 38 positioned about the activity area 16. For example, in some embodiments, prior to commencing a gaming activity, participants 12 may be prompted to reach a particular posture, and the sensing units 38 of the array 36 collect scan data (e.g., volumetric and / or 2D scan data) about each participant. In other embodiments, participants 12 may be volumetrically scanned by a separate system prior to entering the activity area 16. For example, a line of waiting participants may be directed through a pre-scanning system (e.g., similar to a security scanner at an airport), in which each participant is individually scanned (e.g., upon reaching a particular posture) to collect scan data for each participant. In some embodiments, the pre-scanning system may be Figure 1 3D active area 16A or Figure 2 16B, in which an array 36 including one or more sensing units 38 is positioned around an individual participant to collect scan data. In other embodiments, the pre-scan system may include fewer sensing units 38 (e.g., 1, 2, 3) positioned around an individual participant, and the sensing units 38 rotate around the participant to collect complete scan data. It is currently recognized that it may be desirable to collect the scan data indicated in box 112 while the participant 12 is in the active area 16 to enhance the efficiency of the interactive video game system 10 and reduce participant waiting time.

[0045] Next, the interactive video game system 10 generates (box 114) a corresponding model for each participant based on the scan data collected for each participant. As described above, in some embodiments, the processing circuit 35 of the main controller can receive a partial model of each participant from each sensing unit 38 in the array 36, and can appropriately fuse these partial models to generate a suitable model for each participant. For example, the processing circuit 35 of the main controller 34 can generate a participant model (e.g., a volume or 2D participant model) for each participant that generally defines the 2D or 3D shape of each participant. Additionally or alternatively, the processing circuit 35 of the main controller 34 can generate a shadow model for each participant that generally defines the textureless 3D shape of each participant. In addition, the processing circuit 35 can also generate a skeletal model that generally defines the predicted skeletal position and positioning of each participant within the activity area.

[0046] Next, continuing with the example process 110, the interactive video game system 10 generates (block 116) a corresponding virtual representation for each participant based at least in part on the scan data collected for each participant and / or one or more models generated for each participant. For example, in some embodiments, the processing circuit 35 of the master controller 34 may use the shadow model generated in block 114 as a basis to generate a virtual representation of the participant. It can be appreciated that in some embodiments, the virtual representation 14 may have a shape or outline that is substantially similar to the shadow model of the corresponding participant, such as Figure 3 and 4 . In addition to shape, the virtual representation 14 may have other attributes that can be modified to correspond to the attributes of the represented participant. For example, a participant may be associated with various attributes (e.g., items, status, scores, statistics) that reflect their performance in other gaming systems, their purchases in a gift shop, their membership in a loyalty program, and the like. Thus, the attributes of the virtual representation (e.g., size, color, texture, animation, presence of virtual items) may be set in response to the various attributes associated with the corresponding participant, and further modified based on changes to the attributes of the participant during gaming activity. Moreover, the corresponding virtual representation for the participant may be based only in part on scan data and / or a shadow model generated for the participant, such that the virtual representation includes enhanced and / or modified visual characteristics relative to the actual appearance of the participant. For example, in an embodiment, for a participant who is in a seated position (e.g., sitting in a chair or wheelchair), a virtual representation may be generated in which an upper portion of the virtual representation includes a true (e.g., based on a shadow model of the participant) silhouette, while a lower portion of the virtual representation is illustrated in an alternative or abstract manner (e.g., as a floating cloud). In another embodiment, an upper portion of the body of the virtual representation includes a true (e.g., based on a shadow model of the participant) silhouette, while a lower portion of the body of the virtual representation is illustrated as a horse's body, thereby producing a centaur-shaped virtual representation. In such an embodiment, the lower horse portion of the virtual representation may move like a horse in a manner that corresponds to (e.g., synchronized with) the movement of the participant's feet in a directly related or enhanced manner, as discussed in more detail below.

[0047] It may be noted that in some embodiments, the virtual representation 14 of the participant 12 may not have an appearance or shape that is substantially similar to the generated participant or shadow model. For example, in some embodiments, the interactive video game system 10 may include or be communicatively coupled to a pre-generated library of virtual representations based on fictional characters (e.g., avatars), and the system may select a particular virtual representation, or generally provide a recommendation of a particular selectable virtual representation to the participant based on the generated participant model or shadow model. For example, if the game involves a larger hero and a smaller sidekick, the interactive video game system 10 may select or recommend a relatively larger hero virtual representation for an adult participant and a relatively smaller sidekick virtual representation for a child participant from the pre-generated library.

[0048] The process 110 continues with the interactive video game system 10 presenting (block 118) on the display device 24 a corresponding virtual representation 14 of each participant in the virtual environment 32. In addition to the presentation, in some embodiments, the actions in block 118 may also include presenting other introductory presentations, such as a welcome message or orientation / instruction information, to the participants 12 in the activity area 16 before commencing the gaming activity. In addition, in some embodiments, the processing circuitry 35 of the master controller 34 may also provide appropriate signals to set or modify parameters of the environment within the activity area 16. For example, these modifications may include adjusting the brightness and / or color of lights in the house, playing game music or game sound effects, adjusting the temperature of the activity area, activating physical effects in the activity area, etc.

[0049] Once the game play is initiated, the virtual representations 14 generated in block 116 and presented in block 118 can interact with each other and / or with virtual objects (e.g., virtual objects 92 and 94) in the virtual environment 32, as described herein with respect to Figure 3 and 4 During the game play, the interactive video game system 10 typically determines (block 120) the in-game actions of each participant 12 in the active area 16 and the corresponding in-game effects of those in-game actions. Additionally, the interactive video game system 10 typically updates (block 122) the virtual environment 32 and / or the corresponding virtual representations 14 of the participants 12 based on the in-game actions of the participants 12 in the active area 16 and the corresponding in-game effects determined in block 120. As indicated by arrow 124, the interactive video game system 10 may repeat the steps indicated in blocks 120 and 122 until the game play is completed, for example, because one of the participants 12 wins the round of the game play or because the allotted game play time expires.

[0050] Figure 6 is a flow chart illustrating an example embodiment of a more detailed process 130 by which the interactive video game system 10 performs Figure 5 That is, Figure 6 The process 130 indicated in FIG. 1 includes a plurality of steps of determining in-game actions of each participant in the active area and corresponding in-game effects of those in-game actions (as indicated by bracket 120), and a plurality of steps of updating the virtual environment and / or the corresponding virtual representation of each participant (as indicated by bracket 122). In some embodiments, the actions described in process 130 may be encoded as instructions in a suitable memory such as memory circuit 33 of main controller 34 and executed by a suitable processor such as processing circuit 35 of main controller 34 of interactive video game system 10. It should be noted that the illustrated process 130 is provided as an example only, and in other embodiments, some of the actions described may be performed in a different order, may be repeated, or may be skipped altogether.

[0051] Figure 6 The process 130 begins with the processing circuit 35 receiving (block 132) a partial model from a plurality of sensing units in the active area. Figure 1 and 2 As discussed, the interactive video game system 10 includes an array 36 of sensing units 38 disposed in different locations around the active area 16, and each of these sensing units 38 is configured to generate one or more partial models (e.g., partial participant, shadow, and / or skeletal model) for at least a portion of the participant 12. Additionally, as mentioned, the processing circuitry 35 may also receive data about the actions of the participant 16 disposed within the active area 16 from other devices (e.g., RF sensor 45, input device 76). Furthermore, as mentioned, these partial models may be time-stamped based on a signal from the clock 100 and provided to the processing circuitry 35 of the main controller 34 via the high-speed IP network 48.

[0052] For the illustrated embodiment of process 130, after receiving the partial models from sensing unit 38, processing circuit 35 fuses the partial models to generate (block 134) updated models (e.g., participants, shadows, and / or skeletons) for each participant based on the received partial models. For example, processing circuit 35 may update a previously generated model, such as in Figure 5 The initial skeletal model generated in block 114 of process 110. Additionally, as discussed, when combining the partial models, the processing circuitry 35 may filter or remove inconsistent or delayed data to improve accuracy when tracking participants despite potential occlusions or network delays.

[0053] Next, the illustrated process 130 continues with the processing circuit 35 identifying (block 136) one or more in-game actions for the corresponding virtual representation 14 of each participant 12 based at least in part on the updated model of the participant generated in block 134. For example, the in-game action may include jumping, running, sliding, or other movement of the virtual representation 14 within the virtual environment 32. The in-game action may also include interacting with (e.g., moving, acquiring, losing, consuming) items such as virtual objects in the virtual environment 32. The in-game action may also include completing a goal, defeating another participant, winning a round, or other similar in-game actions.

[0054] Next, the processing circuitry 35 may determine (block 138) one or more in-game effects triggered in response to the identified in-game actions of each participant 12. For example, when the determined in-game action is movement of the participant, the in-game effect may be a corresponding change in the position of the corresponding virtual representation within the virtual environment. When the determined in-game action is jumping, the in-game effect may include moving the virtual representation along the y-axis 20, such as Figure 1-4 . When a determined in-game action is activating a particular power-up item, the in-game effect may include modifying a state associated with the participant 12 (e.g., health state, strength state). Additionally, in some cases, the movement of the virtual representation 14 may be emphasized or enhanced relative to the actual movement of the participant 12. For example, as discussed above with respect to modifying the appearance of the virtual representation, the movement of the virtual representation of the participant may be temporarily or permanently amplified relative to the actual movement of the participant (e.g., being able to jump higher, being able to jump farther), based on attributes associated with the participant (including items acquired during gaming activities, items acquired during a period of other gaming activities, items purchased in a gift shop, etc.).

[0055] The illustrated process 130 continues with the processing circuitry 35 generally updating the presentation to the participants in the activity area 16 based on each participant's in-game actions and corresponding in-game effects, as indicated by the classification 122. In particular, the processing circuitry 35 updates (block 140) the corresponding virtual representation 14 of each participant 12 and the virtual environment 32 based on the updated model (e.g., shadow and skeletal model) of each participant generated in block 134, the in-game actions identified in block 136, and / or the in-game effects determined in block 138 to advance the game activity. For example, for Figure 1 and 2 In the embodiment illustrated in FIG. 2 , processing circuitry 35 may provide appropriate signals to output controller 56 so that processing circuitry 58 of output controller 56 updates virtual representation 14 and virtual environment 32 presented on display device 24 .

[0056] Additionally, the processing circuitry 35 may provide appropriate signals to generate (block 142) one or more sounds and / or one or more physical effects (block 144) in the active area 16 based at least in part on the determined in-game effect. For example, when the in-game effect is determined to be a particular virtual representation of a participant striking a virtual pool, the main controller 34 may cause the output controller 56 to signal the speaker 62 to generate an appropriate splashing sound and / or cause the physical effects device 78 to generate an air wave of mist. Additionally, the sounds and / or physical effects may be generated in response to any number of in-game effects, including, for example, gaining a power-up, losing a power-up, scoring a point, or moving through a particular type of environment. Figure 5 Mentioned, Figure 6 Process 130 may be repeated until the gaming activity is completed, as indicated by arrow 124 .

[0057] In addition, it can be noted that the interactive video game system 10 can also use the scan data collected by the array 36 of sensing units 38 to implement other functionalities. For example, as mentioned, in some embodiments, the processing circuit 35 of the main controller 34 can generate a participant model (e.g., a volumetric or 2D participant model) that includes both the texture and shape of each participant. At the end of the game activity, the processing circuit 35 of the main controller 34 can generate a simulated image that uses the participant's model to render a 2D or 3D likeness of the participant within a portion of the virtual environment 32, and these can be provided (e.g., printed, electronically transferred) to the participant 12 as a souvenir of their game activity experience. For example, this can include the printing of a simulated image that illustrates a volumetric model of the participant crossing a finish line within the scene from the virtual environment 32.

[0058] Figure 7-13 Example embodiments of an interactive video game system 10 are illustrated that enable generation of a virtual representation having an enhanced appearance and / or movement relative to the appearance and / or movement of a participant. For these example embodiments, although only a single participant is illustrated for simplicity, it is contemplated that these interactive video game systems may be used simultaneously by any suitable number of participants (e.g., 12 participants), as discussed above. Additionally, although not illustrated for simplicity, Figure 7-13 The example interactive video game system illustrated in includes any suitable features mentioned herein (eg, sensors, controllers, display devices, physical effects devices, etc.) to enable operation of system 10, as discussed above.

[0059] In view of the foregoing, in some embodiments, the virtual representation may be modified to appear and / or move differently from the corresponding participant. That is, in some embodiments, the virtual representation associated with a particular participant may be able to transform or move in a manner that does not directly correspond to the appearance or movement of the participant (e.g., not completely the same as the appearance or movement of the participant). In some embodiments, the virtual representation is not limited by the real-world physical limitations imposed on the appearance or movement of the participant, and therefore, can be described as being associated with superhuman abilities. For example, in some embodiments, the virtual representation may include a character with stronger than normal or superhuman abilities, such as a character that can jump higher or stretch farther than a real human can. In other embodiments, these superhuman abilities may include other super speeds, super strength, the ability to change size (e.g., shrink and grow), the ability to launch projectiles from various body parts (e.g., laser-firing eyes or hands, throwing fire or ice), etc. Therefore, when a participant controls such a virtual representation, these superhuman abilities of the virtual representation are triggered (e.g., converted) by the special actual or real-world movement of the participant. By way of further example and certain embodiments, the virtual representations may be representations of non-human entities. For example, in certain embodiments, the virtual representations may be animal-based representations of participants, wherein these representations have capabilities (e.g., modes or patterns of movement) that are different from and / or enhanced relative to the capabilities of an ordinary human.

[0060] exist Figure 7 In one example illustrated in FIG. 1 , during a gaming activity of the interactive video game system 10, the participant 12 is positioned within the activity area 16, and a portion of the virtual environment 32 is presented on the display device 24. More specifically, the virtual environment 32 includes a virtual representation 14 representing the participant 12. Thus, for the illustrated example, the virtual representation 14 has an appearance that is generally similar to the appearance of the participant 12, based on the scan data and various models discussed above. However, unlike the other examples discussed above, Figure 7 The virtual representation 14 illustrated in FIG. 1 shows an incremental physical movement relative to the detected movement of the participant 12 .

[0061] for Figure 7In the example illustrated in , participant 12 is illustrated as jumping a modest distance from the floor of activity area 16, while virtual representation 14 is illustrated as performing a substantially larger jumping motion relative to the floor of virtual environment 32. Thus, virtual representation 14 exhibits an enhanced (e.g., enhanced, enlarged) jumping ability beyond that of a normal human (e.g., superhuman jumping ability). In some cases, the enhanced jumping ability may be performed by virtual representation 14 after acquiring a special item (e.g., a power-up) within virtual environment 32, and the ability may be temporary or permanent after acquiring the item. In other embodiments, the enhanced jumping ability may be a feature or aspect of a special character (e.g., a fictional character from a video game, book, or movie) on which virtual representation 14 is based. For such embodiments, by selecting a character associated with the enhanced jumping ability, virtual representation 14 may display this enhanced jumping ability throughout the game activity. It should be appreciated that Figure 7 The enhanced jump illustrated in the figure is merely one example of an enhanced movement, and in other embodiments, in accordance with the present disclosure, any other suitable type of participant movement (e.g., running, jumping, spinning, dancing, etc.) may be identified and enhanced by the processing circuitry 35 of the main controller 34 based on the scanning data and models (e.g., skeletal models) discussed above.

[0062] In some embodiments, the virtual representation 14 may be associated with the ability to affect both the appearance and movement of the virtual representation 14 in response to particular movements of the participant 12. Figure 8 In the example of FIG. 1 , a player 12 is positioned in an activity area 16 during a gaming activity of an interactive video game system 10, and a corresponding virtual representation 14 associated with a size-changing super-human ability is presented on a display device 24. Figure 8 In the particular example illustrated in , the participant 12 has lowered into a crouching position during the gaming activity. This crouching position, when detected by the processing circuitry 35 of the master controller 34 in the scan data and one or more of the models discussed above, represents a special or control gesture that triggers a special enhanced capability of the virtual representation 14 or virtual environment 32. It can be appreciated that in other embodiments, other control gestures may be used in accordance with the present disclosure.

[0063] for Figure 8In the example illustrated in FIG. 3 , in response to detecting a participant 12 in a control posture (e.g., a crouching posture), the size of the illustrated virtual representation 14 is significantly reduced, thereby effectively compacting the virtual representation 14 within the virtual environment 32. In some embodiments, the virtual representation 14 may only maintain a reduced or reduced size while the participant 12 remains in a crouching position. In other embodiments, once the controller 34 determines that the participant 12 has taken a crouching control posture, the participant 12 may stand upright again without the virtual representation 14 returning to its previous or original size. For such embodiments, the size of the virtual representation 14 may remain reduced until the master controller 34 determines that the participant 12 has taken a second control posture (e.g., standing with arms and legs extended in a generally "X" shape), thereby triggering an enlargement of the virtual representation 14. In this way, upon detecting one or more control postures, the master controller 34 may trigger one or more special abilities or superpowers temporarily or permanently associated with the virtual representation 14.

[0064] It can be understood that Figure 8 , the modified appearance of the virtual representation 14 can also be associated with the difference in the movement and / or ability of the virtual representation 14 within the virtual environment 32. For example, in some cases, the virtual representation 14 that is sized smaller can show enhanced (e.g., enhanced, enlarged) movement relative to the movement of the detected participant 12. That is, in some cases, the virtual representation 14 that is sized smaller can continue to jump as high and run as fast as the participant 12, regardless of its reduced size. In other cases, the movement of the virtual representation 14 that is sized smaller can be reduced or reduced relative to the movement of the detected participant 12 until the virtual representation 14 is restored to full size. In some embodiments, the virtual representation that is sized smaller can show an enhanced effect relative to the features within the virtual environment 32. For example, the virtual representation 14 that is sized smaller may be more susceptible to displacement or influence by wind or current moving in the virtual environment, or may obtain a position into the virtual environment 32 that is inaccessible to the virtual representation 14 that is sized larger.

[0065] for Fig. 9 In the example of FIG. 1 , during a game activity of the interactive video game system 10, the participant 12 is positioned in the activity area 16, and the corresponding virtual representation 14 associated with the super-human stretching ability is presented on the display device 24. Fig. 9In the particular example illustrated in , the participant 12 extends or stretches his arm from his side during the game activity. For the illustrated embodiment, the virtual representation 14 associated with the hyper-extension ability can be based on a character selection by the participant 12 at the beginning of the game activity, or can be based on a special item (e.g., a hyper-extension power-up) obtained by the virtual representation 14 in the virtual environment 32 during the game activity.

[0066] for Fig. 9 In the embodiment illustrated in , in response to the main controller 34 determining that the participant 12 is extending their arms to a maximum extent, the processing circuit 35 modifies both the appearance and movement of the virtual representation 14 such that the arms of the virtual representation 14 extend in an enhanced (e.g., enhanced, enlarged) manner. It will be appreciated that this may enable the virtual representation 14 to perform particular tasks in the virtual environment 32. In other cases, the virtual representation 14 may extend in different manners (e.g., from the legs, from the torso, from the neck) based on other movements or postures of the participant 12 during the gaming activity. This enhanced extension capability may typically enable the virtual representation to access elements (e.g., items, weapons, entrances / exits, enemies, allies) in the virtual environment 32 that would otherwise be inaccessible, thereby providing the participant 12 with an engaging and creative problem-solving experience. Additionally, although in Fig. 9 Super stretch capability is illustrated in FIG, but in other embodiments, other enhanced capabilities such as super speed, super strength, etc. may also be achieved according to the present disclosure.

[0067] In some embodiments, instead of fully reproducing the appearance and movement of the participant, the virtual representation can appear and move like a real or fictitious non-human entity (such as an animal virtual representation). In some embodiments, the participant can select a special animal-based virtual representation at the beginning of the game activity, and in other embodiments, the animal-based virtual representation can be automatically assigned based on the scan data and / or model associated with the participant. In some embodiments, once selected or assigned, the virtual representation can remain the same throughout the game activity, and in other embodiments, the virtual representation can change periodically, or change in response to the special movement or achievement of the participant (for example, different animal representations of different terrains in a virtual environment or different levels). When the virtual representation takes the form of a special animal, the virtual representation can have a special type of ability (for example, the type of movement) different from the ability of the participant 12, including some abilities that may be difficult or impossible for the participant 12 to actually perform (for example, trotting like a horse, jumping like a kangaroo, swimming like a fish, flying like a bird, etc.). Thus, the appearance and movement detected by the master controller 34 can be enhanced (e.g., expanded, enhanced) so that the participant 12 can use feasible, realistic human gestures and movements within the activity area 16, which are enhanced to generate movement of the animal-based virtual representation 14.

[0068] Fig.10 An example embodiment is illustrated in which a participant 12 is positioned in an activity area 16 during a gaming activity of an interactive video game system 10, and a corresponding animal-based virtual representation 14 capable of leaping and moving is presented on a display device 24. For the illustrated embodiment, the virtual representation 14 is a stag virtual representation 14, which may be selected by the participant 12 at the start of the gaming activity, or by the processing circuitry 35 of the master controller 34 based on scan data and / or a model associated with the participant 12. For example, in one embodiment, the stag virtual representation 14 is selected for the participant 12 by the master controller 34 upon detecting that the participant 12 has a "ponytail" or "braid" that vaguely resembles the antlers of a stag. For such an embodiment, the virtual representation 14 may include one or more features or characteristics that correspond to enhanced (e.g., enhanced, enlarged) features of the participant 12, generally in a manner similar to comic book art.

[0069] for Fig.10, the master controller 34 detects, based on the scan data and one or more models, that a participant 12 jumps across an activity area 16 during a game activity. The master controller 34 translates the detected movement of the participant 12 into appropriate movement for the stag virtual representation 14. In particular, the master controller 34 augments (e.g., strengthens, amplifies) the detected movement of the participant 12 so that the stag virtual representation 14 jumps and walks in the virtual environment 32 to a greater height and / or distance than detected (and potentially greater than may be possible for the participant 12). Additionally, as mentioned above, the participant 12 may use the augmented movement (e.g., jump, jump walk, bounding) capabilities exhibited by the virtual representation 14 to achieve particular goals in the virtual environment 32.

[0070] In some embodiments, one or more real-world characters (e.g., robotic elements, animated devices) may be part of the interactive video game system 10. For example, in some embodiments, in addition to or in lieu of the virtual representation 14, the interactive video game system 10 may include a robotic representation, such as a robotic stag representation. Like the stag virtual representation 14 discussed above, the robotic hart is controlled by the master controller 34 based on the detected movements of the participant 12, and the controller 34 may enhance (e.g., enhance, amplify) the detected movements of the participant 12 when determining how to move the robotic hart representation. Additionally, in some embodiments, the interactive video game system 10 may include other robotic elements, such as the illustrated robotic rabbit 150 and robotic squirrel 152. In some embodiments, the movement of these additional robotic elements 150, 152 may be controlled based on the movement of other participants in the activity area 16. In other embodiments, these additional robotic elements 150, 152 may move in response to what is happening in the virtual environment 32, the movement of the robot or virtual representation 14, or a combination thereof, to provide a more immersive experience that includes the movement of a 3D, real-world character.

[0071] Fig.11Another example of an animal-based virtual representation (e.g., an aquatic-type animal) is illustrated that implements augmented movement (e.g., swimming movement) relative to the detected movement of the participant. In the illustrated example, the participant 12 moves through the activity area 16 in an undulating manner (e.g., walking, lunging). In response to detecting this movement, the master controller 34 moves the dolphin virtual representation 14 in a corresponding undulating manner that is augmented (e.g., enlarged, enhanced) relative to the movement of the participant 12. Other participant movements can be detected and converted into movement of the dolphin virtual representation 14. For example, the master controller 34 can convert a detected jumping movement of the participant 12 into a significant breaking jump above the surface of the water in the virtual environment 32, or convert a detected swimming movement of the participant 12's arm into a tail flicking movement of the dolphin virtual representation 14. For one or more of these movements, the master controller 34 can augment (e.g., enlarge, enhance) the movement of the dolphin virtual representation 14 relative to the actual detected movement of the participant 12.

[0072] Fig.12 Another example of an animal-based virtual representation that implements enhanced flight movement relative to detected movement of a participant is illustrated. In the illustrated example, the participant 12 is positioned in the activity area 16 so that the master controller 34 detects and determines the movement of the participant 12 during game play of the interactive video game system 10. For the illustrated example, the participant 12 controls the bat virtual representation 14 so that when the participant poses or moves in a particular manner, these are translated and enhanced into movement of the bat's wings.

[0073] for Fig.11 In the example illustrated in FIG, in response to detecting a participant 12 extending their arms, the master controller 34 may extend the wings of the virtual representation of the bat. Furthermore, in response to detecting that the participant 12 is leaning left or right, the master controller 34 may cause the virtual representation of the bat 14 to lean and turn left or right in a corresponding manner. Additionally, in some embodiments, the participants 12 may flap their arms to cause the virtual representation of the bat 14 to flap to gain altitude, and the participants 12 may also tuck their arms to cause the bat to dive. Furthermore, for the illustrated embodiment, the display device 24 includes a plurality of screens 154 (e.g., screens 154A, 154B, and 154C), which may enable the virtual representation of the bat 14 to fly from one screen to another around at least a portion of the perimeter of the activity area 16. It will be appreciated that, although with respect to FIG. Figure 10-12Examples of stag, dolphin, and bat virtual representations 14 are discussed, but the same technology can be applied to other animals with other abilities and / or forms of movement. For example, it is anticipated that the technology can be applied to enable a participant to pose and move in unique ways to control an animal-based virtual representation (e.g., jump like a kangaroo, burrow like a meerkat, slide like a snake, etc.). It can also be appreciated that the technology can also be applied to virtual entities and animals, so that the disclosed system implements enhanced movement of the virtual representation 14 based on the detected position and movement of the participant, such as galloping like a unicorn or flying like a dragon.

[0074] Fig.13 An embodiment of an interactive video game system 10 is illustrated that includes enhanced physical effects associated with enhanced capabilities of a virtual representation 14. For the illustrated embodiment, a participant 12 is positioned in an active area 16 near (e.g., below) two physical effects devices 156 and 158. In other embodiments, the physical effects devices 156 and 158 may be located above the active area 16, integrated into the floor of the active area 16, integrated into the interface panel 74 (e.g., Figure 2 ) or otherwise positioned to direct physical effects toward participants in the activity area 16. Physical effects device 156 is a thermal physical effects device designed to provide thermal effects (e.g., infrared (IR) light, air waves of cold / hot air, air waves of hot or cold mist) to participants 12 corresponding to events occurring within the virtual environment 32. In contrast, physical effects device 158 is an ultrasonic haptic device that is capable of providing a physical sensation of touch to participants 12 through the air using ultrasonic waves, wherein the physical sensation corresponds to events occurring within the virtual environment 32.

[0075] More specifically, for Fig.13, the virtual representation 14 has received a power boost related to fire and is contacting an obstacle 160. That is, in the illustrated example, a first hand 162 of the virtual representation 14 is associated with a fire power boost, and a fire or sun symbol 164 is positioned near the first hand 162 of the virtual representation 14. Thus, the thermal physical effects device 156 may be an IR source (e.g., an IR bulb) activated by the master controller 34 and directed toward the first hand 166 of the participant 12. Additionally, a second hand 168 of the virtual representation 14 is illustrated as being in contact with the obstacle 160 in the virtual environment 32. Thus, the ultrasonic haptic physical effects device 158 is activated by the master controller 34 and directed toward the second hand 170 of the participant 12. Thus, the participant 12 experiences a more immersive experience by feeling physical effects based on events and situations occurring in the virtual environment 32. In this way, the interactive video game system 10 may provide enhanced feedback to the participant 12 that extends one or more aspects of the enhanced movement and capabilities into the participant's 12 real-world experience.

[0076] The technical effects of the present method include an interactive video game system that enables multiple participants (e.g., two or more, four or more) to perform actions in a physical activity area (e.g., a 2D or 3D activity area) to control corresponding virtual representations in a virtual environment presented on a display device near the activity area. The disclosed system includes multiple sensors and suitable processing circuits, which are configured to collect scan data and generate various models for each participant, such as a participant model, a shadow model, and / or a skeletal model. The system generates a virtual representation of each participant based at least in part on the generated participant model. Additionally, the interactive video game system can set or modify the properties of the virtual representation (such as size, texture, and / or color) based on various attributes associated with the participant (such as points, purchases, power-ups). Moreover, the interactive video game system implements enhanced movement (e.g., superhuman abilities, animal-based movement) that is enhanced or expanded relative to the actual detected movement of the participant in the activity area. Additionally, embodiments of the interactive video game system may include robotic devices and / or physical effects devices that provide feedback relative to these enhanced movements and abilities to provide participants with an immersive gaming experience.

[0077] Although only certain features of the present technology have been illustrated and described herein, many modifications and changes will occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the present technology. Additionally, the technology proposed and claimed herein is cited and applied to material objects and specific examples of practical nature, which have certainly improved the field of technology, and therefore are not abstract, intangible or purely theoretical. In addition, if any claim attached at the end of this specification contains one or more elements designated as "means for [performing] [function] ..." or "steps for [performing] [function] ...", it is intended to interpret such elements according to 35 U.S.C. 112 (f). However, for any claim containing elements designated in any other way, it is intended not to interpret such elements according to 35 USC 112 (f).

Claims

1. An interactive video game system comprising: at least one sensor disposed proximate to the activity area and configured to collect scan data of a participant positioned in the activity area; at least one display device disposed proximate the activity area and configured to present a virtual representation associated with the participant; as well as a controller communicatively coupled to the at least one sensor and the at least one display device, wherein the controller is configured to: Receiving the scan data of the participants in the activity area from the at least one sensor generating at least one model based on the received scan data of the participant; identifying movements of the participants in the activity area based on the generated at least one model; generating the virtual representation of the participant based on the generated at least one model and the identified movements of the participant; identifying one or more attributes of the participant, wherein the one or more attributes relate to performance by the participant in a game played via the interactive video game system; as well as A generated virtual representation of the participant in the virtual environment is presented on the at least one display device, wherein corresponding actions of the presented virtual representation are amplified relative to the identified actions of the participant based on the one or more attributes of the participant, and wherein the presented virtual representation includes a silhouette of the participant.

2. An interactive video game system according to claim 1, wherein at least one of the generated models includes a shadow model, and wherein the controller is configured to generate the virtual representation of the participant based on the shadow model of the participant.

3. An interactive video game system according to claim 1, wherein at least one of the generated models includes a skeletal model, and wherein the controller is configured to recognize the actions of the participant based on the skeletal model of the participant.

4. The interactive video game system of claim 1, wherein the controller is configured to: generating an animal-based virtual representation of the participant based on the generated at least one model and the identified movements of the participant; and The animal-based virtual representation of the participant in the virtual environment is presented on the display device, wherein the corresponding actions of the presented animal-based virtual representation correspond to the identified actions of the participant and are amplified relative to the identified actions of the participant based on the one or more attributes of the participant.

5. The interactive video game system of claim 1, wherein: The identified action of the participant is jumping, and the corresponding action of the presented virtual representation is jumping in a manner that is magnified relative to the jumping of the participant, or the identified action of the participant is running, and the corresponding action of the presented virtual representation is running in a manner that is magnified relative to the running of the participant.

6. The interactive video game system of claim 1, wherein the generated virtual representation is associated with super-human capabilities triggered by the identified actions of the participant and the one or more attributes of the participant, and wherein the controller is configured to: The generated virtual representation of the participant in the virtual environment is presented on the display device, wherein the corresponding actions of the virtual representation correspond to the super-human ability.

7. An interactive video game system according to claim 1, wherein the corresponding shape of the virtual representation corresponds to the corresponding shape of the participant, the identified action of the participant is flapping arms, and the corresponding action of the presented virtual representation is flying.

8. An interactive video game system according to claim 1, wherein the identified action of the participant is jumping to a first height, crossing a first distance, or both, and the corresponding action of the presented virtual representation is jumping to a second height greater than the first height, a second distance greater than the first distance, or both.

9. An interactive video game system according to claim 1, wherein the one or more attributes of the participant include a number of items obtained during the game or a number of points obtained during the game.

10. A method of operating an interactive video game system, comprising: receiving, via processing circuitry of a controller of the interactive video game system, scan data of a participant positioned within an active area; generating, via the processing circuit, a shadow model and a skeletal model of the participant based on the received scan data; generating, via the processing circuitry, a virtual representation of the participant based on the generated shadow model, wherein a corresponding shape of the virtual representation corresponds to a corresponding shape of the participant; identifying, via the processing circuit, actions of the participant in the activity area based on the generated skeletal model; identifying, via the processing circuitry, one or more attributes of the participant, wherein the one or more attributes include a first item acquired during a game played by the participant via the interactive video game system, a number of points earned during the game, a second item acquired in a physical store, or any combination thereof; as well as A generated virtual representation in a virtual environment performing a corresponding action is presented via a display device of the interactive video game system, wherein the corresponding action corresponds to the action of the participant and is enlarged relative to the action of the participant, the enlargement being responsive to recognizing the action of the participant in the active area and based on the one or more attributes of the participant.

11. The method according to claim 10, comprising: receiving, via the processing circuit, second scan data of a second participant positioned within the activity area; generating, via the processing circuit, a second shadow model and a second skeletal model of the second participant based on the received second scan data; generating, via the processing circuit, a second virtual representation of the second participant based on the generated second shadow model, wherein a corresponding shape of the second virtual representation corresponds to a corresponding shape of the second participant; identifying, via the processing circuit, a second action of the second participant in the activity area based on the generated second skeletal model; as well as A second virtual representation generated in the virtual environment performing a corresponding action is presented via the display device, wherein the corresponding action corresponds to a second action identified by the second participant and is enlarged relative to the second action identified by the second participant, the enlargement being in response to the second action identified by the second participant in the activity area.

12. An interactive video game system comprising a controller, the controller being configured to: receiving scan data of a participant in an activity area from at least one sensor of the interactive video game system; generating a shadow model and a skeletal model based on the received scan data of the participant; generating a virtual representation of the participant based on the generated shadow model; Recognizing the actions of the participant in the activity area based on the generated skeletal model of the participant; identifying one or more attributes of the participant, wherein the one or more attributes relate to performance by the participant in a game played via the interactive video game system; triggering a corresponding action of the generated virtual representation based on the identified action and the one or more attributes, wherein the corresponding action of the generated virtual representation corresponds to the identified action of the participant and is amplified relative to the identified action of the participant; as well as The generated virtual representation in the virtual environment in which the triggered corresponding action is performed is presented on a display device of the interactive video game system, wherein the presented virtual representation includes a silhouette of the participant.

13. An interactive video game system according to claim 12, wherein the generated virtual representation includes an animal-based virtual representation, the animal-based virtual representation includes an aquatic animal, the recognized action includes a swimming movement of the participant's arm, and the corresponding action includes swimming.

14. An interactive video game system according to claim 12, wherein the generated virtual representation includes an animal-based virtual representation, and the animal-based virtual representation is an animal that jumps and walks, the identified action includes a jump of the participant, and the corresponding action includes jumping and walking.

15. The interactive video game system of claim 12, wherein the corresponding actions include super-human abilities.

16. The interactive video game system of claim 12, wherein the controller is configured to: One or more physical effects devices of the interactive video game system are activated in response to recognizing the action of the participant.

17. An interactive video game system according to claim 16, wherein the activated one or more physical effects devices include a thermal physical effects device, which is configured to heat or cool a portion of the participant in response to recognizing the action of the participant.

18. An interactive video game system according to claim 16, wherein the activated one or more physical effects devices include an ultrasonic tactile physical effects device, and the ultrasonic tactile physical effects device is configured to provide a touch sensation to a portion of the participant in response to recognizing the action of the participant.

19. The interactive video game system of claim 12, wherein the controller is configured to activate one or more robotic devices proximate the activity area in response to recognizing the action of the participant.

20. The interactive video game system of claim 12, wherein the controller is configured to modify at least one of a size, a color, or a texture of the virtual representation of the participant during the game based on the one or more attributes.

21. An interactive video game system comprising: at least one sensor disposed proximate to the activity area and configured to collect scan data of a participant positioned in the activity area; at least one display device disposed proximate the activity area and configured to present a virtual representation associated with the participant; as well as a controller communicatively coupled to the at least one sensor and the at least one display device, wherein the controller is configured to: Receiving the scan data of the participants in the activity area from the at least one sensor generating at least one model based on the received scan data of the participant; identifying movements of the participants in the activity area based on the generated at least one model; generating the virtual representation of the participant based on the generated at least one model and the identified movements of the participant; identifying one or more attributes of the participant, wherein the one or more attributes relate to performance by the participant in a game played via the interactive video game system, and wherein the one or more attributes include a number of items acquired during the game or a number of points earned during the game; as well as A generated virtual representation of the participant in the virtual environment is presented on the at least one display device, wherein corresponding actions of the presented virtual representation are amplified relative to the identified actions of the participant based on the one or more attributes of the participant.

22. An interactive video game system comprising: A controller, the controller being configured to: receiving scan data of participants in an activity area from at least one sensor of the interactive video game system; generating a shadow model and a skeleton model based on the received scan data of the participant; generating a virtual representation of the participant based on the generated shadow model; Recognizing the actions of the participant in the activity area based on the generated skeletal model of the participant; identifying one or more attributes of the participant, wherein the one or more attributes relate to performance by the participant in a game played via the interactive video game system; triggering a corresponding action of a generated virtual representation based on the identified action and the one or more attributes, wherein the corresponding action of the generated virtual representation corresponds to the identified action of the participant and is amplified relative to the identified action of the participant; as well as presenting, on a display device of the interactive video game system, the generated virtual representation in the virtual environment in which the triggered corresponding action is performed, wherein the generated virtual representation includes an animal-based virtual representation, and wherein: The animal-based virtual representation includes an aquatic animal, the identified action includes a swimming motion of an arm of the participant, and the corresponding action includes swimming; or The animal-based virtual representation is a jumping animal, the identified action includes a jump of the participant, and the corresponding action includes jumping.

23. An interactive video game system comprising: A controller, the controller being configured to: receiving scan data of participants in an activity area from at least one sensor of the interactive video game system; generating a shadow model and a skeleton model according to the scan data of the participant; generating a virtual representation of the participant based on the shadow model; identifying a movement of the participant in the activity area based on the skeletal model of the participant, wherein the movement triggers an augmentation capability associated with the virtual representation; as well as presenting, on a display device of the interactive video game system, the virtual representation in a virtual environment in which the augmented capability is executed, wherein the virtual representation includes an animal-based virtual representation, and the augmented capability includes a method of movement associated with the animal-based virtual representation, and wherein: The animal-based virtual representation includes an aquatic animal and the augmented ability is swimming; or The animal-based virtual representation is an animal that walks by jumping, and the augmented ability is walking by jumping.

24. An interactive entertainment system comprising: a plurality of sensing units, wherein each of the plurality of sensing units is configured to collect corresponding scan data of a person positioned in an interaction area and is configured to generate a corresponding partial model of the person by a corresponding processing circuit of the sensing unit; a video display disposed proximate the interaction area and configured to present a virtual representation associated with the person; and a controller communicatively coupled to the plurality of sensing units and the video display, wherein the controller is configured to: receiving the corresponding portion model of the person from each of the plurality of sensing units; fusing each received partial model to generate a model of the person; generating the virtual representation of the person based on the model and the identified movements; and The generated virtual representation of the person is presented on the video display in a virtual environment, the virtual representation performing augmented movements related to the identified actions of the person and enhanced according to the model.

25. The interactive entertainment system of claim 24, wherein the generated virtual representation of the person is a silhouette of the person.

26. The interactive entertainment system of claim 24, wherein the model comprises a shadow model, and the controller is configured to generate the virtual representation based on the shadow model.

27. The interactive entertainment system of claim 24, wherein the model comprises a skeletal model of the person, and wherein the controller is configured to recognize the actions of the person based on the skeletal model of the person.

28. The interactive entertainment system of claim 24, wherein the controller is configured to generate a virtual object and present the virtual object on the video display based on the identified motion of the person.

29. The interactive entertainment system of claim 28, wherein the identified action of the person is a throwing motion and the virtual object is a thrown object.

30. The interactive entertainment system of claim 29, wherein the enhanced motion corresponds to the throwing motion and an augmented virtual action of the thrown object.

31. The interactive entertainment system of claim 24, wherein the generated virtual representation depicts super-human capabilities triggered by the identified actions of the person.

32. The interactive entertainment system of claim 24, comprising a movable object model configured to generate a virtual moving object.

33. An interactive entertainment system according to claim 32, wherein the controller is configured to render the virtual moving object to move on the video display in coordination with the generated virtual representation of the person based on the movable object model and the model.

34. A method of operating an interactive entertainment system, the method comprising: receiving, via processing circuitry of a controller, a corresponding portion model of a person from each of a plurality of sensing units, wherein each of the plurality of sensing units is configured to collect corresponding scan data of the person and generate the corresponding portion model of the person by corresponding processing circuitry of the sensing unit; fusing, via the processing circuit of the controller, each received partial model to generate a model of the person; identifying, via the processing circuit of the controller, a movement of the person based on the model; generating, via the processing circuitry of the controller, a virtual representation of the person based on the model and the movements; and The generated virtual representation is presented in a virtual environment via a video display viewable by the person, the virtual representation performing virtual actions corresponding to and augmented with respect to the actions of the person.

35. The method of claim 34, comprising identifying a physical item held or worn by the person.

36. The method of claim 35, wherein the virtual action is based on a characteristic of the physical object.

37. The method of claim 34, comprising generating the virtual representation of the person as a silhouette.

38. The method of claim 34, comprising generating, via the processing circuitry of the controller, a virtual moving object in coordination with the virtual representation of the person. The method of claim 38 , wherein the virtual moving object is presented based on the virtual motion.

40. An interactive entertainment system comprising a controller, wherein the controller is configured to: receiving a corresponding partial model of the person from each of the plurality of sensing units, wherein: Each of the plurality of sensing units is configured to collect corresponding scan data of the person, and generate the corresponding part model of the person by a corresponding processing circuit of the sensing unit; fusing each received partial model to generate a model of the person; recognizing an action of the person based on the model of the person; generating a virtual representation of the person based on the model and the identified movements; activating an enhanced activity of the virtual representation based on the recognized action, wherein the enhanced activity comprises a virtual enlargement of the recognized action; as well as On a video display, the virtual representation in a virtual environment performing the enhanced activity is presented.

41. An interactive entertainment system according to claim 40, wherein the controller is configured to generate a virtual moving object based on the identified motion, the enhanced activity, or both.

42. The interactive entertainment system of claim 40, wherein the controller is configured to generate the virtual representation as a silhouette of the person.

43. The interactive entertainment system of claim 40, wherein the controller is configured to generate a virtual moving object based on the identified motion of the person corresponding to a throwing or kicking motion and to present the virtual moving object on the video display.

Images