Interactive video game system

Abstract

To provide an interactive video game system that enables simultaneous multi-player game play.SOLUTION: An interactive video game system includes at least one sensor and at least one display device that are disposed near a play area. The system also includes a controller communicatively coupled to the at least one sensor and the at least one display device, where the controller is configured to: receive, from the at least one sensor, scanning data of the player in the play area; generate at least one model from the scanning data of the player; identify an action of the player in the play area based on the at least one model; generate a virtual representation for the player based on the at least one model and the action of the player; and present, on the display device, the virtual representation of the player in a virtual environment, where an action of the virtual representation is augmented relative to the action of the player.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to video game systems, and more particularly to interactive video game systems that enable simultaneous multiplayer game play. [Background technology]

[0002] Generally, video game systems allow players to control characters within a virtual environment to achieve predetermined goals or objectives. Traditional video game systems typically rely on manual input devices, such as joysticks, game controllers, and keyboards, to enable players to control characters within the virtual game environment. Some modern video game systems also include cameras that track player movements, allowing players to control video game characters based on their movements. However, these systems typically suffer from occlusion issues, in which portions of the player are at least temporarily obscured from the camera, resulting in the system being unable to accurately track the player's position or movement. For example, occlusion can cause jittering or stuttering in the character's movements within the virtual environment, as well as other issues that can lead to inaccurate or erroneous translation of player movements into in-game character movements. In addition, in multiplayer video game systems, the likelihood of occlusion increases dramatically with the number of players. Summary of the Invention [Means for solving the problem]

[0003] The present embodiment relates to an interactive video game system including at least one sensor disposed near a play area and at least one display device, the system including a controller communicatively coupled to the at least one sensor and the at least one display device, the controller configured to receive scanning data of a player within the play area from the at least one sensor, generate at least one model from the scanning data of the player, identify player movements within the play area based on the at least one model, generate a virtual representation of the player based on the at least one model and the player movements, and present, on the display device, the virtual representation of the player in the virtual environment, the movements of the virtual representation augmented with respect to the player movements.

[0004] The present embodiment also relates to a method of operating an interactive video game system, the method including receiving, via processing circuitry of a controller of the interactive video game system, scan data of a player positioned within a play area, generating, via the processing circuitry, a shadow model and a skeletal model of the player based on the scan data, generating, via the processing circuitry, a virtual representation associated with an augmented ability of the player based on the shadow model, identifying, via the processing circuitry, based on the skeletal model, a motion that triggers the augmented ability associated with the virtual representation of the player within the play area, and presenting, via a display device of the interactive video game system, the virtual representation within a virtual environment that executes the augmented ability.

[0005] The present embodiment also relates to an interactive video game system including a controller configured to receive scanning data of a player within a play area from at least one sensor of the interactive video game system, generate a shadow model and a skeletal model from the scanning data of the player, generate a virtual representation of the player based on the shadow model, identify actions that trigger an augmented ability associated with the virtual representation of the player within the play area based on the skeletal model of the player, and present the virtual representation within the virtual environment on a display device of the interactive video game system, executing the augmented ability.

[0006] These and other features, aspects, and advantages of the present disclosure will be better understood from the following detailed description when read in conjunction with the accompanying drawings, in which like parts are designated by like numerals throughout. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram of an embodiment of an interactive video game system that allows multiple players to control their respective virtual representations by performing actions within a three-dimensional (3D) play area in accordance with the technology of the present invention. [Figure 2] 1 is a schematic diagram of another embodiment of an interactive video game system having a two-dimensional (2D) play area in accordance with the present technology. [Figure 3] 1A-1C illustrate examples of skeletal and shadow models representing a player within a 3D play area and a corresponding virtual representation of the player presented within the virtual environment in accordance with the present technology. [Figure 4] 1A-1C illustrate examples of skeletal and shadow models representing a player within a 2D play area and a corresponding virtual representation of the player presented within a virtual environment in accordance with the techniques of the present invention. [Figure 5] FIG. 1 is a flow diagram illustrating an embodiment of a process for operating an interactive gaming system in accordance with the present technology. [Figure 6]6 is a flow diagram illustrating an example embodiment of a process by which an interactive video game system performs certain operations illustrated in the flow diagram of FIG. 5 in accordance with the present technology. [Figure 7] 1 illustrates an example embodiment of an interactive video game system that enables the generation of a virtual representation having an appearance and / or movement that is augmented with respect to the appearance and / or movement of a detected player. [Figure 8] 1 illustrates an example embodiment of an interactive video game system that enables the generation of a virtual representation having an appearance and / or movement that is augmented with respect to the appearance and / or movement of a detected player. [Figure 9] 1 illustrates an example embodiment of an interactive video game system that enables the generation of a virtual representation having an appearance and / or movement that is augmented with respect to the appearance and / or movement of a detected player. [Figure 10] 1 illustrates an example embodiment of an interactive video game system that enables the generation of a virtual representation having an appearance and / or movement that is augmented with respect to the appearance and / or movement of a detected player. [Figure 11] 1 illustrates an example embodiment of an interactive video game system that enables the generation of a virtual representation having an appearance and / or movement that is augmented with respect to the appearance and / or movement of a detected player. [Figure 12] 1 illustrates an example embodiment of an interactive video game system that enables the generation of a virtual representation having an appearance and / or movement that is augmented with respect to the appearance and / or movement of a detected player. [Figure 13] 1 illustrates an example embodiment of an interactive video game system that enables the generation of a virtual representation having an appearance and / or movement that is augmented with respect to the appearance and / or movement of a detected player. DETAILED DESCRIPTION OF THE INVENTION

[0008] As used herein, "scanning data" means two-dimensional (2D) or three-dimensional (3D) data collected by sensing (e.g., measuring, imaging, ranging) the visible exterior of a player within a playing area. Specifically, as used herein, "volumetric scanning data" refers to 3D scanning data, such as point cloud data, and can be contrasted with "2D scanning data," such as image data.

[0009] As used herein, a "player model" is a 2D or 3D model generated from scan data of a player that roughly represents the player's exterior and may include texture data. Specifically, as used herein, a "volumetric player model" or "volumetric model" refers to a 3D player model generated from volumetric scan data of a player, and can be contrasted with a "2D player model" generated from 2D scan data of a player.

[0010] As used herein, "shadow model" means a textureless volumetric model of a player that is generated directly from the player's scan data or through a player model, such that when presented on a 2D surface such as a display device, the player's shadow model has a shape substantially similar to the player's shadow or silhouette when backlit.

[0011] As used herein, "skeletal model" means a 3D model generated from scanned data of a player that defines the expected locations and positions of certain bones of the player (e.g., bones associated with the arms, legs, head, and spine) to represent the player's location and pose within the play area. The skeletal model is thus used to determine the player's movements and actions within the play area that cause events within the virtual environment and / or within the play area.

[0012] The present embodiments relate to an interactive video game system that enables multiple (e.g., up to 12) players to perform actions within a physical play area to control virtual representations of the players within a displayed virtual environment. The disclosed interactive video game system includes one or more sensors (e.g., cameras, optical sensors, infrared (IR) sensors) positioned around the play area to capture scanning data (e.g., 2D or volumetric scanning data) of the players. For example, some embodiments of the disclosed interactive video game system include an array having two or more volumetric sensors, such as a depth camera and a light detection and ranging (LIDAR) device, that can volumetrically scan each player. As described below, the system includes suitable processing circuitry that generates a model (e.g., a player model, a shadow model, a skeletal model) of each player based on the scanning data collected by the one or more sensors. During game play, the one or more sensors capture the player's actions within the play area, and the system determines the nature of these actions based on the generated player models. Thus, the interactive video game system continuously updates the virtual representation of the player and the virtual environment based on the player's actions and their corresponding in-game effects.

[0013] As described above, the disclosed interactive video game system includes one or more sensors positioned around the play area to monitor the movements of players within the play area. For example, in some embodiments, an array including multiple sensors may be used to ensure that a skeletal model of each player can be accurately generated and updated throughout game play, generally despite potential occlusion from the perspective of one or more sensors in the array. In other embodiments, fewer sensors (e.g., a single camera) may be used, and the data may be processed using a machine learning algorithm that generates a complete skeletal model of the player despite potential occlusion. In such embodiments, a machine learning agent may be pre-trained using a corpus of scan data where the actual skeletal models of the players are known (e.g., identified manually by a human or using a separate skeletal tracking algorithm) while one or more portions of the player are occluded. Thus, after training, the machine learning agent can generate skeletal models of the players from the scan data despite potential occlusion.

[0014] The system's processing circuitry may also use the scanning data to generate aspects (e.g., size, shape, contours) of each player's virtual representation within the virtual environment. In some embodiments, some aspects (e.g., color, texture, scale) of each player's virtual representation may be further adjusted or modified based on information related to the player. As described below, this information may include information related to gameplay (e.g., items acquired, achievements unlocked), as well as other information related to the player's activities outside the game (e.g., player achievements in other games, items purchased by the player, locations visited by the player). Additionally, the gaming system's processing circuitry may use the scanning data collected by the sensors to generate additional content, such as a commemorative image in which the player model is shown as present within the virtual world.

[0015] Additionally, the system's processing circuitry can use the scan data to enhance the movements of each player's virtual representation. For example, in some embodiments, the system's processing circuitry can use the scan data to generate a skeletal model that indicates the player moving or posing in a particular way. In response, the processing circuitry can enhance or enhance the movement and / or appearance of the virtual representation by augmenting the player's virtual representation to enable movements or changes in the virtual representation that transcend the player's actual movements or poses. For example, in embodiments in which a virtual representation, such as a particular video game character, has particular enhanced abilities (e.g., the ability to jump very high, the ability to swim very fast, the ability to fly), certain movements or poses of the player (e.g., small hops, swimming through the air, flapping wings) can be detected to trigger the enhanced abilities of the player's virtual representation. Thus, the disclosed interactive video game system enables an immersive and engaging experience for multiple simultaneous players.

[0016] With this in mind, FIG. 1 is a schematic diagram of an embodiment of an interactive video game system 10 that enables multiple players 12 (e.g., players 12A and 12B) to each control a respective virtual representation 14 (e.g., virtual representations 14A and 14B) by performing actions within a play area 16. Note that for simplicity, this description is directed to two players 12 using the interactive video game system 10, although in other embodiments, the interactive video game system 10 may support more than two players 12 (e.g., six, eight, ten, twelve, or more). The play area 16 of the interactive video game system 10 shown in FIG. 1 is described herein as a 3D play area 16A. As used herein, the term "3D play area" means that play area 16A has a width (corresponding to x-axis 18), a height (corresponding to y-axis 20), and a depth (corresponding to z-axis 22), and that system 10 generally monitors the movement of each player 12 along x-axis 18, y-axis 20, and z-axis 22. Interactive video game system 10 updates the position of virtual representation 14 presented on display device 24 along x-axis 26, y-axis 28, and z-axis 30 within virtual environment 32 in response to player 12 moving about play area 16A. While 3D play area 16A is shown as generally circular, in other embodiments, 3D play area 16A may be square, rectangular, hexagonal, octagonal, or any other suitable 3D shape.

[0017] The embodiment of interactive video game system 10 shown in FIG. 1 includes a primary controller 34 having a memory circuit 33 and a processing circuit 35 that generally provides control signals that control the operation of system 10. Accordingly, primary controller 34 is communicatively coupled to an array 36 of sensing units 38 arranged around the periphery of 3D play area 16A. Specifically, array 36 of sensing units 38 can be described as being symmetrically distributed around the periphery of play area 16A. In some embodiments, at least a portion of array 36 of sensing units 38 can be positioned above play area 16 (e.g., suspended from the ceiling or on an elevated platform or stand) and oriented at a downward angle to image play area 16. In other embodiments, at least a portion of array 36 of sensing units 38 can be positioned near the floor of play area 16 and oriented at an upward angle to image play area 16. In some embodiments, the array 36 of the interactive video game system 10 may include at least two sensing units 38 for each player (e.g., players 12A and 12B) in the play area 16. Thus, in some embodiments, the array 36 of sensing units 38 is suitably positioned to image a substantial portion of potential vantage points around the play area 16 to reduce or eliminate potential player occlusion. However, as noted above, in other embodiments, the array 36 may include fewer sensing units 38 (e.g., a single sensing unit), and the processing circuitry 35 may rely on machine learning agents to address potential occlusion situations.

[0018] In the illustrated embodiment, each sensing unit 38 includes a respective sensor 40, which may be a volumetric sensor (e.g., an infrared (IR) depth camera, a LIDAR device, or another suitable ranging device) or a 2D imaging device (e.g., an optical camera). For example, in some embodiments, the sensors 40 of the sensing units 38 in the array 36 are all IR depth cameras or LIDAR devices, while in other embodiments, there is a mix of IR depth cameras, LIDAR devices, and / or optical cameras in the array 36. It is now recognized that, as described below, both IR depth cameras and LIDAR devices may be used to volumetrically scan each player 12, and the collected volumetric scan data may be used to generate various models of the player. For example, in some embodiments, the IR depth cameras in the array 36 may be used to collect data and generate a skeletal model, while data collected by the LIDAR devices in the array 36 may be used to generate a player model and / or a shadow model of the player 12, as described in more detail below. It is also recognized that LIDAR devices that collect point cloud data are generally capable of scanning and mapping a larger area than depth cameras, typically with greater accuracy and resolution. Accordingly, in some embodiments, at least one sensing unit 38 of the array 36 includes a corresponding volume sensor 40 that is a LIDAR device in order to increase the accuracy or resolution of the array 36 and / or reduce the total number of sensing units 38 present in the array 36.

[0019] Additionally, each of the illustrated sensing units 38 includes a sensor controller 42 having suitable memory circuitry 44 and processing circuitry 46. The processing circuitry 46 of each sensing unit 38 executes instructions stored in the memory circuitry 44 to enable the sensing unit 38 to scan the players 12 and generate scan data (e.g., volumetric scan data and / or 2D scan data) of each player 12. For example, in the illustrated embodiment, the sensing units 38 are communicatively coupled to the primary controller 34 via a high-speed Internet Protocol (IP) network 48 that enables low-latency data exchange between devices of the interactive video game system 10. Additionally, in some embodiments, each of the sensing units 38 may include a respective housing that packages the sensor controller 42 along with the sensors 40.

[0020] It should be noted that in other embodiments, the sensing unit 38 may not include a respective sensor controller 42. In such embodiments, the processing circuitry 35 of the primary controller 34, or other suitable processing circuitry of the system 10, is communicatively coupled to each sensor 40 of the array 36 to provide control signals directly to and receive control signals from the sensors 40. However, it is now recognized that processing (e.g., filtering, skeletal mapping) the volumetric scan data collected by each of these sensors 40 is processor-intensive. Accordingly, in some embodiments, it may be advantageous to divide the workload by utilizing a dedicated processor (e.g., processor 46 of each sensor controller 42) to process the scan data collected by each sensor 40 before transmitting the processed data to the primary controller 34. For example, in the illustrated embodiment, each processor 46 in the sensor controller 42 processes the scanning data collected by its respective sensor 40 to generate partial models (e.g., partial volumetric or 2D models, partial skeletal models, partial shadow models) of each player 12, and the processing circuitry 35 in the primary controller 34 receives and fuses or combines these partial models to generate a complete model of each player 12, as described below.

[0021] In some embodiments, the primary controller 34 may also receive information from other sensing devices in and around the play area 16. For example, the illustrated primary controller 34 is communicatively coupled to radio frequency (RF) sensors 45 positioned near (e.g., above, below, adjacent to) the 3D play area 16A. The illustrated RF sensors 45 receive uniquely identifying RF signals from wearable devices 47, such as bracelets or headbands with radio frequency identification (RFID) tags, worn by each player 12. In response, the RF sensors 45 provide signals to the primary controller 34 regarding the identities and relative positions of the players 12 within the play area 16. Thus, in the illustrated embodiment, the processing circuitry 35 of the primary controller 34 receives and combines data collected by the array 36 and potentially other sensors (e.g., the RF sensors 45) to determine the identities, locations, and movements of the players 12 within the play area 16 during game play. The illustrated primary controller 34 is also communicatively coupled to a database system 50 or any other suitable data repository that stores player information. The database system 50 includes processing circuitry 52 that executes instructions stored in memory circuitry 54 to store and retrieve information related to player 12, such as various models associated with the player (e.g., player model, shadow model, and / or skeletal model), player statistics (e.g., wins, losses, scores, total game play time), player attributes or inventory (e.g., abilities, textures, items), player purchases in a gift shop, and player points in a loyalty rewards program. The processing circuitry 35 of the primary controller 34 queries, retrieves, and updates the information stored by the database system 50 about player 12 to enable the system 10 to operate as described herein.

[0022] The embodiment of interactive video game system 10 shown in FIG. 1 also includes an output controller 56 communicatively coupled to primary controller 34. Generally, output controller 56 includes processing circuitry 58 that executes instructions stored in memory circuitry 60 to control stimulus outputs (e.g., audio signals, video signals, lights, physical effects) observed and experienced by players 12 within play area 16. Thus, the illustrated output controller 56 is communicatively coupled to audio devices 62 and display device 24 to provide suitable control signals for operating these devices to provide particular outputs. In other embodiments, output controller 56 may be coupled to any number of audio and / or display devices. Display device 24 may be any suitable display device, such as a projector and screen, a flat-panel display device, or an array of flat-panel display devices, positioned and designed to provide players 12 within play area 16 with a suitable view of virtual environment 32. In some embodiments, audio devices 62 may be arranged in an array around play area 16 to enhance player immersion during gameplay. For example, in some embodiments, the primary controller 34 may independently control each hearing device 62 (e.g., each speaker) in such an array so that each player 12 hears different sounds specific to their actions that are distinct from other players. In still other embodiments, the play area 16 may include robotic elements (e.g., androids, robotic animals, etc.) that can act in the real world in response to signals provided by an output controller based on the actions of the players during gameplay. For example, in addition to or instead of virtual representations of the players presented on the display device 24, a robotic representation of the players 12 may provide a non-virtual representation that is controlled in response to the movements and behavior of the players 12 within the play area 16. In other embodiments, the system 10 may not include an output controller 56, but rather the processing circuitry 35 of the primary controller 34 may be communicatively coupled to the hearing devices 62, the display device 24, or the like to generate various stimuli observed and experienced by the players 12 within the play area 16.

[0023] FIG. 2 is a schematic diagram of another embodiment of an interactive video game system 10 that enables multiple players 12 (e.g., players 12A and 12B) to control virtual representations 14 (e.g., virtual representations 14A and 14B) by performing actions within a play area 16. The embodiment of interactive video game system 10 shown in FIG. 2 includes many of the features described herein with respect to FIG. 1, including a primary controller 34, an array 36 of sensing units 38, an output controller 56, and a display device 24. However, the embodiment of interactive video game system 10 shown in FIG. 2 will be described herein as having a 2D play area 16B. As used herein, the term “2D play area” means that the play area 16 has a width (corresponding to the x-axis 18) and a height (corresponding to the y-axis 20), and that the system 10 generally monitors the movement of each player 12 along the x-axis 18 and the y-axis 20. 2 , players 12A and 12B are assigned areas 70A and 70B, respectively, of 2D play area 16B, and players 12 do not move outside their assigned areas during game play. However, other embodiments of interactive video game system 10 may include a sufficient number of sensors (e.g., LIDAR sensors or other suitable sensors 40 positioned above the players) to continuously track each player 12 as they move freely throughout play area 16B, and the system may be understood to take into account potential occlusions by other players as they move. In response to player 12 moving within 2D play area 16B (e.g., running along x-axis 18 or jumping along y-axis 20), interactive video game system 10 updates the position of virtual representation 14 presented on display device 24 along x-axis 26 and y-axis 28 within virtual environment 32. As noted above, in some embodiments, array 36 may include fewer sensors (e.g., a single camera).

[0024] The embodiment of the interactive video game system 10 shown in FIG. 2 also includes an interface panel 74 that can enable enhanced player interaction. As shown in FIG. 2, the interface panel 74 includes a plurality of input devices 76 (e.g., cranks, wheels, buttons, sliders, blocks) designed to receive input from the player 12 during game play. As such, the illustrated interface panel 74 is communicatively coupled to the primary controller 34 to provide signals to the controller 34 indicative of how the player 12 is manipulating the input devices 76 during game play. The illustrated interface panel 74 also includes a plurality of output devices 78 (e.g., auditory output devices, visual output devices, physical stimuli devices) designed to provide auditory, visual, and / or physical stimuli to the player 12 during game play. As such, the illustrated interface panel 74 is communicatively coupled to the output controller 56 to receive control signals and provide appropriate stimuli to the player 12 within the play area 16 in response to appropriate signals from the primary controller 34. For example, the output devices 78 can include auditory devices such as speakers, horns, and sirens. Output devices 78 may also include visual devices such as lights or display devices on interface panel 74 .

[0025] In some embodiments, output device(s) 78 of interface panel 74 include physical effect devices, such as electronically controlled release valves coupled to compressed air lines that emit a jet of warm or cool air or mist in response to suitable control signals from primary controller 34 or output controller 56. It can be appreciated that output devices are not limited to those integrated into interface panel 74. In some embodiments, play area 16 may include output devices that indirectly provide physical effects to a player, such as through air. For example, in some embodiments, a player may experience a corresponding physical effect when striking a particular pose that triggers an ability or action of a virtual representation. As a specific example, in an embodiment where a player can throw snowballs, a gust of cool air may be directed at the player's open palms in response to the player spreading their arms in a particular manner. In an embodiment where a player can throw fireballs, a gust of warm air or IR radiation (e.g., heat) may be directed at the player in response to the player spreading their arms in a particular manner. In yet other embodiments, a player may receive haptic feedback (e.g., ultrasonic haptic feedback) in response to the player's virtual representation interacting with an object in the virtual world. For example, when the player's virtual representation punches a wall in the virtual environment, the player may experience some physical perceptual effect on a part of their body (e.g., their outstretched fist) that corresponds to the action in the virtual environment.

[0026] As shown in FIG. 2 , the array 36 of sensing units 38 arranged around the 2D play area 16B of the illustrated embodiment of the interactive video game system 10 includes at least one sensing unit 38. That is, some embodiments of the interactive video game system 10 shown in FIG. 1 include an array 36 having at least two sensing units 38 per player, while the embodiment of the interactive video game system 10 shown in FIG. 2 includes an array 36 having as few as one sensing unit 38, regardless of the number of players. In some embodiments, the array 36 may include at least two sensing units arranged perpendicular (90°) to the players 12 in the 2D array area 16B. In some embodiments, the array 36 may additionally or alternatively include at least two sensing units arranged on opposite sides (180°) of the players 12 in the play area 16B. As a specific example, in some embodiments, the array 36 may include only two sensing units 38 arranged on different (e.g., opposite) sides of the players 12 in the 2D play area 16B.

[0027] As described above, the array 36 shown in FIGS. 1 and 2 may collect scan data (e.g., volumetric data or 2D scan data) of each player 12 within the play area 16. In some embodiments, as described below, the collected scan data may be used to generate various models of each player (e.g., player model, shadow model, skeletal model), and then update these models based on the player's movements during gameplay. However, it is currently recognized that using volumetric models that include texture data may be significantly more processor-intensive (e.g., involving additional filtering, additional data processing) than using shadow models that lack this texture data. For example, in some embodiments, the processing circuitry 35 of the primary controller 34 may generate a shadow model of each player 12 from the scan data (e.g., 2D scan data) collected via the array 36 by using an edge detection method that distinguishes between the edges of the player 12 and the surroundings of the player 12 within the play area 16. It is currently recognized that such edge detection methods are less processor-intensive and require significantly less filtering than using volumetric models that include texture data. Accordingly, it is presently recognized that some embodiments of interactive video game system 10 generate and update shadow models, instead of volumetric models including textures, which allows for reduced size, complexity, and cost of processing circuitry 35 of primary controller 34. Additionally, as described below, processing circuitry 35 may generate virtual representation 14 of player 12 based at least in part on the generated shadow model.

[0028] As described above, the scanning data collected by the array 36 of the interactive video game system 10 can be used to generate various models of each player (e.g., 2D or volumetric player models, shadow models, skeletal models). For example, FIG. 3 illustrates skeletal models 80 (e.g., skeletal models 80A and 80B) and shadow models 82 (e.g., shadow models 82A and 82B) representing players within the 3D play area 16A. FIG. 3 also illustrates corresponding virtual representations 14 (e.g., virtual representations 14A and 14B) of these players presented within the virtual environment 32 on the display device 24 in accordance with the present technology. As shown, during game play, the represented players reside in different positions within the 3D play area 16A of the interactive video game system 10 as indicated by the locations of the skeletal models 80 and shadow models 82. The virtual representations 14 of the players within the illustrated virtual environment 32 are generated based at least in part on the player's shadow models 82. As described above, as players move within the 3D play area 16A, the primary controller 34 tracks these movements and generates updated skeletal models 80 and shadow models 82, and virtual representations 14 of each player accordingly.

[0029] 1 and 3 also enable movement and tracking of a player along the z-axis 22 and translating this movement into movement of a virtual representation 14 along the z-axis 30. As shown in FIG. 3, this allows a player represented by skeletal model 80A and shadow model 82A to move to a front edge 84 of the 3D play area 16A, resulting in a corresponding virtual representation 14A being presented at a relatively deeper point or height 86 along the z-axis 30 within the virtual environment 32. This allows a player represented by skeletal model 80B and shadow model 82B to move to a rear edge 88 of the 3D play area 16A, resulting in a corresponding virtual representation 14B being presented at a significantly shallower point or height 90 along the z-axis 30 within the virtual environment 32. Furthermore, in the illustrated embodiment, the size of the presented virtual representation 14 is modified based on the player's position along the z-axis 22 within the 3D play area 16A. That is, a virtual representation 14A positioned relatively deep along the z-axis 30 within the virtual environment 32 is presented as being significantly smaller than a virtual representation 14B positioned at a shallower depth or layer along the z-axis 30 within the virtual environment 32.

[0030] 1 and 3, virtual representations 14 can only interact with virtual objects located at a similar depth along the z-axis 30 within the virtual environment 32. For example, in the embodiment shown in FIG. 3, virtual representation 14A can interact with a virtual object 92 located deep along the z-axis 30 within the virtual environment 32, while virtual representation 14B can interact with another virtual object 94 located at a relatively shallow depth along the z-axis 30 within the virtual environment 32. That is, virtual representation 14A cannot interact with virtual object 94 unless the player, represented by models 80A and 82A, changes his or her position along the z-axis 22 within the 3D play area 16A to move virtual representation 14A to a similar depth within the virtual environment 32 as virtual object 94.

[0031] For comparison, FIG. 4 illustrates example skeletal models 80 (e.g., skeletal models 80A and 80B) and shadow models 82 (e.g., shadow models 82A and 82B) representing players within 2D play area 16B. FIG. 4 also illustrates virtual representations 14 (e.g., virtual representations 14A and 14B) of players presented on display device 24. As discussed above, as players move within 2D play area 16B, primary controller 34 tracks these movements and updates each player's skeletal model 80, shadow model 82, and virtual representation 14 accordingly. As discussed above, the embodiment of interactive video game system 10 having 2D play area 16B shown in FIGS. 2 and 4 does not track player movement along the z-axis (e.g., z-axis 22 shown in FIGS. 1 and 3). Instead, in embodiments having 2D play area 16B, the size of the presented virtual representation 14 may be modified based on the player's state or condition within and / or outside of gameplay. 4, virtual representation 14A is significantly larger than virtual representation 14B. In some embodiments, the size of virtual representations 14A and 14B may be enhanced or exaggerated in response to virtual representation 14A or 14B interacting with a particular item, such as virtual representation 14A obtaining a power-up during a current or previous round of gameplay. As described below, in other embodiments, the exaggerated size of virtual representation 14A, as well as other modifications of the virtual representations (e.g., texture, color, transparency, items worn or carried by the virtual representation), may be the result of the corresponding player interacting with an object or item external to interactive video game system 10.

[0032] It is now recognized that embodiments of an interactive video game system 10 utilizing a 2D play area 16B, such as that shown in FIGS. 2 and 4, enable certain advantages over embodiments of an interactive video game system 10 utilizing a 3D play area 16A, such as that shown in FIG. 1. For example, as discussed above, the array 36 of sensing units 38 in an interactive video game system 10 having a 2D play area 16B, such as that shown in FIG. 2, includes fewer sensing units 38 than an interactive video game system 10 having a 3D play area 16A, such as that shown in FIG. 1. That is, depth (e.g., position and movement along the z-axis 22, as shown in FIG. 1) is not tracked in an interactive video game system 10 having a 2D play area 16B, such as that shown in FIGS. 2 and 4. Additionally, because players 12A and 12B remain within their respective assigned areas 70A and 70B of the 2D play area 16B, the likelihood of occlusion is significantly reduced. For example, by keeping players within their assigned areas 70 of the 2D play area 16B, occlusion between players is expected to occur only along the x-axis 18. Thus, by using a 2D play area 16B, the embodiment of the interactive video game system 10 shown in FIG. 2 allows for the use of a smaller array 36 with fewer sensing units 38 for tracking the player 12 compared to the embodiment of the interactive video game system 10 of FIG. 1.

[0033] Accordingly, it will be appreciated that the smaller array 36 of sensing units 38 used in embodiments of interactive video game system 10 having a 2D play area 16B also generates significantly less data to process than embodiments having a 3D play area 16A. For example, in the 2D play area 16B of Figures 2 and 4, occlusion between players 12 is significantly limited and predictable, allowing a significant portion of the potential vantage viewpoints around the play area 16 to be covered while using a smaller number of sensing units 38 in the array 36. Accordingly, in embodiments of interactive video game system 10 having a 2D play area 16B, the processing circuitry 35 of the primary controller 34 may be smaller, simpler, and / or more energy efficient than the processing circuitry 35 of the primary controller 34 in embodiments of interactive video game system 10 having a 3D play area 16A.

[0034] As described above, interactive video game system 10 can generate various models of players 12. Specifically, in some embodiments, processing circuitry 35 of primary controller 34 is configured to receive partial model data (e.g., partial player models, shadow models, and / or skeletal models) from various sensing units 38 of array 36 and fuse these partial models into a complete model (e.g., a complete volumetric model, shadow model, and / or skeletal model) of each player 12. The following provides an example of processing circuitry 35 of primary controller 34 fusing the partial skeletal models received from various sensing units 38 of array 36. It can be appreciated that in some embodiments, processing circuitry 35 of primary controller 34 can fuse partial shadow model data into a shadow model and / or fuse partial volumetric model data using a similar process.

[0035] In one example, partial skeletal models are generated by each sensing unit 38 of interactive video game system 10 and then fused by processing circuitry 35 of primary controller 34. Specifically, processing circuitry 35 may perform a one-to-one mapping of corresponding bones of each player 12 in each partial skeletal model generated by different sensing units 38 positioned at different angles (e.g., on either side, perpendicular) relative to play area 16. In some embodiments, relatively small differences between the partial skeletal models generated by different sensing units 38 may be averaged during fusion by processing circuitry 35 to smooth virtual representation 14 and prevent jerky movements. Additionally, if the partial skeletal model generated by a particular sensing unit differs significantly from the partial skeletal models generated by at least two other sensing units, processing circuitry 35 of primary controller 34 may determine that the data is erroneous and therefore not include this data in skeletal model 80. For example, if a particular partial skeletal model is missing a bone that is present in other partial skeletal models, processing circuitry 35 may determine that the missing bone is likely the result of occlusion and discard all or part of the partial skeletal model accordingly.

[0036] It should be noted that accurate coordination of the components of the interactive video game system 10 is desirable to provide smooth and rapid movement of the virtual representation 14 within the virtual environment 32. Specifically, the processing circuitry 35 may take into account the time at which each partial model (e.g., partial skeletal model, volumetric model, and / or shadow model) was generated by the sensing units 38 in order to properly fuse the partial models generated by the sensing units 38. In some embodiments, the interactive video game system 10 may include a system clock 100, as shown in FIGS. 1 and 2, used to synchronize operations within the system 10. For example, the system clock 100 may be a component of the primary controller 34 or another suitable electronic device capable of generating a time signal that is broadcast over the network 48 of the interactive video game system 10. In some embodiments, various devices coupled to the network 48 may receive and use the time signal to adjust their respective clocks at specific times (e.g., at the start of game play) and subsequently include timing data based on the signals from these respective clocks when providing game play data to the primary controller 34. In other embodiments, various devices coupled to network 48 continuously receive time signals (e.g., at regular microsecond intervals) from system clock 100 throughout game play and subsequently include timing data from this time signal when providing data (e.g., volumetric scan data, partial model data) to primary controller 34. Processing circuitry 35 of primary controller 34 can also determine whether a partial model (e.g., partial volume model, shadow model, or skeletal model) generated by sensing units 38 is sufficiently fresh (e.g., current, contemporaneous with other data) to be used to generate or update a complete model, or whether the data should be discarded as obsolete. Thus, in some embodiments, system clock 100 enables processing circuitry 35 to properly blend the partial models generated by various sensing units 38 into a suitable volumetric, shadow, and / or skeletal model of player 12.

[0037] 5 is a flow diagram illustrating an embodiment of a process 110 for operating interactive video game system 10 in accordance with the present teachings. It will be appreciated that in other embodiments, some steps of the illustrated process 110 may be performed in a different order, repeated multiple times, or skipped entirely in accordance with the present disclosure. Process 110 illustrated in FIG. 5 may be performed by processing circuitry 35 of primary controller 34 alone or in combination with other suitable processing circuitry of system 10 (e.g., processing circuits 46, 52, and / or 58).

[0038] The illustrated embodiment of process 110 begins with interactive video game system 10 collecting scan data for each player (block 112). In some embodiments, such as those shown in FIGS. 1-4, sensing units 38 located around the perimeter of play area 16 may scan or image players 12. For example, in some embodiments, before game play begins, players 12 may be prompted to assume a particular pose while sensing units 38 of array 36 collect scan data (e.g., volumetric scan data and / or 2D scan data) about each player. In other embodiments, a separate system may volumetrically scan players 12 before they enter play area 16. For example, a series of waiting players may be passed through a pre-scanning system (e.g., similar to airport security scanners) that individually scans each player (e.g., while assuming a particular pose) to collect scan data for each player. In some embodiments, the pre-scanning system may be a miniature version of the 3D play area 16A as shown in FIG. 1 or the 2D play area 16B of FIG. 2, in which an array 36 including one or more sensing units 38 is positioned near each player to collect scan data. In other embodiments, the pre-scanning system may include a small number (e.g., one, two, or three) of sensing units 38 positioned around each player, with the sensing units 38 rotating around the player to collect complete scan data. It is now recognized that it may be desirable to collect the scan data shown in block 112 while the player 12 is present within the play area 16, to increase the efficiency of the interactive video game system 10 and reduce player wait time.

[0039] Interactive video game system 10 then generates a corresponding model of each player based on the collected scan data of each player (block 114). As described above, in some embodiments, processing circuitry 35 of primary controller 34 may receive partial models of each player from each sensing unit 38 of array 36 and suitably fuse these partial models to generate a suitable model of each player. For example, processing circuitry 35 of primary controller 34 may generate a player model (e.g., a volumetric player model or a 2D player model) of each player that roughly defines each player's 2D or 3D shape. Additionally or alternatively, processing circuitry 35 of primary controller 34 may also generate a shadow model of each player that roughly defines each player's textureless 3D shape. Furthermore, processing circuitry 35 may generate a skeletal model that roughly defines each player's predicted skeletal positions and locations within the play area.

[0040] Continuing with the example process 110, the interactive video game system 10 then generates a corresponding virtual representation of each player based at least in part on the collected scan data of each player and / or the generated model or models of each player (block 116). For example, in some embodiments, processing circuitry 35 of primary controller 34 may use the shadow model generated in block 114 as a basis for generating the virtual representation of the player. It can be appreciated that in some embodiments, virtual representation 14 may have a shape or contour substantially similar to the corresponding player's shadow model, such as those shown in FIGS. 3 and 4. In addition to shape, virtual representation 14 may also have other characteristics that can be modified to correspond to the characteristics of the represented player. For example, a player may be associated with various characteristics (e.g., items, status, scores, statistics) that reflect performance in other gaming systems, purchases at a gift shop, and membership in a loyalty program. Accordingly, the characteristics of the virtual representation (e.g., size, color, texture, animation, presence of virtual items) may be set according to various characteristics associated with the corresponding player and modified based on changes in the player's characteristics during gameplay. The corresponding virtual representation of the player may also be based only in part on the generated scan data and / or shadow model of the player, and thus may include enhanced and / or modified visual features compared to the player's actual appearance. For example, in one embodiment, for a seated player (e.g., in a chair or wheelchair), a virtual representation may be generated in which an upper portion of the virtual representation includes a realistic silhouette (e.g., based on a shadow model of the player) and a lower portion of the virtual representation is shown alternatively or abstractly (e.g., as a floating cloud). In another embodiment, the upper body of the virtual representation includes a realistic silhouette (e.g., based on a shadow model of the player) and the lower body of the virtual representation is shown as the lower body of a horse, resulting in a centaur-like virtual representation. In such an embodiment, the lower horse portion of the virtual representation may move like a horse in a manner directly correlated to or extensively corresponding to (e.g., synchronized with) the movement of the player's legs, as described in further detail below.

[0041] It should be noted that in some embodiments, the virtual representation 14 of player 12 may not have a substantially similar appearance or shape to the generated player model or shadow model. For example, in some embodiments, the interactive video game system 10 may include or be communicatively coupled to a pre-generated library of virtual representations based on fictional characters (e.g., avatars), and the system may select a particular virtual representation based loosely on the player's generated player model or shadow model, or may recommend a particular selectable virtual representation to the player. For example, if a game includes a large hero and a small sidekick, the interactive video game system 10 may select or recommend from the pre-generated library a larger virtual representation of the hero for an adult player and a smaller virtual representation of the sidekick for a child player.

[0042] Next, in process 110, interactive video game system 10 presents on display device 24 a corresponding virtual representation 14 of each player within virtual environment 32 (block 118). In some embodiments, the act of block 118, in addition to presenting, may also include showing other introductory presentations, such as welcome messages or orientation / control information, to players 12 within play area 16 before game play begins. Further, in some embodiments, processing circuitry 35 of primary controller 34 may provide suitable signals to set or modify parameters of the environment within play area 16. For example, these modifications may include adjusting the brightness and / or color of house lights, playing game music or game sound effects, adjusting the temperature of the play area, activating physical effects within the play area, etc.

[0043] Once game play begins, the virtual representations 14 generated in block 116 and presented in block 118 may interact with each other and / or with virtual objects in the virtual environment 32 (e.g., virtual objects 92 and 94) as described herein with respect to Figures 3 and 4. During game play, the interactive video game system 10 typically determines the in-game actions of each player 12 within the play area 16 and the corresponding in-game effects of those in-game actions (block 120). The interactive video game system 10 also typically updates the corresponding virtual representations 14 of the players 12 and / or the virtual environment 32 based on the in-game actions of the players 12 within the play area 16 and the corresponding in-game effects determined in block 120 (block 122). As indicated by arrow 124, the interactive video game system 10 may repeat the steps illustrated in blocks 122 and 122 until game play is completed, for example, by one of the players 12 winning a round of game play or the allotted game play time expiring.

[0044] Figure 6 is a flow diagram illustrating a more detailed example embodiment of a process 130 by which interactive video game system 10 performs the operations illustrated in blocks 120 and 122 of Figure 5. That is, process 130 illustrated in Figure 6 includes multiple steps for determining in-game actions of each player in the play area and the corresponding in-game effects of those in-game actions, as indicated by bracket 120, and multiple steps for updating each player's corresponding virtual representation and / or virtual environment. In some embodiments, the operations described in process 130 may be encoded as instructions in a suitable memory, such as memory circuitry 33 of primary controller 34, and executed by a suitable processor, such as processing circuitry 35 of primary controller 34 of interactive video game system 10. It should be noted that the illustrated process 130 is provided by way of example only, and that in other embodiments, some of the operations described may be performed in a different order, repeated, or skipped entirely.

[0045] 6 begins with processing circuitry 35 receiving partial models from a plurality of sensing units within the play area (block 132). As described herein with respect to FIGS. 1 and 2, interactive video game system 10 includes an array 36 of sensing units 38 disposed at different locations around play area 16, each of which is configured to generate one or more partial models (e.g., partial player models, shadow models, and / or skeletal models) representing at least a portion of player 12. As also described above, processing circuitry 35 may receive data regarding the movements of player 16 disposed within play area 16 from other devices (e.g., RF sensors 45, input devices 76). Furthermore, as also described above, these partial models may be time-stamped based on signals from clock 100 and provided to processing circuitry 35 of primary controller 34 via high-speed IP network 48.

[0046] In the illustrated embodiment of process 130, after receiving the partial models from sensing units 38, processing circuitry 35 fuses the partial models to generate an updated model (e.g., a player model, a shadow model, and / or a skeletal model) for each player based on the received partial models (block 134). For example, processing circuitry 35 may update a previously generated model, such as the initial skeletal model generated in block 114 of process 110 of FIG. 5. Also, as described above, when combining the partial models, processing circuitry 35 may filter or remove inconsistent or delayed data to increase accuracy in tracking players despite potential occlusions or network delays.

[0047] Next, in illustrated process 130, processing circuitry 35 identifies one or more in-game actions for the corresponding virtual representation 14 of each player 12 based at least in part on the updated player model generated in block 134 (block 136). For example, the in-game actions may include jumping, running, sliding, or another movement of the virtual representation 14 within the virtual environment 32. The in-game actions may also include interacting with items, such as virtual objects, within the virtual environment 32 (e.g., moving, acquiring, losing, or consuming items). The in-game actions may also include completing an objective, defeating another player, winning a round, or other similar in-game actions.

[0048] Processing circuitry 35 may then determine one or more in-game effects to be triggered in response to the identified in-game actions of each player 12 (block 138). For example, if the determined in-game action is a player movement, the in-game effect may be a change in the corresponding position of a corresponding virtual representation in the virtual environment. If the determined in-game action is a jump, the in-game effect may include moving the virtual representation along y-axis 20 as shown in FIGS. 1-4. If the determined in-game action is activating a particular power-up item, the in-game effect may include modifying a state (e.g., health state, ability state) associated with player 12. In some cases, the movement of virtual representation 14 may also be exaggerated or augmented relative to the actual movement of player 12. For example, as described above with respect to modifying the appearance of the virtual representation, the movement of the player's virtual representation may be temporarily or permanently exaggerated (e.g., to jump higher, jump farther) relative to the player's actual movement based on characteristics associated with the player, including items acquired during gameplay, items acquired during other gameplay sessions, and items purchased in a gift shop.

[0049] Next, in the illustrated process 130, processing circuitry 35 generally updates the presentation to players in play area 16 based on each player's in-game actions and corresponding in-game effects, as indicated by bracket 122. Specifically, processing circuitry 35 updates the corresponding virtual representation 14 and virtual environment 32 of each player 12 to advance gameplay (block 140) based on each player's latest model (e.g., shadow model and skeletal model) generated in block 134, the in-game actions identified in block 136, and / or the in-game effects determined in block 138. For example, in the embodiment shown in FIGS. 1 and 2 , processing circuitry 35 can provide suitable signals to output controller 56 to cause processing circuitry 58 of output controller 56 to update virtual representation 14 and virtual environment 32 presented on display device 24.

[0050] Processing circuitry 35 may also provide suitable signals to generate one or more sounds (block 142) and / or one or more physical effects (block 144) within play area 16 based at least in part on the determined in-game effect. For example, if an in-game effect is determined to be a particular virtual representation of a player plunging into a virtual swimming pool, primary controller 34 may cause output controller 56 to send a signal to speaker 62 to generate a suitable splashing sound and / or a signal to physical effects device 78 to generate a puff of mist. Sounds and / or physical effects may also be generated in response to any number of in-game effects, including, for example, gaining a power-up, losing a power-up, scoring points, or moving through a particular type of environment. As discussed above with respect to FIG. 5, process 130 of FIG. 6 may be repeated, as indicated by arrow 124, until game play is completed.

[0051] Additionally, interactive video game system 10 may enable other functions using the scanning data collected by array 36 of sensing units 38. For example, as described above, in some embodiments, processing circuitry 35 of primary controller 34 may generate a player model (e.g., a volumetric player model or a 2D player model) that includes both texture and shape for each player. At the end of game play, processing circuitry 35 of primary controller 34 may use the player's model to generate simulated images representing 2D or 3D likenesses of the players in portions of virtual environment 32, which may be provided (e.g., printed, electronically transferred) to player 12 as a souvenir of the gameplay experience. For example, the images may include a printout of a simulated image showing a volumetric model of the player crossing a finish line in a scene from virtual environment 32.

[0052] 7-13 illustrate example embodiments of an interactive video game system 10 that enable the generation of virtual representations having appearances and / or movements that are augmented with respect to the appearances and / or movements of a player. While these example embodiments depict only a single player for simplicity, it is envisioned that any suitable number of players (e.g., 12 players) may simultaneously use these interactive video game systems, as described above. Additionally, while not shown for simplicity, the example interactive video game systems illustrated in FIGS. 7-13 include any suitable features (e.g., sensors, controllers, display devices, physical effects devices, etc.) described herein to enable operation of system 10 as described above.

[0053] With this in mind, in some embodiments, virtual representations may be modified to look and / or move differently than a corresponding player. That is, in some embodiments, a virtual representation associated with a particular player may be transformed or moved in a manner that does not directly correspond to (e.g., not identical to) the player's appearance or movements. In some embodiments, the virtual representation is not limited by actual physical constraints imposed on a player's appearance or movements, and thus may be represented as associated with superhuman abilities. For example, in some embodiments, the virtual representation may include a character with greater than normal or superhuman abilities, such as a character that can jump higher or stretch farther than a real human can. In other embodiments, these superhuman abilities may include other super speed, super strength, the ability to change size (e.g., stretch and shrink), the ability to fire projectiles from various body parts (e.g., fire lasers from eyes or hands, hurl fire or ice), etc. Thus, when a player is controlling such a virtual representation, certain actual or real movements by the player trigger (e.g., are translated into) these superhuman abilities of the virtual representation. As a further example, and in some embodiments, the virtual representation may be a representation of a non-human entity. For example, in some embodiments, the virtual representation may be an animal-like representation of the player, which has capabilities (e.g., modes or styles of movement) that are different and / or augmented from those of a typical human.

[0054] In one example shown in Figure 7, during game play of interactive video game system 10, player 12 is positioned within play area 16, and a portion of virtual environment 32 is presented on display device 24. Specifically, virtual environment 32 includes a virtual representation 14 representing player 12. Thus, in the illustrated example, virtual representation 14 has an appearance that broadly resembles the appearance of player 12, based on the scanning data and various models described above. However, unlike the other examples described above, virtual representation 14 shown in Figure 7 exhibits enhanced physical movement relative to the detected movement of player 12.

[0055] In the example shown in FIG. 7 , player 12 is shown jumping only a short distance above the floor of play area 16, while virtual representation 14 is shown performing a significantly larger jump relative to the floor of virtual environment 32. Thus, virtual representation 14 exhibits enhanced (e.g., enhanced, exaggerated) jumping abilities (e.g., superhuman jumping abilities) that exceed those of a normal human. In some examples, virtual representation 14 may perform the enhanced jumping abilities after acquiring a particular item (e.g., a power-up) within virtual environment 32, and this ability may be temporary or permanent after acquiring the item. In other embodiments, the enhanced jumping ability may be a feature or aspect of a particular character (e.g., a fictional character from a video game, book, or movie) upon which virtual representation 14 is based. In such an embodiment, selecting a character associated with the enhanced jumping ability may cause virtual representation 14 to exhibit this enhanced jumping ability throughout gameplay. It should be understood that the augmented jump shown in FIG. 7 is just one example of an augmented movement, and that in other embodiments, processing circuitry 35 of primary controller 34 may identify and augment any other suitable type of player movement (e.g., running, jumping, spinning, dancing, etc.) based on the above-described scanning data and models (e.g., skeletal models) in accordance with this disclosure.

[0056] In some embodiments, virtual representation 14 may be associated with the ability to affect both the appearance and behavior of virtual representation 14 in response to specific movements of player 12. In the example of FIG. 8 , player 12 is positioned within play area 16 during gameplay of interactive video game system 10, and display device 24 is presented with a corresponding virtual representation 14 associated with a superhuman ability to change size. In the particular example shown in FIG. 8 , player 12 is crouched into a crouching pose during gameplay. This crouching pose represents a special pose or control pose that, when detected by processing circuitry 35 of primary controller 34 in the scan data and one or more models described above, triggers a specific augmentation ability of virtual representation 14 or virtual environment 32. It will be appreciated that other control poses may be used in other embodiments in accordance with the present disclosure.

[0057] In the example shown in FIG. 8 , in response to detecting player 12 in a control pose (e.g., a crouching pose), the size of the illustrated virtual representation 14 is dramatically reduced, effectively shrinking the virtual representation 14 within the virtual environment 32. In some embodiments, the virtual representation 14 may maintain a small or reduced size while player 12 remains in the crouching position. In other embodiments, once the controller 34 determines that player 12 has assumed a crouching control pose, the virtual representation 14 may again stand up without the virtual representation 14 returning to its previous or original size. In such embodiments, the virtual representation 14 may remain in a reduced state in size until the primary controller 34 determines that player 12 has assumed a second control pose (e.g., a standing pose with arms and legs spread generally in an “X” shape) that causes the virtual representation 14 to expand. In this manner, the primary controller 34 may activate one or more special abilities or superpowers temporarily or permanently associated with the virtual representation 14 upon detecting one or more control poses.

[0058] In the example shown in FIG. 8 , it can be understood that the modified appearance of the virtual representation 14 can also relate to differences in the movement and / or capabilities of the virtual representation 14 within the virtual environment 32. For example, in some situations, the miniature-sized virtual representation 14 can exhibit enhanced (e.g., enhanced, exaggerated) movement relative to the detected movement of the player 12. That is, in some situations, the miniature-sized virtual representation 14 can continue to jump as high and run as fast as the player 12 despite its small size. In other examples, the movement of the miniature-sized virtual representation 14 can be reduced or slowed down relative to the detected movement of the player 12 until the virtual representation 14 resumes life-size. In some embodiments, the miniature-sized virtual representation 14 can exhibit an enhanced effect on features within the virtual environment 32. For example, the miniature-sized virtual representation 14 can be more easily moved or affected by wind or air moving within the virtual environment, or can enter locations within the virtual environment 32 that the larger-sized virtual representation 14 cannot enter.

[0059] 9, during gameplay of interactive video game system 10, player 12 is positioned in play area 16, and a corresponding virtual representation 14 associated with superhuman stretching abilities is presented on display device 24. In the particular example shown in FIG. 9, player 12 has his arms spread or extended to his sides during gameplay. In the illustrated embodiment, the virtual representation 14 associated with superstretching abilities may be based on a character selection by player 12 at the start of gameplay or a particular item (e.g., a superstretching power-up) obtained by virtual representation 14 within virtual environment 32 during gameplay.

[0060] In the embodiment shown in FIG. 9 , in response to primary controller 34 determining that player 12 is extending his / her arms to their maximum extent, processing circuitry 35 modifies both the appearance and movement of virtual representation 14 such that virtual representation 14's arms extend in an enhanced (e.g., enhanced, exaggerated) manner. This can be understood to enable virtual representation 14 to perform specific tasks within virtual environment 32. In other situations, virtual representation 14 can extend in various ways (e.g., from the legs, torso, neck) based on other movements or poses of player 12 during gameplay. Generally, this enhanced stretching ability can enable virtual representation 14 to access elements within virtual environment 32 that would otherwise be inaccessible (e.g., items, weapons, entrances / exits, enemies, allies), providing an engaging and creative problem-solving experience for player 12. Also, while super-stretching abilities are shown in FIG. 9 , in other embodiments, other enhanced abilities, such as super-speed, super-strength, etc., can be implemented in accordance with the present disclosure.

[0061] In some embodiments, the virtual representation may not exactly replicate the player's appearance and movements, but may instead appear and move like a realistic or fictional non-human entity, such as a virtual representation of an animal. In some embodiments, a player may select a particular animal-based virtual representation at the beginning of gameplay, while in other embodiments, an animal-based virtual representation may be automatically assigned based on scan data and / or a model associated with the player. In some embodiments, once selected or assigned, the virtual representation may remain the same throughout gameplay, while in other embodiments, the virtual representation may change periodically or in response to specific player movements or achievements (e.g., different animal representations for different terrain or different levels within the virtual environment). When the virtual representation takes the form of a particular animal, it may have certain types of abilities (e.g., types of movement) that differ from those of player 12, including some abilities that would be difficult or impossible for player 12 to actually perform (e.g., trotting like a horse, hopping like a kangaroo, swimming like a fish, and flying like a bird). Thus, the appearance and movements detected by the primary controller 34 can be augmented (e.g., exaggerated, enhanced) to enable the player 12 to use poses and movements of real humans that are achievable within the play area 16 and augment this to generate the movements of the animal-like virtual representation 14.

[0062] 10 illustrates an example embodiment in which, during game play of interactive video game system 10, player 12 is positioned within play area 16 and a corresponding animal-like virtual representation 14 capable of hopping movements is presented on display device 24. In the illustrated embodiment, virtual representation 14 is a virtual representation 14 of a stag that may be selected by player 12 at the start of game play or by processing circuitry 35 of primary controller 34 based on scan data and / or a model associated with player 12. For example, in one embodiment, stag virtual representation 14 is selected for player 12 by primary controller 34 upon detecting that player 12 has a "ponytail" or "pigtails" that resemble stag antlers from a distance. In such an embodiment, virtual representation 14 may include one or more features or characteristics that generally correspond to the features of player 12, enhanced (e.g., enhanced, exaggerated) in a manner similar to caricature artwork.

[0063] 10 , primary controller 34 detects player 12 skipping across play area 16 during gameplay based on the scanning data and one or more models. Primary controller 34 translates the detected movements of player 12 into movements appropriate for virtual representation of stag 14. Specifically, primary controller 34 augments (e.g., enhances, exaggerates) the detected movements of player 12 to cause virtual representation of stag 14 to leap within virtual environment 32 to heights and / or distances beyond the detected heights and / or distances, potentially beyond heights and / or distances considered possible for player 12. Additionally, as described above, player 12 can use the augmented movement (e.g., jumping, hopping, leaping) capabilities exhibited by virtual representation 14 to accomplish specific goals within virtual environment 32.

[0064] In some embodiments, one or more real-world figures (e.g., robotic elements, animatronic devices) may be part of the interactive video game system 10. For example, in some embodiments, the interactive video game system 10 may include a robotic representation, such as a representation of a robotic stag, in addition to or instead of the virtual representation 14. Like the virtual representation 14 of the stag described above, the robotic stag is controlled by the primary controller 34 based on the detected movements of the player 12, and the controller 34 may augment (e.g., enhance, exaggerate) the detected movements of the player 12 when determining how to move the representation of the robotic stag. Also, in some embodiments, the interactive video game system 10 may include other robotic elements, such as the illustrated robotic rabbit 150 and robotic squirrel 152. In some embodiments, the movements of these additional robotic elements 150, 152 may be controlled based on the movements of other players within the play area 16. In other embodiments, these additional robotic elements 150, 152 may move in response to situations occurring within the virtual environment 32, movements of the robot or virtual representation 14, or a combination thereof, to provide an immersive experience that includes movement of a real 3D figure.

[0065] FIG. 11 illustrates another example of an animal-based virtual representation (e.g., an aquatic animal) that enables enhanced motion (e.g., swimming motion) in response to detected player motion. In the illustrated example, player 12 is undulating (e.g., walking, lunging) through play area 16. In response to detecting this motion, primary controller 34 causes dolphin virtual representation 14 to move in a corresponding undulating manner that is enhanced (e.g., exaggerated, enhanced) relative to player 12's motion. Motions of other players may also be detected and translated into motions of dolphin virtual representation 14. For example, primary controller 34 may translate a detected jumping motion of player 12 into a large jump that leaps upward from the surface of a body of water in virtual environment 32, or translate a detected swimming arm motion of player 12 into a tail slapping motion of dolphin virtual representation 14. Primary controller 34 may enhance (eg, exaggerate, intensify) the movements of virtual representation 14 of dolphin relative to the actual detected movements of player 12 for one or more of these movements.

[0066] 12 illustrates another example of an animal-based virtual representation that enables augmented flying movements in response to detected player movements. In the illustrated embodiment, player 12 is positioned within play area 16 such that primary controller 34 detects and determines the movements of player 12 during gameplay of interactive video game system 10. In the illustrated example, player 12 controls virtual representation 14 of a bat such that, when player 12 poses or moves in a particular manner, these movements are translated and augmented into bat wing movements.

[0067] 11 , in response to detecting player 12 with his arms outstretched, primary controller 34 may spread the wings of the virtual representation of the bat. Further, in response to detecting player 12 leaning left or right, primary controller 34 may tilt virtual representation of bat 14 to correspondingly steer it left or right. Also, in some embodiments, player 12 may flap his arms to cause virtual representation of bat 14 to flap its wings and swoop down by pucking his arms. Furthermore, in the illustrated embodiment, display device 24 includes multiple screens 154 (e.g., screens 154A, 154B, and 154C) that may enable virtual representation of bat 14 to fly between the screens around at least a portion of the periphery of play area 16. While examples of virtual representations 14 of a stag, a dolphin, and a bat have been described with respect to Figures 10-12, it will be appreciated that the same techniques can be applied to other animals having other abilities and / or forms of movement. For example, it is envisioned that the techniques can be applied to allow a player to pose and move in unique ways to control an animal-based virtual representation (e.g., jumping like a kangaroo, digging like a meerkat, meandering like a snake, etc.). For example, it will be appreciated that by applying the techniques to fictional entities and animals, the disclosed system can also enable the augmented movement of the virtual representation 14 to sprint like a unicorn or fly like a dragon based on the detected position and movement of the player.

[0068] FIG. 13 illustrates an embodiment of the interactive video game system 10 that includes enhanced physical effects associated with the augmented capabilities of the virtual representation 14. In the illustrated embodiment, the player 12 is positioned within the play area 16 near (below) two physical effects devices 156 and 158. In other embodiments, the physical effects devices 156 and 158 may be located above the play area 16, integrated into the floor of the play area 16, integrated into the interface panel 74 (as shown in FIG. 2), or otherwise positioned to direct physical effects toward the player within the play area 16. The physical effects device 156 is a thermal physical effects device designed to provide the player 12 with thermal effects (e.g., infrared (IR) light, a blast of cool / hot air, a blast of hot or cold mist) that correspond to events occurring within the virtual environment 32. In contrast, the physical effects device 158 is an ultrasonic haptic device that uses ultrasound to provide the player 12 with a sensation of physical contact through the air, where the physical sensation corresponds to events occurring within the virtual environment 32.

[0069] 13 , virtual representation 14 has received a fire-related power-up and is contacting barrier 160. That is, in the illustrated example, first hand 162 of virtual representation 14 is associated with the fire power-up, and a fire or sun symbol 164 is present near first hand 162 of virtual representation 14. Accordingly, thermal physical effect device 156 may be an IR source (e.g., an IR lamp) activated by primary controller 34 and directed toward first hand 166 of player 12. Additionally, second hand 168 of virtual representation 14 is shown contacting barrier 160 within virtual environment 32. Accordingly, ultrasonic haptic physical effect device 158 is activated by primary controller 34 and directed toward second hand 170 of player 12. Thus, player 12 gains a more immersive experience by feeling physical effects based on events and situations occurring in virtual environment 32. In this manner, interactive video game system 10 can provide enhanced feedback to player 12 that enhances one or more aspects of the enhanced movements and abilities to player 12's actual experience.

[0070] The technical effect of the present method includes an interactive video game system that enables multiple (e.g., two or more, four or more) players to perform actions within a physical play area (e.g., a 2D or 3D play area) to control corresponding virtual representations within a virtual environment presented on a display device near the play area. The disclosed system includes multiple sensors and suitable processing circuitry configured to collect scanning data and generate various models, such as a player model, a shadow model, and / or a skeletal model, for each player. The system generates a virtual representation of each player based at least in part on the generated player model. The interactive video game system can also set or modify characteristics, such as size, texture, and / or color, of the virtual representation based on various characteristics, such as points, purchases, power-ups, etc., associated with the player. Furthermore, the interactive video game system enables enhanced or exaggerated augmented actions (e.g., superhuman abilities, animal-like actions) relative to the player's actual detected movements within the play area. Furthermore, embodiments of the interactive video game system can include robotic and / or physical effects devices that provide feedback on these augmented actions and abilities to provide an immersive gameplay experience for the player.

[0071] While only certain features of the technology have been illustrated and described herein, many modifications and changes will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the disclosure. Moreover, the technology shown and claimed herein refers to and applies to tangible objects and specific examples of a practical nature that effectively improve the art and are therefore not abstract, intangible, or purely theoretical. Furthermore, if any claim appended at the end of this specification contains one or more elements designated as "means for [performing] ... [function]" or "step for [performing] ... [function]," such elements are to be construed pursuant to 35 U.S.C. § 112(f). Conversely, for any claim containing elements designated in any other manner, such elements are not to be construed pursuant to 35 U.S.C. § 112(f).

Claims

1. An interactive amusement system, comprising: a plurality of sensing units, each sensing unit of the plurality of sensing units comprising a sensor for scanning a person disposed in an interaction area from a plurality of different directions to collect scan data of each of the people, and configured to generate a partial model of each of the people based on the scan data by a processing circuit of each of the sensing units; a video display positioned near the interaction area and configured to present a virtual representation associated with the person; a controller communicatively coupled to the plurality of sensing units and the video display, the controller comprising: receiving a respective partial model of the person from each sensing unit of the plurality of sensing units; fusing each received partial model to generate a model of the person; Identifying a motion of the person based on the model; generating a virtual representation of the person based on the model and the identified actions; presenting on the video display the generated virtual representation of the person in a virtual environment performing augmented movements that correlate to the identified movements of the person and that are exaggerated in accordance with the model; An interactive amusement system configured as follows.

2. The interactive amusement system of claim 1 , wherein the generated virtual representation of the person is a silhouette of the person.

3. The interactive amusement system of claim 1 , wherein the model includes a shadow model, and the controller is configured to generate the virtual representation based on the shadow model.

4. The interactive amusement system of claim 1 , wherein the model includes a skeletal model of the person, and the controller is configured to identify the motion of the person based on the skeletal model of the person.

5. The interactive amusement system of claim 1 , wherein the controller is configured to generate a virtual object and present the virtual object on the video display based on the identified movement of the person.

6. The interactive amusement system of claim 5 , wherein the identified action of the person is a throwing action and the virtual object is a thrown object.

7. The interactive amusement system of claim 6 , wherein the augmented motion corresponds to an exaggerated virtual motion of the throwing motion and the thrown object.

8. The interactive amusement system of claim 1 , wherein the generated virtual representation depicts a superhuman ability caused by the identified action of the person.

9. 10. The interactive amusement system of claim 1, including a movable object model configured to generate virtual moving objects.

10. 10. The interactive amusement system of claim 9, wherein the controller is configured to present the virtual moving object to move on the video display in coordination with the movable object model and the generated virtual representation of the person based on the model.

11. 1. A method of operating an interactive amusement system, comprising: receiving, via processing circuitry of the controller, a respective partial model of the person from each of a plurality of sensing units, each sensing unit of the plurality of sensing units comprising a sensor for scanning the person from a plurality of different directions to collect respective scan data of the person, and configured to generate, by the respective processing circuitry of the sensing unit, a respective partial model of the person based on the scan data; fusing, via processing circuitry of the controller, each received partial model to generate a model of the person; identifying, via processing circuitry of the controller, a motion of the person based on the model; generating, via processing circuitry of the controller, a virtual representation of the person based on the model and the movements; presenting, via a video display viewable by the person, the generated virtual representation in a virtual environment performing virtual actions corresponding to and augmented with respect to actions of the person; A method comprising:

12. The method of claim 11 , comprising identifying a physical item held or worn by the person.

13. The method of claim 12 , wherein the virtual action is based on characteristics of the physical item.

14. The method of claim 11 , comprising generating the virtual representation of the person as a silhouette.

15. The method of claim 11 , including generating, via processing circuitry of the controller, a virtual moving object in coordination with the virtual representation of the person.

16. The method of claim 15 , wherein the virtual moving object is presented based on the virtual motion.

17. An interactive amusement system, comprising: a controller, the controller comprising: configured to receive a respective partial model of the person from each sensing unit of a plurality of sensing units, each sensing unit of the plurality of sensing units comprising a sensor for scanning the person from a plurality of different directions to collect respective scan data of the person; and configured to generate a respective partial model of the person based on the scan data by a processing circuit of each of the sensing units; The controller fusing each received partial model to generate a model of the person; Identifying a motion of the person based on the model of the person; generating a virtual representation of the person based on the model and the identified actions; configured to actuate an augmented behavior of the virtual representation based on the identified motion, the augmented behavior comprising a virtual augmentation of the identified motion; The controller configured to present, on a video display, the virtual representation within a virtual environment performing the augmented behavior; Interactive amusement system.

18. The interactive amusement system of claim 17 , wherein the controller is configured to generate a virtual moving object based on the identified motion, the augmented behavior, or both.

19. The interactive amusement system of claim 17 , wherein the controller is configured to generate the virtual representation as a silhouette of the person.

20. 18. The interactive amusement system of claim 17, wherein the controller is configured to generate a virtual moving object and present the virtual moving object on the video display based on an identified action of the person corresponding to a throwing or kicking action.

21. 1. An interactive video game system, comprising: a plurality of sensors positioned proximate a play area, each sensor of the plurality of sensors configured to scan a player located within the play area from a plurality of different directions to collect scan data of the player, and to generate, by a respective processing circuit of each sensor of the plurality of sensors, a partial model of each of the players based on the scan data; at least one display device positioned proximate the play area and configured to present a virtual representation associated with the player; a controller communicatively coupled to each sensor of the plurality of sensors and to the at least one display device; the controller comprising: receiving a respective partial model of the player within the play area from each sensor of the plurality of sensors; fusing the received partial models to generate a model of at least one of the players; identifying a movement of the player within the play area based on the generated at least one model; generating a virtual representation of the player based on the generated at least one model and the identified actions of the player; identifying one or more characteristics of said player; the one or more characteristics relate to a performance by the player in a game played via the interactive video game system, the one or more characteristics including an item obtained during the game or a number of points obtained during the game, and the controller is configured to: presenting the generated virtual representation of the player within a virtual environment on the at least one display device; wherein each movement of the presented virtual representation is an augmented movement that correlates to the identified movement of the player and is exaggerated according to the model based on the one or more characteristics of the player, and the presented virtual representation includes a virtual or augmented appearance of the player. Interactive video game system.

Images