Multi-modal sensor fusion for constructing a virtual

Abstract

A vehicle gaming system for a vehicle includes one or more computing devices configured to obtain body gesture data indicative of one or more recognized gestures of a user of a video game; obtaining speech data indicative of one or more recognized words spoken by a user in a passenger cabin of the vehicle; generating a multi-modal game input by combining the body posture data and the voice data; identifying one or more game commands of the video game being played from a game command data store based on the multi-modal game input; and sending the one or more game commands to a remote game server to obtain game codes associated with the one or more game commands to cause the video game to perform actions associated with the one or more game commands.

Description

Technical Field

[0001] The present disclosure relates to a vehicle gaming system. Background Art

[0002] Vehicles include a variety of functions and features to enhance the user experience. Vehicles are no longer just a mode of transportation, where technological advancements are focused on the driving experience. Today, more focus is placed on the overall experience and convenience of the vehicle's passengers, including entertainment through movies, music and / or video games and other infotainment methods. Summary of the invention

[0003] In one form, the present disclosure is directed to a vehicle gaming system for a vehicle. The vehicle gaming system includes one or more computing devices configured to: obtain body gesture data indicating one or more recognized gestures of a user of a video game; obtain voice data indicating one or more recognized words spoken by the user in a passenger compartment of the vehicle; generate multimodal game input by combining the body gesture data and the voice data; identify one or more game commands of a video game being played from a game command data repository based on the multimodal game input; and send the one or more game commands to a remote game server to obtain game code associated with the one or more game commands to cause the video game to perform actions associated with the one or more game commands.

[0004] In one form, the present disclosure is directed to a method for providing a video game system for a vehicle. The method includes: obtaining body gesture data indicating one or more recognized gestures of a user of a video game; obtaining voice data indicating one or more recognized words spoken by the user in a passenger compartment of the vehicle; generating a multimodal game input by combining the body gesture data and the voice data; identifying one or more game commands of a video game being played from a game command data repository based on the multimodal game input; and sending the one or more game commands to a remote game server to obtain game code associated with the one or more game commands to cause the video game to perform actions associated with the one or more game commands.

[0005] In one form, the present disclosure relates to a vehicle gaming system for a vehicle. The vehicle gaming system includes: one or more computing devices, one or more computing devices configured to operate as a vehicle gaming input system and a video game control module. The vehicle gaming input system is configured to: obtain body gesture data indicating one or more recognized gestures of a user of a video game; obtain voice data indicating one or more recognized words spoken by a user in a passenger compartment of the vehicle; combine the body gesture data and the voice data using supplemental data associated with the body gesture data and the voice data to generate a multimodal gaming input; and based on the multimodal gaming input, identify one or more game commands of a video game being played from a game command data repository. The video game control module is configured to execute one or more game codes associated with the one or more game commands to cause the video game to perform actions associated with the one or more game commands. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order that the present disclosure may be better understood, various forms of the present disclosure will now be described by way of example with reference to the accompanying drawings, in which:

[0007] Figure 1 A vehicle having a vehicle gaming system and a virtual gaming input system according to the present disclosure is shown;

[0008] Figure 2 yes Figure 1 A block diagram of a virtual game input system in ; and

[0009] Figure 3 is a flow chart of an example game routine according to the present disclosure.

[0010] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way. DETAILED DESCRIPTION

[0011] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application or use. It should be understood that corresponding reference numerals in the accompanying drawings represent similar or corresponding parts and features. The drawings are not necessarily drawn to scale, and certain features may be exaggerated or reduced to show the details of specific components. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but only as a representative basis for teaching those skilled in the art how to use the present disclosure in different ways.

[0012] Vehicle entertainment is no longer limited to music, but has also expanded to visual entertainment, such as video games. That is, a vehicle may include a vehicle gaming system that allows a user to access a library of video games. In a non-limiting example, the vehicle gaming system communicates with a game server having a library of video games as part of the vehicle. The video games may be displayed on a display device (e.g., a head-up display or liquid crystal display (LCD)) in the center console of the vehicle or on a headrest display, etc. The user is able to play the video games using a dedicated game controller (e.g., a physical controller or a software application for turning a smartphone into a game controller).

[0013] Current vehicle gaming systems require users to have at least a physical controller or even a smartphone, which prevents users from accessing other features of the smartphone while playing the game.

[0014] Instead of using a dedicated game controller, the vehicle game input system of the present disclosure provides a virtual multi-play gaming experience in which a user controls a video game through multi-modal game input. Specifically, the vehicle game input system generates multi-modal game input based on the user's body gestures and voice using existing devices and / or systems in the vehicle (including but not limited to cameras and / or microphones), thereby eliminating the need for a dedicated game controller.

[0015] refer to Figure 1 , the vehicle 100 communicates with a gaming server 102 to provide video gaming capabilities to one or more users 103 of the vehicle 100. In one form, the vehicle 100 includes, among other systems, a vehicle communication interface system (VCIS) 104, an infotainment system 106, and a vehicle gaming system 107 including a virtual gaming input system (VGIS) 108 and a video game control module (VGCM) 110. Among other components, each of the systems 104, 106, 108, 110 may include one or more computing devices (CDs) 115 having one or more computer hardware processors to perform the various functions described in the present disclosure, the one or more computer hardware processors coupled to one or more computer storage devices that store instructions executable by the processors.

[0016] The game server 102 may be an offline cloud server that is separate and distinct from the vehicle gaming system 107. In one form, the game server 102 includes a database(s) 111 configured to store a plurality of video games that may be accessed by an external system (e.g., the vehicle gaming system 107). A video game is an electronic game that typically involves interaction between a user and an output device that provides a digital representation of the electronic game. In a non-limiting example, the video game receives input associated with defined actions presented by an output device, including, but not limited to: visual actions presented on a display device graphic, audio sounds emitted by a speaker, and / or tactile feedback provided by controlled vibrations produced by an actuator.

[0017] Each video game is defined by a series of commands, and each command is associated with one or more instructions or software codes that can be executed by a computing device to present the video game via one or more output devices. In one form, the game server 102 stores the commands and associated instructions for each of the plurality of video games. That is, the game server 102 is configured to facilitate the execution of the video game by providing the commands to the vehicle gaming system 107. In another example, the vehicle gaming system 107 can download the entire game from the game server 102 and store the video game locally in the memory of the vehicle 100.

[0018] As described herein, the game server 102 is configured to communicate with the vehicle gaming system 107 via one or more communication networks 113. The network 113 may be supported by one or more wireless communication protocols / devices, including but not limited to cellular networks, Wi-Fi, Bluetooth, etc., such as dedicated short-range communications (DSRC). Therefore, the game server 102 includes hardware components and software components, including but not limited to transceivers, routers, and / or processors, to execute selected protocols and communicate via the network 113.

[0019] The game server 102 is described as being in communication with the vehicle 100, but the game server 102 may also be in communication with other gaming systems that may or may not be in the vehicle. For example, the game server 102 may be configured to communicate with a gaming system supported by a smartphone device or a dedicated console. In addition, the user 103 is able to play a selected video game with others via the game server 102.

[0020] The vehicle communication interface system 104 is configured to communicate with an external system (e.g., a game server 102) via a network 113. Therefore, the vehicle communication interface 104 includes hardware components and software components, including but not limited to a transceiver, a router, and / or a processor to execute a selected protocol. The vehicle communication interface 104 is also configured to communicate with a computing device / system within the vehicle 100 using a vehicle network (e.g., a controlled area network (CAN)), thereby allowing the computing device / system to exchange data with an external device.

[0021] The infotainment system 106 is configured to provide the user 103 with access to various auxiliary functions that provide convenience functions such as information, comfort, and entertainment. More specifically, the infotainment system 106 uses one or more interface devices to enable the user 103 to control other vehicle systems, such as a vehicle gaming system 107. In one form, the infotainment system 106 includes an interface device 112 for interacting with the user 103 and / or obtaining information about the user 103, such as an audio device (such as a speaker 112A and / or a microphone 112B), a display device 112C (such as a head-up display and / or a liquid crystal display), and / or a camera 112D.

[0022] The following are non-limiting examples of how various interface devices may be used. In some variations, the infotainment system 106 uses a speaker 112A to emit audio signals, including, but not limited to, music, information / questions from a digital assistant, and / or an audio narration. In some variations, the infotainment system 106 uses a microphone 112B to acquire an audio signal indicating a sound detected in, for example, the cabin of the vehicle 100, wherein the audio signal may be processed to detect words / phrases spoken by the user 103. In some variations, the infotainment system 106 uses a display device 112C to present graphics, including, but not limited to, a map, a series of icons, a video, and / or a picture. If multiple display devices 112C are provided in the vehicle 100, the display device 112C may be operated to present graphics different from other display devices 112C, thereby allowing a first user to view content different from that of a second user. In some variations, the infotainment system 106 uses a camera 112D to acquire (one or more) images (e.g., static or video images) of the cabin of the vehicle 100. In a non-limiting example, images from camera 112D may be processed to recognize user 103 and / or detect postures or gestures of user 103, which may also be used to control various systems of vehicle 100, such as vehicle gaming system 107. Camera 112D may be any sensor suitable for capturing images of an environment, including but not limited to: a video camera (monochrome and / or color), a two-dimensional (2D) camera, a 3D scanner that provides images as a 3D (three-dimensional) point cloud, an infrared camera, and / or a passive infrared sensor.

[0023] The vehicle 100 may be equipped with different types of interface devices 112 and / or one or more of the same interface devices 112. For example, the passenger compartment may include one or more speakers 112A, one or more microphones 112B, one or more display devices, and / or one or more cameras 112D. Although specific examples are provided, the infotainment system 106 may include other interfaces, such as an input interface for obtaining input from the user 103 (e.g., a touch screen provided with a display device 112C and / or buttons / knobs that the user 103 can operate).

[0024] In one example embodiment, the infotainment system 106 may display a menu of available auxiliary functions to the user 103 via the display device 112C. The user 103 may select an auxiliary function, such as a video game, using a touch screen, voice commands, buttons, or other suitable interface devices. Once selected, the infotainment system 106 notifies the vehicle gaming system 107 via the vehicle network and provides additional information to the user 103, such as a list of available video games based on the information from the vehicle gaming system 107. Specific examples are provided herein, but it should be understood that the infotainment system 106 may be configured to provide information to the user 103 and receive input from the user 103 in a variety of suitable ways and should not be limited to the examples provided herein.

[0025] The vehicle gaming system 107 is configured to send game commands 114 to the gaming server 102 and receive instructions or software code 116 for executing actions in the video game associated with the game commands 114. More specifically, the virtual gaming input system 108 is configured to provide the required user input / commands for operating the vehicle gaming system 107, and in particular the video game control module 110. As described herein, the virtual gaming input system 108 is configured to recognize the game commands 114 using data indicative of the behavioral input of the user 103 (e.g., the user's 103's verbal cues and physical gestures), and provide the game commands 114 to the video game control module 110 to cause the video game to execute the actions associated with the game commands 114.

[0026] The video game control module 110 is configured to execute and present video games. Specifically, the video game control module 110 presents accessible video games to the user 103 via, for example, a display device 112C using the infotainment system 106. Once a game is selected, the video game control module 110 presents the game via the display device 112C and controls the operation of the game based on input from the user 103. In some embodiments, the video game control module 110 is provided as a software application supported by the game server 102. More specifically, in some variants, the video game control module 110 sends game commands 114 to the game server 102 via the network 113, and the game server 102 in turn sends instructions (e.g., software code) 116 for performing actions at the vehicle. The video game control module 110 then executes the instructions so that the actions associated with the game commands are presented on the display device 112C.

[0027] In some variations, the video game control module 110 is configured to store user-accessible video games, and thus, the game commands 114 may be provided directly to the video game control module 110, rather than to the game server 102. In areas of high latency, it may be advantageous to store video games locally in the vehicle 100.

[0028] As described above, current vehicle gaming systems use a dedicated gaming controller in the form of user's body commands. As an alternative to a dedicated gaming controller, the present disclosure uses a virtual gaming input system 108 to provide gaming commands.

[0029] In one form, the game command 114 is selected based on a multimodal game input of the user 103's gesture and voice. Specifically, the virtual game input system 108 is configured to obtain body gesture data indicating one or more gestures of the user 103 and voice data indicating one or more recognized words spoken by the user 103. The virtual game input system 108 is configured to fuse the body gesture data and the voice data using supplementary data associated with the body gesture data and the voice data, in other words, to combine, thereby generating a multimodal game input. The supplementary data provides information related to one or more aspects of the data, including but not limited to a timestamp of the data, an identification of the original device that created the data (e.g., an identification associated with a microphone 112B that captures an audio signal and / or an identification associated with a camera 112D that captures an image), and / or a location of the original device in the vehicle. Using the multimodal game input, the virtual game input system 108 identifies one or more game commands 114 for the video game.

[0030] refer to Figure 2 In one form, the virtual game input system 108 includes a body gesture detector 202 , a speech recognition detector 204 , a multimodal input generator 206 , and a game command generator 208 having a game command data repository 210 .

[0031] The body posture detector 202 is configured to detect the body posture / posture of the user 103 using data from one or more occupant detection sensors (e.g., camera 112D). More specifically, the body posture detector 202 detects physical behavior or, more generally, one or more body postures of the occupant, including but not limited to the user's gaze direction, user gestures, head position, hand movements, and / or arm positions.

[0032] Using known body pose / gesture detection techniques provided as a pose detection software program (PDSP 212), the body pose detector 202 is configured to detect and track the poses and / or movements of the user 103 using images from the camera 112D. In a non-limiting example, reference is made to Figure 1, the body posture detector 202 is configured to identify key points 120 that can be associated with joints of the user 103, and can track the movement of the key points 120 to determine the posture and / or gesture of the user 103. For illustrative purposes, the key points 120 are provided as points connected to each other by segments 122. It should be understood that additional key points 120 can be identified and should not be limited to Figure 1 For example, additional key points may be identified in the hands of user 103. In one form, body gesture detector 202 is configured to output gesture data 216 indicative of a body gesture and supplemental data (e.g., a timestamp associated with the detected body gesture).

[0033] In some variations, the body gesture detector 202 may include a learning function to learn / compute the specific location of the user 103 in the vehicle 100 and the key points 120 of the user 103. For example, each user 103 may create a profile in the vehicle gaming system 107, and the body gesture detector 202 stores information related to the physical location of the user 103 in the vehicle 100, which may be provided by the user 103 or may be detected by the camera 112 D. The body gesture detector 202 may then learn or optimize the location of the detected key points 120 to improve the accuracy of the body gesture / gesture. A specific user identification may also be used to apply a specific user profile to the body gesture detector 202. In one example, the user 103 may be identified using a key fob, a user device (such as a mobile phone), or even data from a biometric sensor.

[0034] The speech recognition detector 204 is configured to analyze the audio signal obtained from the vehicle 100 to detect one or more recognized words spoken by the user 103. In a non-limiting example, the speech recognition detector 204 obtains the input audio signal from the microphone 112B in the vehicle 100. The microphone 112B can be part of the vehicle 100, and the audio signal is obtained via the vehicle network. In another example, a dedicated microphone can be provided for the virtual game input system 108 to provide the audio signal directly to the speech recognition detector 204.

[0035] Using known speech recognition techniques, the speech recognition detector 204 applies suitable automatic speech recognition techniques provided as a speech recognition software program (SRSP 214) to recognize words and / or phrases in the input audio signal. For example, the speech recognition detector 204 may use one or more acoustic models, dictionaries, grammars, and / or language models. The speech recognition detector 204 is configured to output speech data 218 indicating the detected recognized words or phrases. The speech recognition detector 204 may also output supplemental data related to the original input audio signal, including but not limited to a timestamp.

[0036] In some variations, the speech recognition application employed by the speech recognition detector 204 also includes a natural language processing model to analyze the detected words and / or phrases to determine the meaning expressed by these words and / or phrases. For example, the user 103 may say the term "jump over the hole" to indicate that the character they wish to control jumps, or the term "select potion" to indicate that the character they wish to control selects a specific item. This information can be used to improve the game system 102 based on the user 103 and enable the game system 102 to update the actions being performed. In a non-limiting example, the natural language processing may include one or more ontologies to link words or phrases to concepts that the user 103 may express, such as ontologies related to user interface interaction concepts or other tasks that the user 103 may request to perform.

[0037] The multimodal input generator 206 is configured to receive gesture data 216 and voice data 218 to generate multimodal game input 220. In one form, the multimodal input generator 206 fuses, or in other words, combines, gesture data 216 and voice data 218 based on supplemental data such as timestamps. Thus, the multimodal input generator 206 provides a temporal correlation between the gestures and voice of the user 103 when the user controls the video game.

[0038] The game command generator 208 uses the multimodal game input 220 to select one or more game commands 114 from the multiple commands provided by the command data repository 210. More specifically, the command data repository 210 is configured to associate multiple defined multimodal game inputs with multiple defined game commands. In one form, a single game command 114 can be selected using one or more game inputs. For example, a game command 114 to move right can be input by a right-leaning gesture, right hand movement, a verbal statement of the word "right", or a combination thereof. In another example, the duration of a gesture or voice can be used to detect the number of times the selected action is requested. That is, if the user 103 leans to the right for 2 seconds, the gesture can be equivalent to the user 103 pressing a button 2 times in one second. In yet another example, if the user 103 moves his right hand downward and then moves to the left when saying the words "down" and "left", the multimodal game input defined in the command data repository may only associate the movement of the right hand with the corresponding commands of "move down" and "move right". Thus, the multimodal game inputs 220 may include both gesture and voice data, but actual game instructions may only be associated with one input. Each multimodal game input 220 includes one or more recognized words, one or more recognized gestures, or a combination of one or more recognized words and one or more recognized gestures.

[0039] In one form, various factors may be considered when identifying game inputs for corresponding game commands. In a non-limiting example, factors may include latency of game inputs, latency / pacedness of the game, availability of standard physical controllers for the game, and / or physical posture limitations within a vehicle. With respect to latency, latency of game inputs may be associated with the pacing of the game, e.g., a fast-paced game with continuous movement (e.g., a racing game) may use game inputs with lower latency, while a game with intermittent movement (e.g., an adventure game with rest periods) may use game inputs with higher latency. Thus, for a fast-paced game like a racing game, if the detection speed of the game input is faster than the movement of the user's fingers, the game input may be associated with the movement of the user's body.

[0040] The game command generator 208 sends the identified game command 114 to the game server 102, which then sends instructions or software code 116 to perform the action associated with the game command 114. For example, with respect to the user 103 leaning to the right for 2 seconds, the command data repository 210 can associate the leaning to the right with forward movement and indicate that the duration of the leaning is equal to two steps per second. Therefore, if the user 103 keeps leaning for 2 seconds, the game command generator 208 determines that the game command 114 is to move forward 4 steps, which can be sent as four transmissions of the game command 114 associated with forward movement. As described above, once the command is received, the game server 102 sends instructions to perform the action to the vehicle gaming system 107, where the instructions can include, for example, outputting a specific sound from the speaker 112A and / or presenting one or more graphics on the display device 112C.

[0041] If the game command generator 208 fails to recognize the multimodal game input 220, the game command generator 208 is configured to notify the user 103 via one or more interface devices 112. For example, the game command generator 208 is configured to provide an audio message via the speaker 112A and / or via a text message presented on the display device 112C, where the message may be "Game command not recognized."

[0042] In another variant, the game command generator 208 is configured to associate a specific interface of a dedicated game controller with a specific behavior pattern. In a non-limiting example, the arrows of the dedicated game controller may be associated with predefined body gestures, while the control buttons may be associated with verbal terms.

[0043] In some embodiments, game commands 114 may include generic command inputs that may be used by most video games supported by game server 102 and / or may include unique commands for a particular game (e.g., a game that employs avatars may have different commands for selecting a weapon than a puzzle game).

[0044] In some embodiments, the game command generator 208 sends the identified game command 114 to the video game control module 110, which in turn sends the game command to the game server 102. In this configuration, the game command 114 is like being provided by a dedicated physical controller. In another example embodiment, the game command generator 208 is configured to send the game command 114 to the game server 102 via the vehicle communication interface system 104, and the video game control module 110 executes the received instructions / software code 116.

[0045] Certain variations are provided below that may be implemented in combination with the vehicle gaming system 107 and / or the gaming server 102 or alone.

[0046] In some variations, the vehicle gaming system 107 and the gaming server 102 are configured to support multiplayer gaming, wherein the users 103 in the vehicle 100 play a selected video game with other passengers in the vehicle 100 and / or people not in the vehicle 100. For example, for other users in the vehicle 100, the virtual game input system 108 is configured to associate each player with a selected area of ​​the vehicle 100 (e.g., each seat in the vehicle 100 can be used as an area), and use the data from the microphone 112B and the camera 112D in the selected area to determine the multimodal game input of the user 103. That is, the body gesture detector 202 can output first gesture data for the first player and second gesture data for the second player using the supplemental data that provides identification information of the camera 112D. Alternatively, if there is one camera that captures an image of both players, the body gesture detector 202 is configured to define the respective areas in the image and then process the data associated with each area to identify the players and determine the first gesture data and the second gesture data. Similarly, the voice recognition detector 204 can receive data from two independent microphones 112B associated with each player and process the data to obtain first voice data and second voice data of the first player and the second player. Alternatively, the voice recognition detector 204 is configured to receive one audio signal but is able to distinguish the voice using the voice sample of the user 103. Once acquired, the multimodal input generator 206 generates two multimodal game inputs and provides them to the game command generator 208 to obtain commands for each player.

[0047] In another variation, the body gesture detector 202 and the voice recognition detector 204 are configured to isolate input associated with the user 103 of the game from data associated with other passengers in the vehicle 100. Similar to the variation described above regarding another player, the body gesture detector 202 is configured to use data from a selected camera associated with the area where the user 103 is located and ignore data from other cameras. If there is only one camera, the body gesture detector 202 is configured to identify the user 103 in the image and generate gesture data 216 based on data associated with the area with the image of the identified user 103. Similarly, the voice recognition detector 204 can receive data only from the microphone 112B associated with the user 103 to generate voice data 218. Alternatively, the voice recognition detector 204 is configured to receive an audio signal, but is able to use, for example, voice samples of passengers in the vehicle 100, including the user 103, to distinguish the voices.

[0048] In yet another variation, the virtual game input system 108 may be provided in whole or in part in a cloud server (not shown) rather than in the vehicle 100. In this variation, data indicative of the user's 103 behavioral input data (e.g., the user's 103 voice prompts and body gestures) is sent to the remote virtual game input system 108 using the network 113, and the virtual game input system 108 sends the command 114 to the video game control module 110.

[0049] In yet another variation, the virtual game input system 108 is configured to emulate various types of physical game controllers, such as game controllers for XBOX, PLAYSTATION, etc. Therefore, behavioral inputs for physical game controllers can be associated with the same game commands.

[0050] In yet another variation, voice commands may be generic and / or optimized for a particular game or category of games. For example, if the video game is a racing type game, more natural spoken phrases such as "faster" or "slower" may be associated with commands to increase or decrease speed rather than specific numerical values ​​associated with physical gestures (e.g., finger movements). In another example, a joystick may also be optimized based on the number of directions that can be moved (e.g., 1 or 2 axes for 2 or 4 directions of movement).

[0051] In yet another variation, in addition to the images from the camera 112D, the body posture detector 202 may also receive data from other occupant detection sensors, such as pressure sensors in vehicle seats that are capable of detecting user posture based on pressure data. Figure 3 , an example game routine 300 supported by the vehicle game system 107 of the present disclosure is provided. According to the specific details provided above, in operation 302, the system 107 obtains body gesture data and voice data from vehicle sensors. The body gesture data indicates one or more gestures of the user 103, and the voice data indicates one or more recognized words or phrases spoken by the user 103 in the cabin of the vehicle 100.

[0052] In operation 304, the system 107 generates a multimodal game input. For example, the system combines the body gesture data and the voice data using supplemental data associated with the body gesture data and the voice data to generate the multimodal game input.

[0053] In operation 306, the system 107 identifies one or more game commands 114 of the video game being played from the command data repository 210 based on the multimodal game input. In operation 308, the system 107 sends the one or more game commands 114 to the game server 102 to obtain software code / instructions (i.e., game code) associated with the one or more game commands 114. In operation 310, the system 107 executes the game code to cause the video game to perform actions associated with the one or more game commands 114.

[0054] The gaming routine 300 is only one example of a control routine performed by a vehicle gaming system and may include other operations including, but not limited to, generating gesture and voice data based on input from a camera and microphone.

[0055] The description of each embodiment is for illustrative purposes and is not intended to be exhaustive or to limit the disclosed embodiments. It should be appreciated by those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments.

[0056] Unless otherwise expressly stated herein, all numerical values ​​expressing mechanical / thermal properties, composition percentages, dimensions and / or tolerances or other characteristics when describing the scope of the present disclosure should be understood to be modified by the term "about" or "approximately". This modification is for various reasons, including industrial practice, materials, manufacturing and assembly tolerances, and testing capabilities.

[0057] Various aspects of the present embodiment may be embodied as a system, method, or computer program product. Therefore, various aspects of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or a combination of hardware and software, and the embodiments may be collectively referred to herein as a "module" or "system". In addition, various aspects of the present disclosure may take the form of a computer program product included in one or more computer-readable media, which includes computer-readable program code.

[0058] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be: including but not limited to an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device, or any appropriate combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include the following: an electrical connector with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the above. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus or device.

[0059] Various aspects of the present disclosure are described according to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products of the embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer program instructions. Computer program instructions can be used for processors of general-purpose computers, special-purpose computers or other programmable data processing devices to produce machines, so that instructions executed by processors of computers or other programmable data processing devices can implement the functions / operations specified in the flowchart and / or block diagram. The processor includes but is not limited to a general-purpose processor, a special-purpose processor, a special-purpose processor or a field programmable processor.

[0060] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, segment, or code portion, which includes one or more executable instructions for implementing (one or more) specified logical functions. It should also be noted that in some alternative embodiments, the functions indicated in the box may not occur in the order shown in the figure. For example, two boxes shown in succession may actually be executed simultaneously, or the boxes may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart description, as well as the combination of boxes in the block diagram and / or flowchart description, can be implemented by a special-purpose hardware system or a combination of special-purpose hardware and computer instructions that performs the specified function or action.

[0061] The description of the present disclosure is merely exemplary in nature, and therefore, variations that do not depart from the essence of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A vehicle game system for a vehicle, the vehicle game system comprising: One or more computing devices, the one or more computing devices being configured to: acquiring body posture data indicative of one or more identified postures of a user of a video game; acquiring speech data indicative of one or more recognized words spoken by the user within a passenger compartment of the vehicle; generating a multimodal game input by combining the body gesture data and the voice data; identifying, from a game command data repository, one or more game commands for the video game being played based on the multimodal game input; as well as The one or more game commands are sent to a remote game server to obtain game code associated with the one or more game commands to cause the video game to perform actions associated with the one or more game commands.

2. The vehicle gaming system according to claim 1, wherein: The game command data store is configured to associate a plurality of defined multimodal game inputs with a plurality of game commands.

3. The vehicle gaming system according to claim 2, wherein: Each multimodal game input includes a recognized word, a recognized gesture, or a combination of a recognized word and a recognized gesture.

4. The vehicle gaming system according to claim 1, wherein: The one or more computing devices are further configured to detect the one or more identified gestures using an image capturing the portion of the passenger cabin with the user and a gesture detection software program.

5. The vehicle gaming system according to claim 1, wherein: The one or more computing devices are also configured to present an image of the video game being played on a display device of the vehicle.

6. The vehicle gaming system according to claim 1, wherein: The body gesture data and the voice data are associated based on time stamp data of each body gesture data and each voice data.

7. A method for providing a video game system for a vehicle, the method comprising: acquiring body gesture data indicative of one or more identified gestures of a user of a video game; acquiring speech data indicative of one or more recognized words spoken by a user within a passenger compartment of the vehicle; generating a multimodal game input by combining the body gesture data and the voice data; identifying, from a game command data repository, one or more game commands for the video game being played based on the multimodal game input; as well as The one or more game commands are sent to a remote game server to obtain game code associated with the one or more game commands to cause the video game to perform actions associated with the one or more game commands.

8. The method according to claim 7, wherein: The game command data store is configured to associate a plurality of defined multimodal game inputs with a plurality of game commands.

9. The method according to claim 8, wherein: Each multimodal game input includes a recognized word from the one or more recognized words, a recognized gesture, or a combination of a recognized word and a recognized gesture.

10. The method according to claim 7, further comprising: The one or more identified gestures are detected using an image capturing the portion of the passenger cabin with the user and a gesture detection software program.

11. The method according to claim 7, further comprising: The video game is presented on a display device positioned in the vehicle.

12. The method according to claim 7, wherein: The body gesture data and the voice data are combined based on time stamp data of each body gesture data and each voice data.

13. A vehicle game system for a vehicle, the vehicle game system comprising: One or more computing devices configured to: Operates as a vehicle gaming input system configured to: acquiring body gesture data indicative of one or more identified gestures of a user of a video game; acquiring speech data indicative of one or more recognized words spoken by the user within a passenger compartment of the vehicle; combining the body gesture data and the voice data using supplemental data associated with the body gesture data and the voice data to generate a multimodal game input; as well as identifying, from a game command data repository, one or more game commands for the video game being played based on the multimodal game input; as well as Operating as a video game control module, the video game control module is configured to execute one or more game codes associated with the one or more game commands to cause the video game to perform actions associated with the one or more game commands.

14. The vehicle gaming system according to claim 13, wherein: The game command data store is configured to associate a plurality of defined multimodal game inputs with a plurality of game commands.

15. The vehicle gaming system of claim 13, wherein: Each multimodal game input includes a recognized word, a recognized gesture, or a combination of a recognized word and a recognized gesture.

16. The vehicle gaming system of claim 13, wherein: The vehicle gaming input system is further configured to detect the one or more identified gestures using an image capturing the portion of the passenger cabin with the user and a gesture detection software program.

17. The vehicle gaming system of claim 13, wherein: The video game control module is also configured to present an image of the video game being played on a display device of the vehicle.

18. The vehicle gaming system of claim 13, wherein: The supplemental data includes time stamp data, and the body gesture data and the voice data are combined based on the time stamp data.

19. The vehicle gaming system of claim 13, wherein: The vehicle game input system is further configured to send the one or more game commands to a remote game server to cause the remote game server to provide the one or more game codes.

20. The vehicle gaming system of claim 13, wherein: The video game control module is further configured to send the one or more game commands to a remote game server to cause the remote game server to provide the one or more game codes.

Images