How to create a mixed reality gaming system for automobiles

Abstract

The automotive mixed reality gaming system for vehicles includes vehicle sensors that collect vehicle data related to the vehicle's movement and position along the road, detection sensors that collect environmental data related to the vehicle's external environment, and a processing unit including a processor and memory. The processing unit receives data from the vehicle sensors and detection sensors, converts the environmental data into a game-centric coordinate system stored in memory, and generates an interactive gameplay application including a mixed reality environment and virtual avatars based on the game-centric coordinate system. Furthermore, the processing unit transmits the application to a mobile device, receives user input from the mobile device, and controls the avatar's activities and gameplay settings based on the vehicle data and user input. The avatar traverses the mixed reality environment at a speed proportional to the vehicle.

Description

Technical Field

[0001] Modern entertainment systems in vehicles can enable vehicle occupants to listen to audio or watch video. Some advanced entertainment systems even include computers and / or gaming consoles. Thus, vehicle occupants with such advanced entertainment systems are provided with the opportunity to play video games while the vehicle is in motion.

Background Art

[0002] In addition, vehicles are increasingly equipped with a number of sensors. Such sensors include, for example, cameras, LIDAR sensors, ultrasonic sensors, and the like. These sensors can support driver assistance systems and / or the implementation of autonomous driving. Other applications can also benefit from the availability of signals obtained from these sensors.

Summary of the Invention

[0003] One or more embodiments of the present invention relate to an automotive augmented reality gaming system for vehicles, including a vehicle sensor that collects vehicle data related to the movement and position of the vehicle along a road, a detection sensor that collects environmental data related to the external environment of the vehicle, and a processing unit including a processor and a memory. The processing unit receives data from the vehicle sensor and the detection sensor, converts the environmental data into a game-centered coordinate system stored in the memory, and generates an interactive game play application including an augmented reality environment and a virtual avatar based on the game-centered coordinate system. Further, the processing unit transmits the application to a mobile device, receives user input from the mobile device, and controls the activities of the avatar and the settings of the game play based on the vehicle data and the user input. The avatar traverses the augmented reality environment at a speed proportional to the vehicle.

[0004] One or more embodiments of the present invention relate to a method for operating an automotive mixed reality gaming system for a vehicle. The method includes: collecting vehicle data relating to the movement and position of a vehicle along a road; collecting environmental data relating to the external environment of the vehicle; transmitting the vehicle data and environmental data to a processing unit; converting the environmental data to a game-centered coordinate system stored in memory; and generating an interactive gameplay application including a mixed reality environment and a virtual avatar based on the game-centered coordinate system. The method further includes: transmitting the gameplay to multiple mobile devices; receiving user input from multiple mobile devices; controlling the avatar's activities and adjusting one or more gameplay settings based on the vehicle data and user input.

[0005] Other aspects and advantages of the claimed subject matter will become apparent from the following description and the attached claims.

[0006] Specific embodiments of the technology disclosed herein will be described in detail with reference to the accompanying drawings. Similar elements in various drawings are indicated by similar reference numerals for consistency. The size and relative position of elements in the drawings are not necessarily to scale. For example, the shapes and angles of various elements are not necessarily to scale, and some of these elements may be arbitrarily enlarged and positioned to improve the readability of the drawings. [Brief explanation of the drawing]

[0007] [Figure 1] This specification shows a system according to one or more embodiments disclosed herein.

[0008] [Figure 2] This specification shows a system according to one or more embodiments disclosed herein.

[0009] [Figure 3]This specification shows a system according to one or more embodiments disclosed herein.

[0010] [Figure 4A] This specification shows a visualization of a process according to one or more embodiments disclosed herein. [Figure 4B] This specification shows a visualization of a process according to one or more embodiments disclosed herein.

[0011] [Figure 5] A flowchart of a method according to one or more embodiments disclosed herein is shown.

[0012] [Figure 6] This specification shows a system according to one or more embodiments disclosed herein.

[0013] [Figure 7] This specification shows a system according to one or more embodiments disclosed herein.

[0014] [Figure 8] A flowchart of a method according to one or more embodiments disclosed herein is shown.

[0015] [Figure 9] This specification shows a system according to one or more embodiments disclosed herein.

[0016] [Figure 10] This specification shows a system according to one or more embodiments disclosed herein.

[0017] [Figure 11] This specification shows a system according to one or more embodiments disclosed herein.

[0018] [Figure 12] This specification shows a system according to one or more embodiments disclosed herein.

[0019] [Figure 13] Shows a flowchart of a method according to one or more embodiments disclosed herein.

Embodiments for Carrying Out the Invention

[0020] In the following detailed description of embodiments of the present disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0021] Throughout this application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any noun in this application). The use of ordinal numbers is not intended to imply or create a particular ordering of elements, nor is it intended to limit any element to being only a single element unless explicitly disclosed such as using terms like "before", "after", "single", and other such terms. Rather, the use of ordinal numbers is for distinguishing between elements. As an example, a first element is different from a second element, the first element may include two or more elements, and may follow (or precede) the second element in the ordering of elements.

[0022] Generally, embodiments of the present invention are directed to an automotive mixed reality gaming system for vehicles, and a method of using such a mixed reality gaming system. The techniques described in this disclosure can advantageously be implemented on a mobile device of a vehicle occupant. Further, the techniques described in this disclosure can advantageously utilize, in addition to vehicle data, a plurality of video feeds captured by a plurality of detection sensors of the vehicle as a background for gameplay. Further, the techniques described in this disclosure are beneficial for incorporating environmental objects and weather conditions of the external environment of the vehicle detected by a plurality of detection sensors as interactive elements within the gameplay.

[0023] Figure 1 shows an example of a vehicle 1 according to one or more embodiments disclosed herein. The vehicle 1 may be a passenger car, a bus, or any other type of vehicle 1. As shown in Figure 1, the vehicle 1 includes a plurality of vehicle sensors 3, a plurality of detection sensors 5, and a processing unit 7. In addition, the vehicle 1 may further include an infotainment module 9. The components of the vehicle 1 are interconnected using a bus 11, which is at least one wire, wire harness, and / or connector that is responsible for transmitting data throughout the vehicle 1.

[0024] Multiple vehicle sensors 3 collect vehicle data (not shown) related to the movement and position of vehicle 1 along the current road. For example, multiple vehicle sensors 3 may measure and / or record vehicle data such as current speed, acceleration, steering angle, yaw rate, braking state, blind spot, and proximity alerts. Furthermore, multiple vehicle sensors 3 may include navigation sensors (not shown). Navigation sensors receive signals containing the global coordinates of vehicle 1. Navigation sensors may be, for example, Global Positioning System (GPS) sensors, or equivalent sensors that determine the position of vehicle 1 in relation to the external environment of vehicle 1, and trilateration, triangulation, or similar procedures may be used to determine its position. The signals may also include information such as the direction and speed of vehicle 1. The direction and speed can be derived by comparing the global coordinates with previous corresponding direction and speed to determine the length of time vehicle 1 has been moving between two points. The signals may further include information related to the coordinates themselves, for example, which country the coordinates are located in, or the designated speed limit of the road on which the coordinates are located.

[0025] In addition to the multiple vehicle sensors 3, vehicle 1 includes multiple detection sensors 5 that collect quantitative environmental data (e.g., Figure 3) related to the external environment of vehicle 1. The external environment may include elements such as roads, other traffic including additional vehicles (e.g., Figure 4A), pedestrians, and buildings. Therefore, vehicle 1 may move within the external environment so that vehicle 1 traverses the external environment.

[0026] Examples of detection sensors 5 include a two-dimensional camera, a three-dimensional camera or stereo camera, a radar unit, a LiDAR unit, an ultrasonic sensor, or an equivalent sensor or component that perceives the environment of vehicle 1. Multiple detection sensors 5 may all be of the same type, or they may be a combination of different types of detection sensors 5.

[0027] In one or more embodiments, the plurality of detection sensors 5 include a plurality of visual sensors (e.g., Figure 2). Each visual sensor may capture a video feed (not shown) related to the field of view of the external environment of the vehicle 1 from a forward, rear, and / or lateral position outside or inside the vehicle 1. Each video feed may be formed by a series of image frames (not shown). The series of image frames may be processed and then displayed on a plurality of mobile devices 13 and / or the infotainment module 9 of the vehicle 1. In one or more embodiments, the series of image frames are expanded before being displayed on the plurality of mobile devices 13 and / or the infotainment module 9.

[0028] In one or more embodiments, as shown in Figure 1, the server 15 may include a processing unit 7 that wirelessly communicates with a plurality of mobile devices 13 and / or the electronic control unit (ECU) 17 of the vehicle 1. The processing unit 7 includes one or more processors (e.g., Figure 11) and memory (e.g., Figure 11). The plurality of mobile devices 13 may be personal devices of the occupants of the vehicle 1 that have been brought into the vehicle 1 by the occupants. The plurality of mobile devices 13 may be embodied as smartphones, tablets, and portable game consoles. The occupants of the vehicle 1 can use the plurality of mobile devices 13 to play interactive gaming applications with gameplay that includes a mixed reality environment (e.g., Figure 4B), the mixed reality environment utilizing vehicle data and environmental data collected by a plurality of vehicle sensors 3 and a plurality of detection sensors 5, respectively.

[0029] Multiple mobile devices 13 transmit user input to a processing unit 7 to control activities within the gameplay and / or adjust one or more gameplay settings (not shown) (e.g., Figure 3). In one or more embodiments, the multiple mobile devices 13 are connected to the vehicle 1 by wired or wireless technology (e.g., WiFi, Bluetooth, etc.). Alternatively, both the vehicle 1 and the multiple mobile devices 13 may be connected to a cloud-based server 15 of the processing unit 7 via wireless technology (e.g., cellular, 4G, 5G, satellite, etc.), thereby avoiding the need for the multiple mobile devices 13 to be directly connected to the vehicle 1. In one or more embodiments, one of the multiple mobile devices 13 may be located outside the vehicle 1 and communicate with the server 15.

[0030] The transfer of a series of image frames (for example, from multiple detection sensors 5 to multiple mobile devices 13) can be performed via the server 15, as shown in Figure 1. That is, augmentation and / or other operations can be performed on the server 15. The augmentation process of a series of image frames is described in further detail in Figures 3 and 5.

[0031] In one or more embodiments, the infotainment module 9 of the vehicle 1 may be used to display gameplay or gameplay features to the occupants of the vehicle 1. As shown in Figure 2, the infotainment module 9 may include a display system 19 and an interface 21. The display system 19 may receive gameplay or gameplay-related information data wirelessly from the processing unit 7 or via the bus 11. Furthermore, the display system 19 displays the received gameplay or gameplay-related information data to the occupants of the vehicle 1. The display system 19 may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), or an equivalent display.

[0032] The interface 21 of the infotainment module 9 is used to receive user interaction with the infotainment module 9. The interface 21 may include a dial, button, or knob. In one or more embodiments, the interface 21 may be a touchscreen of the display system 19. Furthermore, in one or more embodiments, an interface device (e.g., a wired and wireless game controller, a mobile device 13, etc.) may be connected to the interface 21 of the infotainment module 9 via a wired or wireless connection to transmit user input to the infotainment module 9. User input captured via the interface 21 of the infotainment module 9 is transmitted to the processing unit 7. The infotainment module 9 may further allow the configuration of one or more gameplay settings. In addition, the infotainment module 9 may be used to control in-game activities on behalf of the mobile device 13.

[0033] In one or more embodiments, the processing unit 7 and / or the infotainment module 9 may transmit gameplay to other mobile devices 13 of the vehicle occupants 1, which may be in the form of augmented reality glasses or a virtual reality headset. In such cases, the infotainment module 9 may be used to configure these forms of mobile devices 13 before the gameplay is transmitted to the mobile devices 13 or before the gameplay begins.

[0034] The gameplay is described in further detail in Figures 3 and 5, which, respectively, illustrate systems and methods for mixed reality in automotive applications according to one or more embodiments.

[0035] The ECU 17 of vehicle 1 may be formed from one or more processors, microprocessors, integrated circuits (ICs), or equivalent computing configurations. Therefore, vehicle data and environmental data recorded by multiple vehicle sensors 3 and multiple detection sensors 5 may be transmitted to the ECU 17 (i.e., via the bus 11 or wirelessly) and processed by the ECU 17 before being transmitted to the server 15.

[0036] In one or more embodiments, the server 15 may include a cloud-based instance of the Developer Kit (not shown). In addition, the ECU 17 or a plurality of mobile devices 13 may include an in-vehicle instance of the Developer Kit (not shown). The Developer Kit may include tools for facilitating the development of mixed reality applications and for processing a series of image frames, other environmental data, and / or vehicle data acquired in raw or pre-processed form. In one or more embodiments, the Developer Kit provides creation tools for creating virtual objects (e.g., Figure 6) and / or physics engines. The Developer Kit may further provide a developer interface (not shown) for developers and / or users to access a series of image frames and / or other environmental and vehicle data for further use (e.g., by a game engine) to create a virtual scene of the external environment of vehicle 1 within a mixed reality environment (e.g., Figure 4B). The developer interface may be provided in the form of one or more application programming interfaces (APIs) (not shown). The APIs may exist to provide access to vehicle and environmental data (e.g., a series of image frames, LIDAR data, etc.) acquired from a plurality of vehicle sensors 3 and a plurality of detection sensors 5. The API may also exist for the 3D development space and for rendering 2.5D output, as further explained with reference to Figures 6 and 7. The API may provide access to any of the functions described below.

[0037] The developer interface can be standardized for direct access to vehicle and environmental data for a wide variety of applications that use vehicle and environmental data. As will be discussed later in Figure 5, the developer interface may provide access not only to vehicle and environmental data acquired from multiple vehicle sensors 3 and multiple detection sensors 5, but also to data resulting from machine learning-based processing of vehicle and environmental data. The developer kit may enable third parties (e.g., application developers) to provide content to run on the system, as will be further explained with reference to Figure 3.

[0038] A cloud-based interface may be a component of the development kit accessible to application developers for application testing and deployment. An in-vehicle instance is a component of the development kit that runs via the ECU 17 or locally on multiple mobile devices 13. The in-vehicle instance may collect user input commands, perform actions related to the visualization of content displayed to the user, and / or run an application.

[0039] Alternatively, in one or more embodiments, the ECU 17 of the vehicle 1 may include a processing unit 7, as shown in Figure 2. Specifically, the ECU 17 may include an in-vehicle computer (not shown) of the development kit, and the multiple mobile devices 13 may include an end-user interface (not shown) of the development kit. The in-vehicle computer and end-user interface, although local implementations without cloud processing, may be functionally equivalent to a cloud-based interface and in-vehicle instance, respectively. Periodic communication with other remote computer devices (e.g., cloud-based components) may be limited to, for example, application downloads and application maintenance.

[0040] Figure 3 is a block diagram of a system 23 for mixed reality in an automotive application, according to one or more embodiments. In one or more embodiments, the system 23 is associated with a vehicle 1. For example, one or more elements of the system 23 may be components of the vehicle 1, and the system 23 may be used by a user of the vehicle 1 (e.g., occupants and / or driver). Specific scenarios are described below.

[0041] The system 23 includes a plurality of detection sensors 5, a plurality of vehicle sensors 3, a processing unit 7, and a plurality of mobile devices 13. In one or more embodiments, the processing unit 7 includes an environment interpretation engine 25, a content augmentation engine 27, and a rendering engine 29. Alternatively, in one or more embodiments, the plurality of mobile devices 13 include the content augmentation engine 27 and / or the rendering engine 29. The environment interpretation engine 25, the content augmentation engine 27, and the rendering engine 29 may be elements of a machine learning model 31. Each of these components will be described later.

[0042] Multiple detection sensors 5 are used to capture environmental data 33 related to the external environment of the vehicle 1. In particular, multiple visual sensors 35 among the multiple detection sensors 5 may capture multiple video feeds, each capturing a portion of the external environment. Each video feed may consist of a series of image frames of the external environment. The image frames may be repeatedly captured at a fixed or adjustable frame rate. The multiple visual sensors 35 can be of any type and may have any field of view, resolution, orientation, etc. In one embodiment, a fisheye camera is used, but other sensors such as an infrared camera, pinhole camera, LIDAR and radar sensors, or equivalent sensors may be used. The environmental data 33 (e.g., image frames) from the multiple visual sensors 35 can be provided in any format (e.g., representing RGB pixel values) and may be received by the vehicle 1's ECU 17. The ECU 17 may include components for hardware-accelerated video processing, including machine learning-based video processing. The ECU 17 may further execute various components of the system 23 (e.g., an environment interpretation engine 25, a content augmentation engine 27, and / or a rendering engine 29).

[0043] In addition, vehicle data recorded by multiple vehicle sensors 3 can be provided in any format and may depend on the type of sensor. For example, a steering angle sensor may provide a value that reflects the measured steering angle.

[0044] In one or more embodiments, the environmental data 33 and / or vehicle data may be preprocessed as described further below with reference to the flowchart.

[0045] In one or more embodiments, the environment interpretation engine 25 receives environment data 33 (e.g., a series of image frames) and identifies environment objects 37 within each image frame. For example, roads, pedestrians, additional vehicles, buildings, the sky, etc., may be identified from the video feed of the vision sensor 35. The environment interpretation engine 25 may perform image processing methods to identify the environment objects 37. Any type of image processing may be used, including the use of machine learning-based algorithms.

[0046] Machine learning, in a broad sense, is the extraction of patterns and insights from data. The terms “artificial intelligence,” “machine learning,” “deep learning,” and “pattern recognition” are often intertwined, interchangeable, and used synonymously with “machine learning.” For consistency, this specification will adopt the term “machine learning.” However, those skilled in the art will recognize that the concepts and methods detailed below are not limited by this nomenclature.

[0047] The types of machine learning models 31 used herein for video processing and image frame processing may include, but are not limited to, generalized linear models, Bayesian regression, random forests, and deep learning models such as neural networks, convolutional neural networks, and recurrent neural networks. Machine learning models 31 are typically associated with additional “hyperparameters” that further describe the model, regardless of whether they are considered deep learning models. For example, hyperparameters that provide further details about a neural network may include, but are not limited to, the number of layers in the neural network, the choice of activation function, the inclusion of batch normalization layers, and the regularization strength. It should be noted that in the context of machine learning, regularization of a machine learning model 31 refers to a penalty applied to the loss function of the machine learning model 31, which may be input to the machine learning model 31 using, for example, a backpropagation process. Generally, the selection of hyperparameters surrounding a machine learning model 31 is called the selection of the model “architecture.” Once the type of machine learning model and hyperparameters are selected, the machine learning model 31 is trained to perform the task. As described herein, training the machine learning model 31 includes supplying the machine learning model 31 with various “test” data image sets and providing the machine learning model 31 with affirmation based on whether the model correctly identified objects in the test data image sets. According to one or more embodiments, the type of machine learning model and associated architecture are selected by the manufacturer of vehicle 1, the machine learning model 31 is trained to perform video processing to determine environmental image data, the performance of the machine learning model 31 is evaluated, and the machine learning model 31 is used in the production environment (also known as the deployment of the machine learning model 31).

[0048] The operations performed by the environment interpretation engine 25 are described below with reference to the flowchart. The environment interpretation engine 25 may be run by a machine learning model 31 within the server 15 or the ECU 17 of the vehicle 1.

[0049] In one or more embodiments, the content augmentation engine 27 receives identified environment objects 37 and determines augmented content 39 based on the environment objects 37.

[0050] In one or more embodiments, the extended content 39 includes any modification of an image frame for the next display to the user. For example, the extended content 39 may include blurring, marking, highlighting, distorting, removing, moving, recoloring, animating, or any other modification of one or more identified environment objects 37. The extended content 39 may further include the addition of any number of objects. The added objects may be static or dynamic objects. Static objects may be indicators (e.g., arrows pointing to another object, icons representing an object), while dynamic objects may be objects whose properties (e.g., shape, size, color, location, etc.) change over time (e.g., frame by frame). Such dynamic objects may include, for example, an animated character or avatar (e.g., Figure 4B).

[0051] In one or more embodiments, the augmented content 39 may be controllable by a user, directly or indirectly. The mobile device 13 may enable the user to provide input device commands as user input 41. For example, the user may provide steering commands via the mobile device 13 to control the position, orientation, movement, etc., of an object that is an element of the augmented content 39. Control of the augmented content 39 may be context-specific and may enable interaction within the augmented content 39, etc. The mobile device 13 enabling control over the augmented content 39 may be a smartphone, tablet, portable game console, or similar device that enables the user to provide input. The mobile device 13 may be communicably connected to one or more computer devices in the vehicle 1 (e.g., ECU 17) or directly to the server 15. A more detailed description of different types of augmented content 39 and possible interactions with the augmented content 39 by one or more users is provided below in the description of various possible applications.

[0052] In one or more embodiments, the rendering engine 29 receives environment data 33 or a series of image frames and augmented content 39 and generates a series of augmented image frames 43. In preparation for displaying the augmented image frames 43, the rendering engine 29 may perform occlusion detection to render only the pixels that should be visible. For example, if augmented content 39 is present, the rendering engine 29 may generate an object mask, which is then used to identify occlusion and render only the desired pixels of the environment data 33 or image frames. In one or more embodiments, machine learning data is used to generate the object mask, which is then used to identify occlusion and render only the desired pixels. Rendering engines 29 such as Unity or Unreal may be used.

[0053] A mobile device 13 is used to display a series of extended image frames 43 to the user. In one or more embodiments, an infotainment module 9 may be used, together with or instead of, multiple mobile devices 13, to display a series of extended image frames 43 to the user.

[0054] System 23 enables various implementations of mixed reality experiences. In one or more embodiments, 3D gameplay can be established based on environmental data 33 captured by a plurality of detection sensors 5 related to the external environment of Vehicle 1. In this case, the mixed reality environment of the gameplay is generated in real time based on the actual external environment surrounding Vehicle 1. In one specific example, the forward-facing visual sensor 35 of Vehicle 1 may be used to capture a video feed (i.e., a series of image frames). The video feed can then be displayed to the user with various enhancements. The user can control an avatar in this environment by user input 41 via a mobile device 13. The gameplay may include obstacles to be avoided (e.g., specific environmental objects 37 or virtual objects added as enhancement content 39). The gameplay may also include targets (e.g., virtual objects added as enhancement content 39) that reward or punish the user of System 23 based on the identification information of the objects.

[0055] The gameplay is a mixed reality application, but a corresponding virtual reality implementation may be provided by completely replacing all identified environment objects 37 with augmented content 39. For example, identified roads may be rendered as artificial roads, and identified additional vehicles may be rendered as additional vehicles of different forms, and so on. In this configuration, the video feed itself may be excluded from a series of augmented image frames 43 provided to the user as output. The described gameplay may operate in real time (based on image frames processed in real time) or non-real time (depending on previously recorded image frames).

[0056] The system 23 according to embodiments of this disclosure may support multiple different gameplays that differ in purpose, style, level of mixed reality, etc. A user may select a desired gameplay from a set of available gameplay applications stored in the memory of the processing unit 7, or the gameplay may be specified by a third party. For example, a parent may select a particular gameplay application for their child that has age-appropriate expansion content 39 embedded in it.

[0057] Figures 1 to 3 show the configuration of the components, but other configurations may be used without departing from the scope of this disclosure. For example, various components may be combined to create a single component. Furthermore, a function performed by a single component may be performed by two or more components. Various operations performed by system 23 may be performed on a computer system such as the ECU 17 of vehicle 1. At least some of these operations (e.g., environment interpretation, rendering, etc.) may benefit from the availability of a graphics processing unit (GPU) (not shown) and may be performed by it. Thus, the ECU 17 or another computer system may be equipped with a GPU. Furthermore, although not expressly shown, operations performed by the various components described with reference to Figures 1 and 2 may be performed locally or remotely (e.g., on server 15).

[0058] Figures 4A and 4B show visualizations of the process for generating a mixed reality environment 45 for an automotive application, according to one or more embodiments. Specifically, in Figure 4A, vehicle 1 is traveling on a road 46. In this non-limiting example, vehicle 1 is traveling behind an additional vehicle 48 on the road 46, and several environmental objects 37 (i.e., road signs and trees) are positioned along the outer edge of the road 46. While vehicle 1 is traveling along the road 46, multiple detection sensors 5 capture environmental data 33, and multiple vehicle sensors 3 record vehicle data. Simultaneously, while capturing the environmental data 33, the environmental data 33 is processed by a processing unit 7 to identify the environmental objects 37, as described above in the description of Figures 1 to 3. Thus, the processing unit 7 identifies the road signs and trees captured by the multiple detection sensors 5 as environmental objects 37.

[0059] Figure 4B shows an exemplary mixed reality environment 45 related to environmental data 33 recorded by vehicle 1 in Figure 4A. In the mixed reality environment 45, elements and images may be from live or recorded video feeds, some of the elements being virtual and superimposed on the video feed. In Figure 4B, the mixed reality environment 45 is an augmented image frame 43 that utilizes image frames captured by the visual sensor 35, one of the vehicle 1's multiple detection sensors 5. The augmented image frame 43 includes a visualization of the vehicle 1's external environment from the viewpoint of the visual sensor 35. That is, the mixed reality environment 45 includes the road 46, additional vehicles 48, road signs, and trees captured by the multiple detection sensors 5 in Figure 4A.

[0060] In addition, the mixed reality environment 45 in Figure 4B includes virtual objects. Here, the virtual object is an avatar 50 represented as a car. The avatar 50 is a virtual object and element of gameplay that can be controlled by user input 41 and vehicle data. Furthermore, the avatar 50 can be represented as any virtual object (e.g., a car, an airplane, a human, an animal, etc.).

[0061] In one or more embodiments, environmental data 33 captured by multiple detection sensors 5 may be processed using the techniques described above in Figure 3 to provide target identification and tracking (e.g., additional vehicles 48 and other environmental objects 37), lane identification and tracking, depth maps, semantic segmentation, weather conditions, sun position, ambient brightness levels, and other information that may be relevant to gameplay.

[0062] In one or more embodiments, the processing unit 7 may detect the position of the sun relative to the vehicle 1 from the environmental data 33 at the time the video feed was captured. For this purpose, a virtual object representing a virtual light source may be included in the mixed reality environment 45 of the gameplay. The virtual light source may be positioned within the mixed reality environment 45 relative to the position of the sun in the external environment captured by the video feed. As a result, the virtual object and / or environmental object 37 may include lighting, reflections, and virtual shadows to match the lighting in the external environment.

[0063] Furthermore, in one or more embodiments, the processing unit 7 may receive or determine the weather conditions of the external environment of the vehicle 1 from environmental data 33 captured by a plurality of detection sensors 5. For example, the processing unit 7 may determine that a form of precipitation is present during the processing of a video feed. In addition, the plurality of detection sensors 5 may include an external thermometer or solar radiation sensor 52 (e.g., Figure 2) to obtain the weather conditions of the external environment. The solar radiation sensor 52 may be located inside the vehicle 1 and detect the amount of light and / or ambient brightness level received inside the vehicle 1. This information may then be used to change the brightness level of the mixed reality environment 45 for gameplay, or the brightness level of the display screen of the mobile device 13 (e.g., Figure 9). In one or more embodiments, the processing unit 7 may adjust the mixed reality environment 45 based on weather conditions received from an external server or user input 41.

[0064] Figure 5 shows a flowchart 500 of a method for generating a mixed reality environment 45 for an automotive application, according to one or more embodiments. The execution of one or more steps in Figure 5 may involve one or more components of the system 23 described in Figures 1 to 4. Although the various steps in Figure 5 are presented and described sequentially, those skilled in the art will understand that some or all of the steps may be performed in a different order, combined or omitted, and some or all of the steps may be performed in parallel. Furthermore, these steps may be performed actively or passively. The actions of performing the described steps may be represented by electronically readable instructions, such as software code or firmware code, which may be stored in one or more non-temporary media.

[0065] In step 510, environmental data 33 and vehicle data are acquired. In one or more embodiments, the environmental data 33 includes a series of image frames acquired by the plurality of detection sensors 5 as described above. In one or more embodiments, the plurality of detection sensors 5 include a fisheye camera. Step 510 may include preprocessing specific to one or more of the plurality of detection sensors 5. For example, a series of image frames acquired from the visual sensor 35 may be preprocessed to eliminate distortions such as distortion typical of a fisheye camera. That is, each image frame may be cropped to obtain a series of rectangular image frames of content of interest. Any other preprocessing such as brightness adjustment, contrast correction, compression, and resizing may be performed.

[0066] In one or more embodiments, a series of image frames may be projected in a format determined by the user or application (e.g., standard projection, cylindrical projection, etc.). In addition, image frames from various visual sensors 35 may be merged to project a larger view of the external environment of the vehicle 1. Thus, the merged image frames may be projected so that the mixed reality environment 45 is viewed from several different angles or viewpoints, as further illustrated in Figure 10.

[0067] In step 520, the environment object 37 is identified within a series of image frames. The series of image frames may be analyzed, for example, to detect motion. In one or more embodiments, AI-based image processing is used to identify the environment object 37. For example, deep learning-based image processing such as OmniDet, Single-Shot Deep MANTA, Single Shot Detection (SSD), Region-Based Convolutional Neural Network (R-CNN), You Only Look Once (YOLO) Single Shot Detection, Fast R-CNN, and Histogram of Oriented Gradients (HOG) can be used.

[0068] The output obtained as a result of deep learning-based image processing may include depth estimation utilizing environmental data 33 captured by multiple detection sensors 5, such as stereo or monocular cameras, radar units, LiDAR units, ultrasonic sensors, etc. Furthermore, the output obtained as a result of deep learning-based image processing may include semantic segmentation, visual odometry, motion segmentation, and / or object detection of environmental objects 37. For example, machine learning data from image processing may be used to mark different segments in an image. This information may then be used to generate navigation paths for characters in the scene. The output may be stored and / or transferred as descriptions of identified environmental objects 37. The descriptions of identified environmental objects 37 may be linked to the frame numbers of the corresponding image frames for synchronization purposes. Additional detections may be performed (e.g., detection of lens smudges). Examples of environmental objects 37 that can be identified include, but are not limited to, roads 46, lane markings, curbs, pedestrians, additional vehicles 48, occupants, plants, and traffic signs.

[0069] In one or more embodiments, environmental data 33 from multiple detection sensors 5 may be used to improve the accuracy of detection, depth estimation, and / or semantic segmentation, while vehicle data from multiple vehicle sensors 3 may be used to determine an accurate estimate of the current position and / or orientation of the vehicle 1.

[0070] In step 530, post-processing is performed on the identified environment objects 37. Post-processing may be performed to obtain data related to the environment objects 37 in a format suitable for input to the rendering engine 29 (for example, as described in step 550). The rendering engine 29 may assume that the description of the environment objects 37 is the boundary of these environment objects 37. In contrast, the output of the operation in step 520 may be pixel data. In step 530, data manipulation necessary to obtain the boundary is performed. Post-processing may be performed on any of the identified environment objects 37.

[0071] Upon completion of steps 520 and 530, a scene understanding is obtained in which the environment objects 37 are identified based on the performed object detection and classification, and a depth map for the environment objects 37 is available, ensuring that their position, orientation, movement, etc., are known. This includes the identification of planes (e.g., horizontal planes set based on the surface of the road 46, vertical planes set based on detected walls or other structures, etc.). The resulting scene understanding can define possible movements in the mixed reality environment 45, which will be rendered as described below.

[0072] In step 540, the augmented content 39 is determined. The augmented content 39 added to the rendered mixed reality environment 45 depends on the intended gameplay of the mixed reality environment 45. The augmented content 39 includes added objects, modifications to content within image frames by masking, etc. Added objects include, for example, one or more avatars 50, objects, symbols, labels, features, animated characters, text, etc. Each object may be static or dynamic. Dynamic objects may be user-controlled or (for example, in the case of computer-controlled characters (enemies, etc.) moving within the mixed reality environment 45) may be controlled by the content augmentation engine 27. Furthermore, objects may change their behavior based on context, contact, etc. Masks for modifying content within image frames include opaque or partially transparent overlays (e.g., color filters, blur filters, etc.) which may be static or dynamic (e.g., changing properties such as color, contrast, blinking on / off). In one or more embodiments, the augmented content 39 is linked to the frame number of the corresponding image frame for synchronization purposes.

[0073] Objects may be positioned considering previously identified planes. For example, a horizontal plane may function as a drivable area for an avatar 50 representing a car, while a vertical plane may function as a boundary. Character scaling and path planning may be further performed to generate a navigation path for the character / avatar 50. For example, A * Alternatively, any other route planning algorithm may be used.

[0074] In step 550, a series of extended image frames 43 are rendered based on a series of image frames and extended content 39. In one or more embodiments, rendering may be performed based on inputs including each image frame itself, identified environment objects 37, and extended content 39. The description of the identified environment objects 37 may include geometric shapes. The geometric shapes may be in the form of boundary pixels for each previously segmented object (e.g., ground, pedestrians, additional vehicles 48, etc.). Depth information may be included in the description. Similarly, the description of the extended content 39 includes geometric shapes of the extended content 39. Further inputs that may affect rendering may be received from the user. For example, the user may change the rendered view by control commands, steering control movements, etc.

[0075] Rendering takes these inputs and generates pixel position points that can then be used for mapping on the displayed 2D scene. In one or more embodiments, rendering determines occlusion by taking into account the geometric shapes of the objects (i.e., environment objects 37 and extended content 39). Based on the detected occlusion, an invisible mask is determined based on the contours of the occluded geometric shapes, and rendering is performed only on the parts that are not occluded based on the mask.

[0076] In one or more embodiments, rendering ensures synchronization between image frames, descriptions of identified environment objects 37, and extended content 39. Synchronization may be performed based on frame number.

[0077] In step 560, the series of extended image frames 43 are displayed on the mobile device 13 and / or the infotainment module 9 of the vehicle 1. If rendering is performed for multiple users, the extended image frames 43 may be displayed on multiple mobile devices 13. Furthermore, steps 510-560 may be performed in a loop, and the loop may be based on a fixed frame rate. This frame rate may correspond to or differ from the frame rate of the detection sensor 5.

[0078] Figures 6 and 7 show a game-centered coordinate system 54 used to generate a mixed reality environment 45 for gameplay according to one or more embodiments. The game-centered coordinate system 54 can be generated as part of the rendering of the 2.5D output during step 550 of the flowchart 500 in Figure 5. Specifically, environmental data 33 collected by multiple sensing sensors 5 and processed by the techniques described above in Figure 3 is transformed into a mixed reality environment 45 based on the game-centered coordinate system 54 of the system 23 for gameplay. In Figure 6, the game-centered coordinate system 54 includes a virtual camera 56 and a viewing frustum 58 defined by multiple projection planes. In addition, the game-centered coordinate system 54 includes the shapes, sizes, and positions of environmental objects 37 and virtual objects in 3D space.

[0079] The frustum of the game-centered coordinate system 54 is defined by the intersection of the far plane 60 and the ground at the first end, and the intersection of the ground and the near plane 62 at the second opposite end. Any virtual objects contained within these boundaries are drawn as overlays on the projected image frame. Any virtual objects outside these boundaries will not be included in the mixed reality environment 45 of the gameplay.

[0080] The image frame is projected onto the far-field plane 60 of the game-centered coordinate system 54. In this way, the far-field plane 60 of the game-centered coordinate system 54 embodies the frustum 58 of the viewing cone of the virtual camera 56 in the scene. Based on the parameters of the image frame (e.g., horizontal and vertical fields of view), the virtual camera 56 is positioned in the scene at a virtual distance from the far-field plane 60. Thus, the virtual camera 56 can capture a desired view of the image frame. That is, the virtual camera 56 can view the image frame projected onto the far-field plane 60 in a similar manner to the view of the external environment from a designated detection sensor 5.

[0081] The ground in the game-centered coordinate system 54 is also known as the “game board” 64 of the mixed reality environment 45 for gameplay. The first edge of the ground substantially touches the far plane 60 at the horizon 66 on the image frame. The horizon 66 on the image frame is based on the position and lens parameters of the detection sensor 5 associated with the image frame. The ground is perpendicular to the far plane 60, but may be tilted up or down based on the slope of the road 46 on which the vehicle 1 is located. The second edge of the ground opposite the first edge is positioned at a distance below the virtual camera 56, based on the height of the visual sensor 35 associated with the image frame, the tilt of the camera, etc. As a result, the ground, as seen from the virtual camera 56, coincides with the viewpoint captured by the detection sensor 5 associated with the image frame. In this way, virtual objects placed along the game board 64 of the mixed reality environment 45 for gameplay will appear as if they are on the ground of the external environment when projected in front of the image frame on the far plane 60. In addition, each virtual object added along the game board 64 may include a shadow. The shadow cast on the ground will generally appear as if it is correctly cast on the ground when merged with the image frame through the viewpoint of virtual camera 56.

[0082] The near plane 62 of the game-centered coordinate system 54 represents the position where the visual sensor 35 associated with the image frame is positioned along the length of the vehicle 1. Any virtual object positioned outside the frustum 58 behind the near plane 62 would not be visible to the visual sensor 35 that generated the image frame projected onto the far plane 60, and therefore would be hidden from the field of view of the virtual camera 56.

[0083] The scaling of the 3D game board 64 and the mixed reality environment 45 also depends on the visual sensors 35 associated with the image frames (e.g., position, lens parameters, etc.). The scaling coefficient is calculated based on environmental data 33 received by the machine learning model 31 for specific visual sensors 35 associated with the image frames used in the mixed reality environment 45.

[0084] As described in the flowchart 500 in Figure 5, the machine learning model 31 identifies environment objects 37 within the image frame. Once the environment objects 37 are identified, 2D or 3D boundary polygons and their positions relative to the vehicle 1 are created within the game-centered coordinate system 54. These boundary polygons are placed in the scene to represent the positions of the associated environment objects 37 relative to the vehicle 1 in 3D space on the game board 64. In particular, the positions of the boundary polygons on the game board 64 are determined by depth estimation of each environment object 37.

[0085] In Figure 6, the first 3D boundary polygon 68 and the second 3D boundary polygon 70, representing the first and second environmental objects identified from the image frames captured by the aforementioned detection sensor 5, are drawn along the game board 64. The first boundary polygon 68 is positioned closer to the near side 62 along the game board 64 than the second boundary polygon 70, thereby indicating that the first environmental object is closer to the vehicle 1 than the second environmental object in the external environment.

[0086] In addition, the machine learning model 31 may place virtual objects within the game-centered coordinate system 54 for the purpose of selected gameplay. In Figure 6, a first virtual object 72 representing an avatar 50 is included in the game-centered coordinate system 54. The avatar 50 can be any virtual object and is controlled by user input 41 and vehicle data. In addition, Figure 6 includes a second virtual object 74 representing obstacles in gameplay, which is positioned on the game board 64.

[0087] Since the second virtual object 74 is not positioned between the boundary polygon and the far plane 60, the second virtual object 74 is projected onto the current image frame as seen on the display. If the virtual object is positioned or moved between one of the boundary polygons of the identified environment object 37 and the far plane 60, the machine learning model 31 can make the virtual object transparent or remove it to avoid the virtual object being positioned at the same location as the environment object 37 in the output image frame.

[0088] The movement of virtual objects within the game-centered coordinate system 54 is linked to the movement of vehicle 1 in the external environment. Specifically, virtual objects move within the game-centered coordinate system 54 based on vehicle data of vehicle 1. For example, in Figure 6, the movement of the first virtual object 72, which is an avatar 50 controlled by vehicle data, along the X direction is stationary within the game-centered coordinate system 54 because the first virtual object 72 is moving at the same speed as vehicle 1, which is capturing the image frame. However, if the second virtual object 74 is designed to appear static on the game board 64, then the second virtual object 74 must move in the game-centered coordinate system 54 toward the near direction 62 and the virtual camera 56 at the same speed as vehicle 1 is moving in the external environment as depicted in the image frame.

[0089] In another example, the second virtual object 74 may move through the mixed reality environment 45 of the gameplay at approximately the same speed as the first virtual object 72. If the second virtual object 74 increases its speed relative to the first virtual object 72, the second virtual object 74 moves along the X-axis of the game-centered coordinate system 54 toward the far direction 60 on the game board 64. If the second virtual object 74 decreases its speed relative to the first virtual object 72, the second virtual object 74 moves along the X-axis of the game-centered coordinate system 54 toward the near direction 62 on the game board 64.

[0090] In one or more embodiments, a virtual object can move laterally along the Y-axis of the game-centered coordinate system 54. Lateral movement shifts the position of the virtual object left or right relative to the image frame on the game board 64. However, because the projected image frame shows forward motion, the virtual object may appear to be continuously moving forward.

[0091] In one or more embodiments, the lateral movement of a virtual object may be restricted. Specifically, a virtual object may be restricted by its sides (not shown). These sides may be the vertical sides of a viewing frustum 58 extending between the near plane 62 and the far plane 60. If a virtual object moves sufficiently laterally along the game board 64 in the game center coordinate system 54 to reach its sides, the virtual object is prevented from moving further lateral. Alternatively, a virtual object may be restricted by the width of a detected road 46 projected onto the game board 64. Therefore, if a first virtual object 72 is closer to the near plane 62 than a second virtual object 74, the first virtual object 72 may move a greater lateral distance than the second virtual object 74 in the game center coordinate system 54 due to the perspective of the road 46, which reduces the width of the road 46 as it extends toward the vanishing point of the horizon 66.

[0092] In one or more embodiments, a virtual object may be able to move along the Z-axis within the game-centered coordinate system 54. That is, a virtual object may be able to lift off the game board 64. For example, a virtual object may be able to jump to avoid collisions with additional virtual objects or environment objects 37. The movement of a virtual object in the Z direction may be restricted by the game board 64, or the ground, and the horizontal upper surface (not shown) of a viewing frustum 58 extending between the near plane 62 and the far plane 60.

[0093] In Figure 7, the game-centered coordinate system 54 includes an exemplary image frame projected onto the far-field plane 60. Here, the image frame is of the external environment of vehicle 1, collected by the vehicle's visual sensor 35. Similar to Figure 6, the game-centered coordinate system 54 includes a first 3D boundary polygon 68 and a second 3D boundary polygon 70, representing a first environment object and a second environment object, respectively, identified from the image frame projected onto the far-field plane 60. In this case, the first boundary polygon 68 and the second boundary polygon 70 represent an additional vehicle 48 along the same road 46 as vehicle 1. Thus, the game board 64 of the game-centered coordinate system 54 represents this road 46 in the image frame. In addition, the avatar 50 in this example of Figure 7 is represented as a sports car. Furthermore, the second virtual object 74 is superimposed or fused onto an image frame of the mixed reality environment 45 by the machine learning model 31 (i.e., an augmented image frame 43 is created) so that the second virtual object is visible to the gameplay user.

[0094] In the resulting mixed reality gameplay, the detected environment objects 37 can become elements of the gameplay. For example, the gameplay may require an avatar 50 controlled by user input 41 to avoid collisions with the environment objects 37 detected from the image frame and virtual objects introduced by the machine learning model 31. If user input 41 fails to move the avatar 50 in a direction that avoids the environment objects 37, a collision may occur between the avatar 50 and the environment objects 37. If a collision occurs, the avatar 50 may appear to bounce off the environment objects 37 by giving the environment objects 37 changes in velocity and direction, determined by the machine learning model 31 and, in one or more embodiments, uncontrollable by user input 41, to the environment objects 37. Alternatively, the avatar 50 may be visually depicted as exploding based on contact with the environment objects 37. Similarly, if a virtual object not under the control of user input 41 interacts with the position of the environment objects 37, the virtual object may give the environment objects 37 changes in velocity and direction, pass through the environment objects 37, or be completely removed from the mixed reality environment 45. If Avatar 50 interacts with the position of different virtual objects, Avatar 50 and / or the virtual objects may cause the other to change in velocity and direction, pass through the other, or be removed from the mixed reality environment 45. Alternatively, depending on the gameplay, gameplay may end if Avatar 50 collides with an environment object 37 or a different virtual object.

[0095] Figure 8 shows a flowchart 800 of a method for changing the mixed reality environment 45 for gameplay based on multiple conditions, according to one or more embodiments.

[0096] In step 802, the processing unit 7 of the system 23 creates a mixed reality environment 45 for gameplay based on environmental data 33 acquired from multiple detection sensors 5 of the vehicle 1. The gameplay selected from multiple gameplay applications is stored in the memory of the processing unit 7.

[0097] In step 804, the processing unit 7 starts gameplay upon receiving instructions from the user. To this end, the processing unit 7 transmits the gameplay to multiple mobile devices 13 so that it can be viewed on the display screens of the mobile devices 13.

[0098] In step 806, the processing unit 7 receives environment data 33 and vehicle data from multiple detection sensors 5 and multiple vehicle sensors 3, respectively. Thus, the positions of environment objects 37 and their respective boundary polygons are updated within the game-centered coordinate system 54, and as a result, the mixed reality environment 45 is also updated in this manner. In addition, the activities of the avatar 50 (e.g., position, speed, etc.) can be updated and controlled based on the latest received vehicle data. In one or more embodiments, the avatar 50 is configured to traverse the mixed reality environment 45 in one direction at a speed proportional to the vehicle 1.

[0099] In addition, upon receiving the latest environmental data 33 from multiple detection sensors 5 (which may occur in real time while the vehicle is moving), the processing unit 7 may change the weather and time of day (e.g., brightness / darkness, sun / moon position, etc.) depicted in the mixed reality environment 45 of the gameplay. For example, if rain is detected by multiple detection sensors 5 in the external environment of the vehicle 1, the machine learning model 31 may generate rain in the gameplay. Thus, the physics calculations of the gameplay may be adjusted in accordance with the weather changes. For example, if the gameplay includes rainy and / or snowy weather, virtual objects may be given a lower virtualized coefficient of friction, resulting in virtual objects skidding or sharp turns in response to user input 41 of acceleration and lateral movement. In addition, the lighting, reflections, and virtual shadows of virtual objects and environment objects 37 depicted in the gameplay may also be adjusted according to the weather and time of day.

[0100] In step 808, the processing unit 7 receives user input 41 from multiple mobile devices 13. Next, the processing unit 7 controls the activities of the avatar 50 within the game-centered coordinate system 54, and therefore within the mixed reality environment 45, based on the received user input 41. For example, user input 41 could be steering commands that control the position, orientation, movement, etc., of the avatar 50. In addition, user input 41 may adjust one or more settings of the gameplay. For example, one or more settings that can be adjusted could be the viewpoint of the virtual camera 56 in the mixed reality environment 45, the weather, the physics calculations of the gameplay (e.g., gravity, acceleration across the game board 64 in the game-centered coordinate system 54, etc.), the damage tolerance or durability of the avatar 50, the amount / frequency / type of additional virtual objects introduced in the gameplay, etc.

[0101] In step 810, the processing unit 7 determines whether gameplay has ended. That is, the processing unit 7 determines whether gameplay was paused / canceled by user input 41, whether the end of gameplay was reached (e.g., by winning / losing the game, by a time limit, etc.), whether the avatar 50 was affected (e.g., by a collision with an environment object 37 or a virtual object), etc. If "yes" (i.e., gameplay has actually ended), the method can terminate. If "no", the method can proceed to step 812.

[0102] In step 812, processing unit 7 determines whether vehicle 1 is in motion based on vehicle data. That is, processing unit 7 may determine whether vehicle 1 is parked based on vehicle data. Alternatively, processing unit 7 may determine whether vehicle 1 has reached a familiar destination or the end of a navigation route based on the navigation data of the vehicle data. If "yes" (i.e., vehicle 1 is in motion), the method can return to step 806. If "no", the method can proceed to step 814.

[0103] In step 814, the processing unit 7 sends a request to determine whether the user of the gameplay wishes to continue playing the game. The request may be made via multiple mobile devices 13 or the infotainment module 9 of the vehicle 1 and may require user input 41. The request may be visual and / or auditory and may include haptic feedback if applicable. If "yes" (i.e., to continue playing the game), the method may return to step 806. If "no", the method may terminate.

[0104] Figure 9 shows an exemplary gameplay visualization, as seen on a mobile device 13, in one or more embodiments. Here, the mobile device 13 is depicted as a smartphone. However, in one or more embodiments, the mobile device 13 may be a tablet or a portable game console. Each mobile device 13 may communicate wirelessly with the processing unit 7 of the system 23. In addition, in one or more embodiments, the gameplay may be displayed on the infotainment module 9 of the vehicle 1. In this way, the mobile device 13 can control the gameplay viewed via the display system 19 of the infotainment module 9.

[0105] In Figure 9, the gameplay is displayed to the user of system 23 via the display screen 76 of the mobile device 13. The gameplay of system 23 utilizes a mixed reality environment 45 as described in Figures 3 to 7. As previously mentioned, the mixed reality environment 45 for gameplay includes a series of augmented image frames 43 that include environment objects 37 and superimposed virtual objects. The location of the mixed reality environment 45 is the external environment of vehicle 1, captured by multiple detection sensors 5 of vehicle 1. Here, in Figure 9, the location of the mixed reality environment 45 for gameplay is a road 46 that crosses a bridge.

[0106] In one or more embodiments, the gameplay may be multiplayer. For this purpose, the mixed reality environment 45 may include multiple avatars 50. Each avatar 50 may be controlled by vehicle data and user input 41 via a different mobile device 13. In this way, if multiple occupants of the vehicle 1 are accessing the gameplay via their respective mobile devices 13, each occupant can participate in the same gameplay. Alternatively, each occupant may participate in a different gameplay on their respective mobile device 13 from the gameplay of other occupants. When multiple users are participating in the same gameplay, each user's mobile device 13 may display the same view of the mixed reality environment 45, including the same environment objects 37 and the same virtual objects. In this way, users of multiplayer gameplay can compete with each other or work as a team to complete gameplay challenges and activities. In one or more embodiments, the multiplayer gameplay may be displayed via the display system 19 of the infotainment module 9 for multiple occupants of the vehicle 1 and / or users of the gameplay to view a single display screen 76 during gameplay.

[0107] In one or more embodiments, each user's mobile device 13 may display different viewpoints of the mixed reality environment 45 for each user of the gameplay. For example, in Figure 9, a first user can control a first avatar 78 depicted as a car, and a second user can control a second avatar 80 depicted as a motorcycle driver. The first user's mobile device 13 may display the mixed reality environment 45 of the gameplay as seen on the mobile device 13 depicted in Figure 9. However, a second user may adjust the gameplay settings on their respective mobile device 13 so that the mixed reality environment 45 of the second user's gameplay is displayed from a different viewpoint. Specifically, the viewpoint of the second user's gameplay may be formed from a video feed captured by a different visual sensor 35 (e.g., a different position along the vehicle 1, different lens parameters, etc.) than the visual sensor 35 that captured the video feed for the first user's gameplay.

[0108] In one or more embodiments, one or more virtual objects may be virtual icons. A virtual icon is a virtual object that remains stationary in a reserved area of ​​the display screen 76 of the mobile device 13. That is, a virtual icon may be projected onto the near plane 62 of the game-centered coordinate system 54. In this way, a virtual icon is always drawn in front of virtual objects, environment objects 37, and the current image frame.

[0109] Virtual icons can have several applications, including displaying gameplay information, vehicle data, and facilitating user input 41. Figure 9 shows some examples of virtual icons. The first virtual icon 82 is depicted as a speedometer that notifies the gameplay user of recorded vehicle data and / or speed associated with the gameplay avatar 50. The second virtual icon 84 and the third virtual icon 86 depict a toggle and a button, respectively, which facilitate user input 41 on a mobile device 13 with touchscreen functionality. Thus, virtual icons can be used to facilitate the activities and movements of the avatar 50.

[0110] In one or more embodiments, the display screen 76 of the mobile device 13 may be a touchscreen used to transmit user input 41 to the processing unit 7. Alternatively, the mobile device 13 may include physical control functions (e.g., buttons, joysticks, etc.) used to facilitate user input 41 to gameplay. Furthermore, in one or more embodiments, the mobile device 13 may include an accelerometer (not shown) for measuring the movement of the mobile device 13 and converting the measured movement into user input 41 to control the activities of the avatar 50. In this way, tilting the mobile device 13 from side to side may cause lateral movement of the avatar 50. In addition, tilting the mobile device 13 forward (i.e., away from the user) may cause acceleration of the avatar 50, while tilting the mobile device 13 backward (i.e., towards the user) may cause deceleration of the avatar 50.

[0111] The mixed reality environment 45 of the gameplay depicted in Figure 9 also includes virtual objects related to the specific gameplay, which are placed along the road 46 game board 64 by a machine learning model 31. Specifically, in this non-limiting example, virtual objects depicting coins 88 and holes 90 in the road are placed along the road 46. Here, these virtual objects appear stationary within the gameplay, as previously described in Figure 6. That is, these virtual objects move within the game-centered coordinate system 54 toward a virtual camera 56 in the game-centered coordinate system 54 at the same speed as the vehicle 1 travels through the external environment. In the depicted gameplay, the objective of the gameplay may be for the user to move the avatar 50 across the game board 64 so that the avatar 50 collects as many coins 88 as possible during the gameplay. The fourth virtual icon 92 in Figure 9 depicts a scoreboard that records the number of virtual objects collected during the gameplay. In one or more embodiments, the coins 88 and other virtual objects may disappear from the mixed reality environment 45 upon collision with the avatar 50. In one or more embodiments, the objective of gameplay may be to avoid specific virtual objects and / or environmental objects 37. For example, in the gameplay depicted in Figure 9, the objective of gameplay may be for the user to move the avatar 50 across the game board 64 so that the avatar 50 avoids holes in the road while attempting to collect coins 88. If the avatar 50 collides with a hole in the road 90, visual effects may be applied to the avatar 50 (e.g., the avatar 50 shakes, rotates, flashes, etc.), the scoreboard may be reset to zero, the game may end, etc.

[0112] In one or more embodiments, the attributes of detected environmental objects 37 may be recorded by multiple detection sensors 5 of the vehicle 1. These recorded attributes of environmental objects 37 may be incorporated into gameplay. For example, if an additional vehicle 48 in front of vehicle 1 in the external environment is braking, has its turn signal on, is a specific color, or is changing lanes, this detection may trigger an event in gameplay.

[0113] In one or more embodiments, alerts from the vehicle 1's warning system (not shown) can be incorporated into the gameplay. For example, if vehicle 1 includes a blind spot warning system that frequently issues alerts because an additional vehicle 48 is located within a defined blind spot of vehicle 1, the alerts can be incorporated into the gameplay to warn the gameplay user to avoid spaces where environmental objects 37 associated with the additional vehicle 48 may appear in the mixed reality environment 45. In this way, a collision between the avatar 50 and the environmental objects 37 associated with the additional vehicle 48 may be avoided. The alerts may be visual alerts 94 in the gameplay, such as glowing overlays, flashing overlays, virtual objects, or virtual icons.

[0114] In one or more embodiments, multiple detection sensors 5 may detect and track the lanes of a road 46 in the external environment (e.g., the number of lanes on road 46, the lane on road 46 in which vehicle 1 is positioned, the width of the lanes on road 46, etc.). Thus, the content augmentation engine 27 may place the lanes on the game board 64 according to the lanes of road 46. In this way, the placement of virtual objects may be done so that the virtual objects appear to be within the range of the lanes. For example, if an additional vehicle 48 is located in the blind spot to the right of vehicle 1, as described in the points above, the visual alert 94 in gameplay may be a colored overlay mapped onto the surface of the right lane in the mixed reality environment 45.

[0115] The fifth virtual icon 96 in Figure 9 depicts a bird's-eye view of the external environment or mixed reality environment 45. In one or more embodiments, the fifth virtual icon 96 may instead display a rear view of the external environment or mixed reality environment 45. This fifth virtual icon 96 allows the gameplay user to gain an additional understanding of the positions of the environment objects 37 and / or virtual objects along the game board 64 of the mixed reality environment 45, even if all environment objects 37 and / or virtual objects are not yet visible from the avatar 50's current viewpoint in the gameplay.

[0116] In one or more embodiments, the viewpoint of the avatar 50 in the mixed reality environment 45 can be adjusted by user input 41. Alternatively, the viewpoint can be dynamically shifted between different views of the mixed reality environment 45 based on different video feeds captured by various visual sensors 35 of the multiple detection sensors 5. For example, the initial viewpoint of the mixed reality environment 45 may utilize a series of image frames captured by a forward visual sensor 35 located at the front of the vehicle 1, but based on the activity of the avatar 50, the viewpoint of the mixed reality environment 45 may be changed to utilize a side visual sensor 35 located along the side of the vehicle 1, and then a rear visual sensor 35 located at the rear of the vehicle 1, as the viewpoint of the avatar 50 shifts around the vehicle 1 along with the position of the avatar 50 in the mixed reality environment 45.

[0117] In Figure 10, gameplay is displayed to the user of the system 23 via the display screen 76 of a mobile device 13, according to one or more embodiments. Here, the mobile device 13 is a portable game console. In this non-limiting example, the portable game console includes a display screen 76 and a number of controllers 98. Each controller 98 of the portable game console includes buttons 100 and a joystick 102 to facilitate user input 41 to gameplay. Furthermore, the display screen 76 of the portable game console may be a touchscreen.

[0118] The mixed reality environment 45 depicted in Figure 10 displays a 3D bowl-view of the avatar 50. In one or more embodiments, the vehicle 1 may include multiple vision sensors 35 that capture a video feed in a 360-degree range around the vehicle 1. Multi-camera synthesis, or fusion of image frames captured by multiple vision sensors 35 capturing a 360-degree range around the vehicle 1, can create a bowl-view of the external environment.

[0119] In Figure 10, vehicle 1 is depicted as a virtual object behind avatar 50. This is because multiple visual sensors 35, which capture a 360-degree range around vehicle 1, cannot directly capture vehicle 1. Furthermore, to provide the gameplay user with a more realistic view, avatar 50 is depicted in a way that allows it to hide parts of vehicle 1 in the mixed reality environment 45.

[0120] By employing a bowl view, gameplay users can view the avatar 50 from any viewpoint within the bowl. That is, the viewpoint of the avatar 50 within the mixed reality environment 45 is adjustable between top view, front view, rear view, side view, and angle view. In addition, the bowl view allows the avatar 50 to be controlled from the front, back, or side of a virtual object representing the vehicle 1.

[0121] To adjust the viewpoint of the avatar 50, the user may select a desired viewpoint by adjusting one or more gameplay settings via user input 41. In the non-limiting example of Figure 10, the mixed reality environment 45 includes a settings virtual icon 104 that depicts a shortcut to the gameplay settings. To access the settings, the user can press a button 100 on the controller 98 associated with the settings or touch the settings virtual icon 104. In one or more embodiments, the mixed reality environment 45 may include a orientation virtual icon 106. If the user wants to change the viewpoint of the avatar 50, the user can touch and drag the orientation virtual icon 106, which rotates the avatar 50 around its center point (not shown).

[0122] Figures 11 and 12 show a system overview of an automotive mixed reality gaming system 23 for a vehicle 1 according to one or more embodiments of the present invention. In Figure 11, the system 23 includes a vehicle 1, a processing unit 7, and one or more mobile devices 13. The vehicle 1 includes a plurality of detection sensors 5, which may be visual sensors 35 such as a camera, a radar unit, a LiDAR unit, an ultrasonic sensor, or other equivalent sensors known to those skilled in the art. In addition, the vehicle 1 includes a plurality of vehicle sensors 3, which may include a navigation sensor. As described above, the plurality of detection sensors 5 and the plurality of vehicle sensors 3 each capture environmental data 33 and vehicle data and transmit them to the ECU 17 of the vehicle 1. The ECU 17 may be formed of one or more processors, microprocessors, or equivalent computing configurations that perform processing of the various environmental data 33 and vehicle data described herein. The environmental data 33 and vehicle data are transmitted to the ECU 17 via a bus 11. The bus 11 may be in the form of wires, wire harnesses, circuit boards, or equivalent means for interconnecting the various components of the vehicle 1. The ECU17 may include components for hardware-accelerated video processing, including machine learning-based video processing. Thus, the ECU17 processes environmental data 33 and vehicle data using algorithms and processes, and extracts information about environmental objects 37 and contextual information such as the position, speed, and direction of vehicle 1 from the captured data.

[0123] After capturing environmental data 33 and vehicle data, the ECU 17 and / or the multiple detection sensors 5 and the multiple vehicle sensors 3 transmit the environmental data 33 and vehicle data to the memory 108 of the vehicle 1, respectively. This memory 108 is a non-temporary storage medium such as flash memory, random access memory (RAM), hard disk drive (HDD), solid state drive (SSD), a combination thereof, or equivalent. In addition to storing the captured environmental data 33 and vehicle data, the memory 108 stores the inputs, outputs, functions, and processes necessary to perform the gameplay application, user interaction (via the mobile device 13 or infotainment module 9), and other functions and processes described herein. Furthermore, in one or more embodiments, the memory 108 of the vehicle 1 may also store the gameplay application itself.

[0124] In Figure 11, Vehicle 1 is wirelessly communicating with Server 15, which includes a Processing Unit 7. Server 15 is a cloud-based server configured to perform augmentation and / or other operations to create a mixed reality environment 45 for gameplay, which includes captured environmental data 33 and vehicle data. Specifically, Server 15 includes a cloud-based instance of a developer kit that facilitates the development of mixed reality applications. Processing Unit 7 includes tools such as a processor 110 for processing a series of image frames, other environmental data 33, and / or vehicle data acquired in raw or pre-processed form. The Processing Unit 7 of Server 15 may perform an object detection process via dedicated components such as a graphics card 112. In one or more embodiments, the object detection process may be performed by an ECU 17 and then transmitted to Server 15.

[0125] The processing unit 7 of the server 15 includes a memory 108 that holds data for the server 15, or data for other components that may be connected to the server 15 (e.g., the ECU 17 and mobile devices 13) (or a combination of both). For example, the memory 108 may be a database that stores data consistent with the present disclosure. A machine learning model 31, including an environment interpretation engine 25, a content augmentation engine 27, and a rendering engine 29, may reside in the memory 108 of the processing unit 7 and may be present for rendering 2.5D output as described above in Figures 6 and 7. In one or more embodiments, the content augmentation engine 27 and / or the rendering engine 29 of the machine learning model 31 may reside in the memory 108 of multiple mobile devices 13, infotainment modules 9, and / or the ECU 17 of the vehicle 1. Although shown as a single memory 108 in Figure 11, multiple memories 108 may be used according to specific needs, requirements, or specific implementations and described functions of the server 15.

[0126] Furthermore, the server 15 includes a transceiver 114 that wirelessly transmits and receives signals between the transceiver 114 of the vehicle 1 and the user's mobile device 13. Examples of embodiments of the transceiver 114 include an antenna (not shown) and a processor 110 that transmit and receive radar, radio, cellular, satellite, Wi-Fi, or other equivalent signals. The vehicle 1 and the mobile device 13 may be part of a network of the vehicle 1 and the mobile device 13, in which case the transceiver 114 of the server 15 may transmit and receive signals of different and / or more signal types.

[0127] The gameplay of System 23 is transmitted from Server 15 to Mobile Device 13 by signal. Server 15 receives user input 41 as a signal transmitted from Mobile Device 13 (for example, to operate relevant prompts, control the activities of avatar 50 in gameplay, etc.). Each Mobile Device 13 may include a bus 11, a display screen 76, memory 108, a processor 110, and a transceiver 114. Mobile Device 13 can capture user input 41, which is sent back to Processing Unit 7. User input 41 can be captured via the physical control functions of Mobile Device 13 (e.g., buttons 100, joysticks 102, etc.), via the accelerometer of Mobile Device 13 (not shown), or as touch input if the display screen 76 of Mobile Device 13 has touchscreen functionality. Gameplay may be accessible via an application on Mobile Device 13, or similarly via a web interface, a dedicated server, or other equivalent software. Such software is hosted by server 15 via ECU 17 on vehicle 1, on mobile device 13, or a combination thereof. Thus, the user can interact with the mixed reality environment 45 of the gameplay by interacting with the mobile device 13.

[0128] In one or more embodiments, the processing unit 7 may be a component of the ECU 17 of the vehicle 1, as shown in Figure 12. At least some operations of the processing unit 7 when embodied as the ECU 17 (e.g., environment interpretation, rendering, etc.) can benefit from the availability of a GPU or graphics card 112. Thus, the ECU 17 may be equipped with a GPU or graphics card 112. In addition, the ECU 17 may still communicate wirelessly with an external server for purposes such as application downloads and application maintenance.

[0129] Therefore, in the embodiment shown in Figure 12, the gameplay of the system 23 is transmitted by signal from the vehicle 1 to the mobile device 13. The vehicle 1 transmits the user input 41 received from the mobile device 13 to the ECU 17 for execution. In one or more embodiments, the gameplay may be transmitted from the ECU 17 to the infotainment module 9 of the vehicle 1 via the bus 11.

[0130] Figure 13 shows a flowchart 1300 of a method for operating an automotive mixed reality gaming system 23 for a vehicle 1 according to one or more embodiments disclosed herein. The steps of the flowchart shown in Figure 13 may be performed by the automotive mixed reality gaming system 23 as described herein, but are not limited thereto. The configuration steps of the method shown in Figure 13 may be performed in any logical order, and the method is not limited to the order presented.

[0131] As shown in Figure 13, the method begins in step 1310, which involves using multiple vehicle sensors 3 on vehicle 1 to measure vehicle data of vehicle 1. The vehicle data is associated with the movement and position of vehicle 1 as vehicle 1 travels through the external environment. In one or more embodiments, the vehicle data collected includes current speed, acceleration, steering angle, yaw rate, braking state, blind spot and proximity alerts.

[0132] Multiple vehicle sensors 3 may include navigation sensors that receive signals containing the global coordinates of vehicle 1. The navigation sensors may be, for example, GPS sensors or equivalent sensors that determine the position of vehicle 1 in relation to its external environment, and trilateration, triangulation, or similar procedures may be used to determine its position. The signals may also include information such as the direction and speed of vehicle 1.

[0133] In step 1320, the multiple detection sensors 5 capture environmental data 33 of the external environment of the vehicle 1. The environmental data 33 may be captured while the vehicle 1 is stationary or moving. The multiple detection sensors 5 of the vehicle 1 may be positioned along the front, rear, and / or sides of the vehicle 1 and may take the form of a two-dimensional camera, a three-dimensional camera or stereo camera, a radar unit, a LiDAR unit, an ultrasonic sensor, or an equivalent sensor or component that perceives the environment of the vehicle 1. The multiple detection sensors 5 along the vehicle 1 may all be detection sensors 5 of the same type, or they may include a combination of detection sensors 5 of different types.

[0134] In one or more embodiments, the plurality of detection sensors 5 include a plurality of vision sensors 35. Each vision sensor 35 may capture a video feed related to the field of view of the external environment of the vehicle 1. For this purpose, each video feed may be formed from a series of image frames.

[0135] Furthermore, the environmental data 33 collected by the multiple detection sensors 5 may include several different views of the road 46 on which the vehicle 1 is located, as well as additional vehicles 48, obstacles, pedestrians, signs, and other landmarks near the vehicle 1.

[0136] In step 1330, the vehicle data and environmental data 33 are transmitted to the processing unit 7 of the system 23. In one or more embodiments, the processing unit 7 is located in a cloud-based server 15 and includes a processor 110 and memory 108. In this way, the multiple vehicle sensors 3 and the multiple detection sensors 5 can each transmit the vehicle data and environmental data 33 in raw or pre-processed form to the processing unit 7 of the server 15. Alternatively, the multiple vehicle sensors 3 and the multiple detection sensors 5 can each transmit the vehicle data and environmental data 33 wirelessly or via the bus 11 of the vehicle 1 to the ECU 17 of the vehicle 1. The ECU 17 of the vehicle 1 may be formed by one or more processors, microprocessors, integrated circuits (ICs), or equivalent computing configurations. Thus, the vehicle data and environmental data 33 may be processed by the ECU 17 before being transmitted to the server 15. Furthermore, in one or more embodiments, the processing unit 7 is contained within the ECU 17 of the vehicle 1, and the ECU 17 can process the vehicle data and environmental data 33 after receiving it.

[0137] In step 1340, the environment data 33 received by the processing unit 7 is transformed into a game-centered coordinate system 54. In this way, the environment interpretation engine 25 identifies environment objects 37 within a series of image frames of the environment data 33. The environment interpretation engine 25 may utilize deep learning-based image processing such as OmniDet, Single-Shot Deep MANTA, Single Shot Detection (SSD), Region-Based Convolutional Neural Network (R-CNN), You Only Look Once (YOLO) Single Shot Detection, Fast R-CNN, and Histogram of Oriented Gradients (HOG). The output obtained as a result of the deep learning-based image processing may include depth estimation, semantic segmentation, visual odometry, motion segmentation, and / or object detection of environment objects 37.

[0138] Next, post-processing of the identified environment objects 37 may be performed. That is, necessary data manipulation is performed to obtain the boundaries of the identified environment objects 37. In addition, a 2D or 3D boundary polygon is formed for each identified environment object 37. These boundary polygons are given a position relative to the vehicle 1 in the game-centered coordinate system 54 formed by the developer kit of the processing unit 7. For this purpose, the boundary polygons are positioned along the game board 64 of the game-centered coordinate system 54 so that the position of the associated environment object 37 is relative to the vehicle 1 in 3D space.

[0139] In step 1350, gameplay is formed that includes a mixed reality environment 45 utilizing environmental data 33. The mixed reality environment 45 of the gameplay may be generated in real time based on the actual external environment surrounding the vehicle 1. In particular, the mixed reality environment includes a series of image frames of the external environment with various augments. The augmented content 39 added to the mixed reality environment 45 depends on the intended gameplay of the mixed reality environment 45. The augmented content 39 includes added objects, changes to content within image frames by masking, etc. Added objects include, for example, one or more avatars 50, objects, symbols, labels, features, animated characters, text, etc.

[0140] In one or more embodiments, the rendering engine 29 receives a series of image frames and augmented content 39 from the content augmentation engine 27 and generates a series of augmented image frames 43. In one or more embodiments, the rendering of the augmented image frames 43 may be performed based on an input that includes each image frame itself, an identified environment object 37, and the augmented content 39. Further inputs that may affect rendering may be received from the user. For example, the user may change the rendered view by control commands, steering control movements, etc.

[0141] In addition, the content augmentation engine 27 and the rendering engine 29 may position virtual objects within the game-centered coordinate system 54 for the purpose of selected gameplay. For example, in one or more embodiments, gameplay may include specific environment objects 37 and / or virtual objects as augmentation content 39 to be avoided and / or collected, which may punish or reward the user of the system 23 based on the identification information of the augmentation content 39. In addition, in step 1360, an avatar 50 is placed within the mixed reality environment 45 for gameplay. The avatar 50 is a virtual object and element of gameplay controlled by user input 41 and vehicle data. Furthermore, the avatar 50 may be represented as any virtual object (e.g., a car, an airplane, a human, an animal, etc.). In one or more embodiments, the avatar 50 is configured to traverse the mixed reality environment 45 in one direction at a speed proportional to the vehicle 1.

[0142] In step 1370, gameplay (i.e., a series of augmented image frames 43 of the mixed reality environment 45) is displayed on multiple mobile devices 13 and / or the infotainment module 9 of the vehicle 1. The multiple mobile devices 13 may be embodied as smartphones, tablets, and portable game consoles. In one or more embodiments, the multiple mobile devices 13 are connected to the vehicle 1 by wired or wireless technology (e.g., WiFi, Bluetooth, etc.). Alternatively, both the vehicle 1 and the multiple mobile devices 13 may be connected to the cloud-based server 15 of the processing unit 7 via wireless technology (e.g., cellular, 4G, 5G, satellite, etc.).

[0143] When gameplay rendering is performed for multiple users, the extended image frame 43 can be displayed on multiple mobile devices 13. In one or more embodiments, one of the multiple mobile devices 13 may be located outside the vehicle 1 and communicate with the server 15 or the vehicle 1.

[0144] In addition to displaying gameplay to the user of system 23, the multiple mobile devices 13 capture user input 41 to control activities within the gameplay and / or adjust one or more settings of the gameplay. In step 1380, the processing unit 7 receives user input 41 from the multiple mobile devices 13. User input 41 can be captured via the physical control functions of the mobile devices 13 (e.g., buttons 100, joysticks 102, etc.), or via the accelerometers (not shown) of the mobile devices 13, or as touch input if the display screen 76 of the mobile devices 13 has touchscreen functionality.

[0145] In step 1390, the processing unit 7 controls the activity of the avatar 50 within the game-centered coordinate system 54 and therefore within the mixed reality environment 45 based on the received user input 41. For example, the user input 41 could be steering commands that control the position, orientation, movement, etc., of the avatar 50. Next, the processing unit 7 updates the position, orientation, movement, etc., of the avatar 50 according to the user input 41 within the game-centered coordinate system 54 and the mixed reality environment 45. In addition, the user input 41 may adjust one or more settings of the gameplay. For example, one or more settings that can be adjusted may be the viewpoint of the virtual camera 56 in the mixed reality environment 45, the weather, the physics calculations of the gameplay (e.g., gravity, acceleration across the game board 64 in the game-centered coordinate system 54, etc.), the damage tolerance or durability of the avatar 50, the amount / frequency / type of additional virtual objects introduced in the gameplay, etc.

[0146] Furthermore, the processing unit 7 updates and controls the activities of the avatar 50 (e.g., position, speed, etc.) based on the received vehicle data. In one or more embodiments, the avatar 50 is configured to traverse the mixed reality environment 45 in one direction at a speed proportional to the vehicle 1. Therefore, if the vehicle 1 accelerates in the external environment, the processing unit 7 can ensure that the user's avatar 50 accelerates in the same manner as the vehicle 1 in the game-centered coordinate system 54 and therefore in the mixed reality environment 45.

[0147] During gameplay, the processing unit 7 updates the gameplay and mixed reality environment 45 based on the latest environment data 33, vehicle data, and user input 41. In one or more embodiments, the processing unit 7 determines whether gameplay has ended. That is, the processing unit 7 determines whether gameplay has been paused / canceled by user input 41, whether the end of gameplay has been reached (e.g., by winning / losing the game, by a time limit, etc.), whether the avatar 50 has been affected (e.g., by a collision with an environment object 37 or a virtual object), etc. In one or more embodiments, the processing unit 7 sends a request to the user via the display screen 76 of the mobile device 13 to determine whether the user of the gameplay wishes to continue gameplay. If gameplay has ended or has been terminated by the user, the method terminates. If gameplay has not ended, the method may continue in step 1310 to update the gameplay.

[0148] Accordingly, embodiments disclosed herein relate to a system 23 and method useful for operating an automotive mixed reality gaming system 23 for a vehicle 1. The disclosed system 23 and method can advantageously be played on the mobile devices 13 of the occupants of the vehicle 1. Furthermore, the disclosed system 23 and method advantageously provides pre-processed APIs and augmented image frames 43 to multiple mobile devices 13. In addition to using vehicle data within the gameplay of the gaming system 23, the disclosed system 23 and method advantageously utilizes multiple video feeds captured by multiple detection sensors 5 of the vehicle 1 as the background for the gameplay. Furthermore, the disclosed system 23 and method advantageously incorporates environmental objects 37 and weather conditions of the external environment of the vehicle 1 detected by the multiple detection sensors 5 as interactive elements within the gameplay.

[0149] Although only a few embodiments of the present invention have been described in detail above, those skilled in the art will readily understand that many modifications are possible in the examples of embodiments without substantially departing from the present invention. Accordingly, all such modifications are intended to be included within the scope of this disclosure as set forth in the following claims.

[0150] In the claims, the means-plus-function clause is intended to include structures described herein as performing the enumerated functions, and is intended to include not only structural equivalents but also equivalent structures. Thus, while nails and screws may not be structural equivalents in that nails use a cylindrical surface to fasten multiple wooden parts, whereas screws use a helical surface, nails and screws may be equivalent structures in the context of fastening multiple wooden parts. The applicant's express intent is that any limitation in any of the claims herein is not subject to Section 112(f) of the U.S. Patent Act, except for the limitation to claims that explicitly use the phrase “means for” together with the relevant function.

Claims

1. A mixed reality gaming system for vehicles, Multiple vehicle sensors configured to collect vehicle data related to the movement and position of the vehicle along the road, Multiple detection sensors configured to collect environmental data related to the external environment of the vehicle, A processing unit including a processor and memory, The vehicle data is received from the plurality of vehicle sensors. The environmental data is received from the plurality of detection sensors. The aforementioned environmental data is converted to the game-centered coordinate system, The system generates gameplay for an interactive gameplay application stored in the memory, and the gameplay includes a mixed reality environment based on the game-centered coordinate system. A virtual avatar is depicted within the game's central coordinate system, and the avatar is configured to traverse the mixed reality environment in one direction at a speed proportional to the vehicle. The aforementioned gameplay is transmitted to multiple mobile devices, Receiving user input from the aforementioned multiple mobile devices, Furthermore, based on the vehicle data and user input, the system controls the avatar's activities and adjusts one or more settings of the gameplay. The processing unit is configured as follows: A mixed reality gaming system for automobiles, including [unspecified feature].

2. The automotive mixed reality gaming system according to claim 1, wherein the electronic control unit (ECU) of the vehicle includes the processing unit.

3. The automotive mixed reality gaming system according to claim 1, wherein the processing unit includes a server configured to receive the vehicle data and the environmental data via a wireless data connection.

4. The automotive mixed reality gaming system according to claim 1, wherein the game-centered coordinate system includes a plurality of projection planes and is configured to depict environmental objects identified by the processing unit from the environmental data in three-dimensional space.

5. The automotive mixed reality gaming system according to claim 1, wherein the processing unit is further configured to modify the mixed reality environment of the gameplay based on the position of the vehicle in the external environment.

6. The aforementioned vehicle data is The speed of the aforementioned vehicle, The direction of the aforementioned vehicle, The acceleration of the vehicle, and The alert from the vehicle's warning system, The automotive mixed reality gaming system according to claim 1, comprising at least one of the following.

7. The automotive mixed reality gaming system according to claim 1, wherein the gameplay includes interaction with virtual objects superimposed on the mixed reality environment by the processing unit.

8. The automotive mixed reality gaming system according to claim 1, wherein the gameplay is multiplayer gameplay, and multiple virtual avatars controlled by multiple mobile devices are depicted in the mixed reality environment.

9. The automotive mixed reality gaming system according to claim 1, wherein the processing unit is further configured to adjust one or more settings of the gameplay based on weather conditions of the external environment of the vehicle, the weather conditions being received and determined by the processing unit from the environmental data, an external server, and user input.

10. Further including an infotainment module, the infotainment module is A display system configured to receive and display the aforementioned gameplay, An interface configured to select one or more settings for the aforementioned gameplay The automotive mixed reality gaming system according to claim 1, including the following:

11. The automotive mixed reality gaming system according to claim 1, wherein the plurality of detection sensors include a plurality of vision sensors configured to capture a plurality of video feeds, and the viewpoint of the avatar dynamically shifts between the plurality of video feeds based on the avatar's activity.

12. The automotive mixed reality gaming system according to claim 1, wherein the avatar is depicted in a bowl view in the mixed reality environment such that the viewpoint of the avatar in the mixed reality environment can be adjusted between a top view, front view, rear view, side view, and angle view of the avatar based on the user input.

13. A method for operating an automotive mixed reality gaming system for vehicles, To collect vehicle data related to the movement and position of the aforementioned vehicle along the road, Collect environmental data related to the external environment of the aforementioned vehicle, Transmitting the aforementioned vehicle data and environmental data to the processing unit, Converting the aforementioned environmental data to the game-centered coordinate system, To generate gameplay of an interactive gameplay application stored in memory, wherein the gameplay includes a mixed reality environment based on the game-centered coordinate system. The process involves depicting a virtual avatar within the game-centered coordinate system, wherein the avatar is configured to traverse the mixed reality environment in one direction at a speed proportional to the vehicle. The aforementioned gameplay is transmitted to multiple mobile devices, Receiving user input from the aforementioned multiple mobile devices, Controlling the avatar's activities based on the vehicle data and user input, and adjusting one or more settings of the gameplay, Methods that include...

14. Converting the aforementioned environmental data to the game-centered coordinate system is, Obtaining an image frame from the aforementioned environmental data, Identifying at least one environmental object within the aforementioned image frame, Determining the extended content based on at least one of the aforementioned environment objects, Rendering an extended image frame based on the aforementioned image frame and the extended content, The method according to claim 13, including the method described in claim 13.

15. The method according to claim 13, wherein converting the environmental data to the game-centered coordinate system includes drawing at least one environmental object identified from the environmental data in three-dimensional space.

16. The method according to claim 13, further comprising changing the mixed reality environment of the gameplay based on the position of the vehicle.

17. The method according to claim 13, further comprising adjusting one or more settings of the gameplay based on weather conditions of the external environment of the vehicle, wherein the weather conditions are received and determined by the processing unit from the environmental data, an external server, and user input.

18. The method according to claim 13, further comprising transmitting the gameplay to a display system of an infotainment module.

19. The method according to claim 13, wherein transmitting the gameplay to multiple mobile devices includes transmitting the gameplay wirelessly.

20. The method according to claim 13, further comprising dynamically shifting the viewpoint of the avatar between a plurality of video feeds of the environmental data based on the avatar's activities.

Images